As part of WP 12239 we moved h0epics, cdsvmscript1, epics-burt, autoburt off of the cds0proxmox hypervisor.  This resulted in a few minutes of down time for the dust monitor IOC around 8:19am localtime.

After this we done we were able to look at cdsproxmox.  Its boot drive had failed.  After some help from Fil we got the drives replaced and reinstalled proxmox ve on cdsproxmox.  We adjusted DNS and renamed the box cdsproxmox0.

Some notes:

 * This is now in a temporary state.  We aim to retire this hardware by or at the end of O4.  New hypervisor computers are being procured.  As such we did not provision much storage on this.  Just enough to run the hypervisor, relying on the shared storage layer to handle the VM.

 * As per the proxmox administrators guild we removed cdsproxmox from the cluster prior to attaching it back as cdsproxmox0.

Closes FAMIS#28383, last checked 81688

Once again, the coherence for ITMX bias drive bias off is below the coherence threshold - this time the coherence is 0.01, much lower than the threshold of 0.1, so there are once again no new analyzed measurements for ITMX.

ETMX, ETMY, ITMX, ITMY

I've used the pico to center IM4 trans QPD.   See alogs about this history: 80604 78962 78943 78856

 After difficulty with input alignment shifts in the spring, we centered IM4 with the hopes that we could use that as a reference for any future input alignment shifts to reset the IMs.  However this didn't work well when the PMC was swapped.  Now we are recentering IM4 trans to get this new reference since we've been operating here for a while.

TJ, Camilla. WP12162. Continuing work in 76030

We installed a CFCS11-A adjustable SMA fiber collimator on the 50um fiber for the ETMX HWS fiber coupled LED source M530F2. This improved the size of the beam but it was still too large at the edge of the collimators range.

We removed HWS-L3 (D1800270) which made the beam a better size but still a little large ~ 10-15mm diameter at the periscope. We couldn't see a change in the beam by adjusting HWS-L2 (on translation stage). We aligned the system using the retrofection of the ALS beam: aligned ALS beam to output coup;er with HWS-MS1 and then adjusted the fiber collimator (mounted in mirror mount) to aligned the HWS output to the final iris. Confirmed we were getting a reflection of ETMX by mis-aligning it ~20urad. Note that the beam looks cleaner than usual photo, probably because the GV is closed so we are getting no retroflection of ITMX. Set frequecy to 1Hz. Replaced the mask photo and started the HWS code with new references.

The ETMX HWS 520nm beam in now injected into the vacuum where it hasn't been for ~months. Tagging DetChar. 

To do: measure the beam profile of beam out of fiber and re- calculate an imaging solution of the ETMX HR surface. 

GV5 and GV7 were soft closed at ~10am local time today to facilitate equipment craning for crane inspections.  They were opened at ~11am local time at the completion of the vertex crane inspection.

This morning, Ryan let me know that he got a "Check PSL chiller" verbal alarm.  Upon checking the chiller I saw the water level was a little low but not at minimum, but the usual oscillations in the water level as the chiller does its thing were getting a little close; this is likely what caused the alarm.  I added 175 mL of water to the chiller to bring the water level back up to the MAX line.

Closes FAMIS27804

CO2X was found at 29.5, i added 200ml to get it to right under the MAX line at 30.3.

CO2Y was found at 8, I added 250ml to get it to just under the MAX line at 9.4.

Early Monday morning I was alerted to the lag compressor alarm on the instrument air compressor. Both compressors were running and the tank pressure was increasing until it reached cut-out pressure. I inspected for any signs of an air leak but didn't find anything obvious. I observed the compressor and found that when the air dryer swapped desiccant towers it began blowing down a lot of air. The poly-tube connections beneath each dryer had a build-up of ice. I cleared the ice out as best as I could and observed the dryer for another several cycles, but the problem did not reoccur. We will continue to monitor the performance of the air dryer. 

We added some code to recognize the more revent timing board firmware revisions.

Tue Dec 10 10:10:41 2024 INFO: Fill completed in 10min 38secs

Camilla, Robert, WP12232

While looking for coupling of CO2 chillers into DARM at Crab pulsar 59Hz 81246, and also CO2s effect on SQZ 72244, this morning we turned off the CO2s and stayed locked ~40 minutes which is longer than we expected. Unsure of the lock loss cause as maintenance activities had started.

We aimed to see if the 59Hz coupling to DARM peak changed (needs ~20 minutes+ of data), but the accelerators showed that by increasing the CO2 power dumped in the water cooled beam dump, the load on the chillers changed and the 59Hz peak moved ~0.1Hz which means that it would be very difficult to see in DARM.

• 15:41 Turned CO2X and CO2Y power output down to 0W.
• 15:51 closed iris at bottom of CO2Y periscope (there is no place in VP for guillotine)
• 16:01 onwards, Robert was doing backscatter tests on HAM3, making IFO glitches/noise
• 1624 Opened CO2Y irirs, IFO lost lost lock

In attached plot, can see that the SQZ (especially at high frequency) got worse without CO2s. The SQZ ASC also changed the alignment mainly in YAW, does that mean that the CO2s need some alignment improvements in YAW? Looking at HWS signals, the optics substrate absorption changes as expected and started to level out, although the simulation didn't expect them too level out yet.

In future, to repeat this test, we could dump to 1.7W going into DARM using a normal beam-dump to dump the 1.7W before the periscope so that the load on the chiller isn't changed but the CO2 is not on being injected and anything that could backscatter light after the periscope is blocked. Alternatively, we could shake the table. Still the VPs themselves could be a source of scatter as are only AR coated at 10.6um, meaning they reflect up to 15% of 1064nm per surface DCC D1100439.

TITLE: 12/10 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 6mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

• We've been locked for 13:22, we dropped observing at 15:25 for the sus charge and from 15:01 to 15:20 for the magnetic
• The magentic injections have run and the SUS charge are currently running

Workstations were upgraded and rebooted.  This was an OS packages upgraded.  Conda packages were not upgraded.

TITLE: 12/10 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Currently Observing at 158 Mpc and have been Locked for 4 hours. The range is okay now, but we did go through a patch of low range that looks very similar to the low range from the last lock. For both this lock and  the previous lock it looks like the low range started around the same time after locking, 1 hour and 20 minutes after the lock started(ndscope). I couldn't find anything with squeeze or the jitter diaggui.
LOG:

00:26 Running an initial alignment
01:26 Initial alignment done, relocking

02:06 NOMINAL_LOW_NOISE    
02:09 Observing  

TITLE: 12/10 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is LOCKING and in ENVIRONMENT due to EARTHQUAKE.

We had a few good hours of locking today but as with yesterday, earthquakes have been rampant. Here are details:

• Lockloss 1: EQ caused and interrupted comissioning.
  • Tripped ISI HAM7 WD after LL (during high motion)
  • alog 81694
• Lock Re-acquisition:
  • FSS glitchy during initial alignment, resulting in DOWN until FSS relocked and I was able to re-do the initial alignment. After that, locking was fully auto.
  • screenshot attached
• Lockloss 2: EQ caused
  • Tripped many WDs
  • alog 81709
• SQZ Recovery following interlock trip
  • HAM7 interlock tripped, turning off High Voltage and unlocking SQZ. Fil turned it on in the morning and we were able to lock SQZ again. Cause of trip yet unknown but VAC knows about it - there was no immediate culprit or pressure spike when the trip happened.
  • SQZ was successfully brought back using Ryan's alog 79887
  • relevant alogs: alog 81689, alog 81697
  • SDF diffs attached
• Range dips hour into observing
  • We've been seeing our range dip down by ~5-10MPc during observing and have been trying to troubleshoot. This is not a new occurrence.
  • Went into comissioning for 10 mins to turn SQZ off (22:36 - 22:46 UTC) to see the effect on range - we saw no positive range change but seemed to stabilize the noise of the plot.
  • Camilla and TJ turned off CO2 to see if this was a potential cause but they saw no positive range change. alog 81705
  • Investigation ongoing.
• FMCSSTAT Red on CDS Overview
  • Dave noticed FMCSTAT going red on CDS overview and found that the Zone2A LVEA temps were going up very quickly. He informed FAC. Alog 81701
  • Tyler confirmed that it was not an issue, just that the temperature differential today was high and that the Zone 2A thermometer is in the wall, making it more susceptible to such excursions.

LOG:

Camilla C, TJ S

This morning we had another period where our range was fluctuating almost 40Mpc, previously seen on Dec 1 (alog81587) and further back in May (alog78089). Camilla and decided to turn off both TCS CO2s for a short period just to completely rule them out, since previously there was correlation with these range dips and a TCS ISS channel. We saw no positive change in DARM during this short duration test, but we didn't want to go too long and lose lock. CO2s were requested to have no output power from 16:12:30-16:14:30UTC

The past times that we have seen this range loss, the H1:TCS-ITMX_CO2_ISS_CTRL2_OUT_DQ channel and the H1:SQZ-FC_LSC_DOF2_OUT_DQ channel had noise that correlated to the loss, but theISS channel showed nothing different this time (attachment 2). We also were in a state of no squeezing at the time. So it's possible that this is a completely different type of range loss.

DetChar, could we run Lasso or check on HVETO for a period during the morning lock with our noisy range?

Using the darm_integral_compare.py script from the NoiseBudget repo (NoiseBudget/aligoNB/production_code/H1/darm_integral_compare.py) as a starting point, I made a version that is simplified and easy to run for when our range is low and we want to compare range vs frequency with a previous time.

It takes two starting times, supplied by the user, and for each time, it grabs the DARM data between the start time and an end time of starttime+5400 seconds (1.5 hours). Using this data it calculates the inspiral range integrand and returns two plots(pdf) - one showing the range integrand plotted against frequency for each set of data(png1), and then the second plot just shows DARM for each set of data, along with a trace showing the cumulative difference in range between the two sets as a function of frequency(png2). These are saved both as pngs and in a pdf in the script's folder.

This script can be found at gitcommon/ops_tools/rangeComparison/range_compare.py. To run it you just need to supply the gps times for the two sets of time that you want to compare, although there is also an optional argument you can if you want the length of data taken to be different than the default 5400 seconds. The command used to generate the PDF and PNGs attached to this alog was as follows: python3 range_compare.py --span 5000