A long sustained error sound was coming from the MSR.
Upon inspection, the H1FS0 server had one of it's power supplies fail.
Erik and I have replaced the power supply around 22:30 UTC.
The replacement PSU came from a server from the far left side of the test stand rack in the H2 building. The PSU was not being used in the test stand.
Jonathan, Erik, EJ, Dave:
A follow up on the unexpected IPC errors we saw sitewide after the power cycle of h1seib1 Tue 17.47 19th December 2023 PST (cycled because of an ADC issue).
Details can be found in FRS30025
When h1seib1 rejoined the dolphin network (i.e. when it re-enabled its IX dolphin switch port during its boot cycle) most, but not all, frontends with Dolphin and/or long-range Dolphin IPC receivers saw Rx errors (see attachment).
This error condition persisted for 64 seconds.
For some front ends the error rate was small (e.g. a dozen single errors spread over 64 seconds) and for others it was larger (over 100 errors, multiple per second (up to 8), continuous stretch of errors for 15 seconds)
Unfortunately the HAM ISI models were in the list of models which saw 15 seconds of continuous errors. This caused their Payload Watchdogs to trip (they trip on 10 seconds or more continuous IPC Rx errors) (see attachment)
Currently the only event we have detected which happened at the time the IPC errors stopped was when the h1isiitmy model got going on h1seib1, at which point it would have started writing to its dolphin card.
STATE of H1: Observing at 155Mpc
We've been locked and observing the whole shift, we survived a 5.3 from the Dominican Republic earlier.
In advance of the Holiday break, I topped off the water level for the PSL chiller. I added ~150mL of water to hit the MAX line.
(Filiberto C., Gerardo M.)
The motor for the aftercooler was removed and replaced (see aLOG 74833), the fan motor for the aftercooler was tested and is fully functional. Next maintenance period the purge air system will be tested.
Closed FRS ticket 29978.
Fri Dec 22 10:13:19 2023 INFO: Fill completed in 13min 15secs
Gerardo confirmed a good fill curbside. TC mins -99C,-92C, trip is -70C
All looks good
Closes FAMIS 26272
The only interesting item was MY_FAN1_270_{1,2} is seeing some big spikes 1.0< yesterday and today (only really fan2 today).
Closes FAMIS 26223
Laser Status:
NPRO output power is 1.829W (nominal ~2W)
AMP1 output power is 67.86W (nominal ~70W)
AMP2 output power is 136.9W (nominal 135-140W)
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PMC:
It has been locked 15 days, 20 hr 0 minutes
Reflected power = 16.6W
Transmitted power = 109.8W
PowerSum = 126.4W
FSS:
It has been locked for 0 days 5 hr and 2 min
TPD[V] = 0.9381V
ISS:
The diffracted power is around 2.0%
Last saturation event was 0 days 5 hours and 3 minutes ago
Possible Issues: None
No clear reason, https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1387278013
TITLE: 12/22 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 8mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY:
TITLE: 12/22 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY: In Observing and have been Locked for 4.5 hours. One commissioning-caused lockloss during my shift(74976), but relocking went normally.
LOG:
00:00 Detector Commissioning and Locked for 4hours 45mins
00:16:32 Started broadband calibration measurement
00:22:46 Finished broadband calibration measurement
01:24 Lockloss caused by commissioning activities
-- Start Relocking --
02:10 Took us to DOWN to start an initial alignment due to us cycling through CHECK_MICH_FRINGES
03:23 Reached NOMINAL_LOW_NOISE
-- End Relocking --
03:27 Into Observing
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 00:42 | VAC | Gerardo | EX | n | Working on fan | 01:26 |
Currently Observing at 156Mpc and have been Locked for 1.5 hours. Nothing to note.
Jenne, Ryan, Oli, Keita, Louis We moved into theNEW_DARMguardian state once again with the following goals: 1.) update CAL-CS to match the new actuation path configuration to getDELTAL_EXTERNALto be roughly calibrated in the control room. 2.) restart the GDS pipeline with a pyDARM parameter file and GDS FIR filter set that matches the new DARM configuration 3.) sit in the now calibrated state for 20minutes to compare DARM noise in the new state vs the nominal and hopefully show that the noise better. Previously, we'd only been able to show that the non-stationarity was better (LHO:74939). 3.b) start a calibration sweep using Simulines using the retuned injections that I prepared by scaling the nominal injection amplitudes by the changes we've made to the DRIVEALIGN filters. 4.) Test theRETURN_TO_NLN_ETMYstate that Sheila and I prepared to move fromNEW_DARMback toNLN_ETMY. 5.) Test theDARM_RECOVERstate that I prepared to move us back to ETMX in the nominal configuration from ETMY. In short, we were able to move toNEW_DARMbut we could not get DELTAL_EXTERNAL calibrated by CAL-CS to make sense. We were able to push a new configuration to the GDS pipeline but similarly could not get that look right either. We ran out of time to debug further (we were already over time before needing to go back to Observing) so we started back tracking to the nominal DARM state. Instead of trusting all of our moves back to nominal to the guardian state, we ran commands one at a time to try and track down why we've not been having success recently. Jenne caught a few steps missing fromRETURN_TO_NLN_ETMY, still be implemented: 1.) we need to clear the ETMY L2 DRIVEALIGN L2L histories 2.) we need to make sure to adjust the LSC Output Matrix accordingly when moving back to ETMY Even after fixing these two issues we promptly lost lock when swapping the L3 LOCK L gains between ETMX and ETMY. So there's something else we haven't yet caught holding us up there. -- additional information regarding what was done to temporarily update the GDS pipeline, the re-retuned Simulines injections, and general front-end/CAL-CS weirdness that we encountered will come. == I restarted the GDS pipeline with the nominal calibration state and returned the CAL-CS settings to their nominal configuration while we relock back into the nominal configuration. We will leave the IFO in the nominal configuration over the holidays.
Also, in case this helps with troubleshooting, note that when we lost lock between the ETMX -> ETMY gain swap, the ETMX HEPI, Stage 1, and Stage 2 watchdogs all tripped.
Lockloss @ 12/22 01:24UTC from commissioning activities. ETMX HEPI, stage 1, and stage 2 all tripped.
Back to Observing as of 03:27UTC
Naoki, Camilla
We scanned ADF demod phase with cold OM2 as we did with hot OM2 in 74691. The nominal ADF demod phase was 80 with hot OM2 and we tried to scan from 70, but the ADF servo was diverging with 70 demod phase, which means the ADF error signal is out of linear range. We found that the ADF demod phase around 100 would be good with cold OM2 so we scanned ADF demod phase from 90 to 110 using the same command in 74691 as shown in the attached figure. It seems that the ADF demod phase around 90 is optimal and we set the ADF demod phase to 91.
Jenne, Naoki, Louis, Camilla, Sheila
Here is comparison of the DARM CLEAN spectrum with OM2 hot vs cold. The second screenshot shows a time series of OM2 cooling off. The optical gain increased by 2%, as was seen in the past (for example 71087). Thermistor 1 shows that the thermal transient takes much longer (12 + hours) than what thermistor 2 says (2 hours).
Louis posted a comparison of the calibration between the two states, there are small differences in calibration ~1% (74913). While the DARM spectrum is worse below 25Hz, it is similar at 70 Hz where we in the past thought that the sensitivity was worse with OM2 cold. From 100-200 Hz the sensitivity seems slightly better with OM2 cold, some of the peaks are removed by Jenne's jitter subtraction (74879) but there also seems to be a lower level of noise between the peaks (which could be small enough to be a calibration issue). At high frequency the cold OM2 noise seems worse, this could be because of the squeezing. We plan to take data with some different squeezing angles tomorow and will check the squeezing angle as part of that.
So, it seems that this test gives us a different conculsion than the one we did in the spring/summer, and that now it seems that we should be able to run with OM2 cold to have better mode matching from the interferometer to the OMC. We may have not had our feedforwards well tuned in the previous test, or perhaps some other changes in the noise mean that the result is different now.
Is this additonal nosie at low frequency due to the same non-stationarity we oberved before and we believe is related to the ESD upconversion? Probably not, here's why.
First plot compares the strain spectrum from two times with cold and hot OM2. This confirms Sheila's observation.
The second and third plots are spectrograms of GDS-CALIB_STRAIN during the two periods. Both show non-stationry noise at low frequency. The third plot shows the strain spectrogram normalized to the median of the hot OM2 data: beside the non-stationariity, it looks like the background noise is higher below 30 Hz.
This is confirmed by looking at the BLRMS in the 16-60 Hz region for the two times, as shown in the fourth plot: its higher with cold OM2
Finally, the last plot shows the correlation between the ESD RMS and the strain BLRMS, normalized to the hot OM2 state. There is still a correlation, but it appear again that the cold OM2 state has an additional background noise: when the ESD RMS is att the lower end, the strain BLRMS setlles to higher values
Here is the same comparison, without squeezing. Using times from 74935 and 74834
This suggests that where cold OM2 seems better than hot OM2 above that is due to the squeezing (and the jitter subtraction Jenne added, which is also on in this plot for cold OM2 but not for hot OM2). And the additional noise with cold OM2 reaches up to about 45Hz.
After we optimized ADF demod phase in 74972, the BNS range seems better and consistently 160-165Mpc. The attached plot shows the comparison of OM2 cold/hot with/without SQZ. The OM2 cold with SQZ is measured after optimization of ADF demod phase and other measurements are same as Sheila's previous plots.
This plot supports what Sheila says in the previous alogs.
Echo-ing the above, and summarizing a look at OM2 with sqz in both Sept 2023 and Dec 2023 (running gps times dictionary is attached here).
If we compare the effect of squeezing -- there is higher kHz squeezing efficiency with hot OM2. We can look at either just the darm residuals dB[sqz/unsqz] (top), or do subtraction of non-quantum noise (bottom) which shows that hot OM2 improved the kHz squeezing level by ~0.5 dB at 1.7 kHz (the blue sqz blrms 5). This is consistent with summary pages: SQZ has not reached 4.5 dB since cooling OM2 74861. Possibly suggests better SQZ-OMC mode-matching with hot OM2.
Without squeezing, cold om2 has more optical gain and more low-freq non-quantum noise. Better IFO-OMC mode-matching with cold OM2.
In total, it's almost a wash for kHz sensitivity: heating OM2 loses a few % optical gain, but recovers 0.2-0.5 dB of shot noise squeezing.
It's worth noting the consistent range increases with SQZ tuning + improvements: even in FDS, there is a non-zero contribution of quantum noise down to almost 50 Hz. For example Naoki's adjustment of sqz angle setpoint on 12/21 74972 improved range, same for Camilla's Jan sqz tuning 75151. Looking at DARM (bottom green/purple traces), these sqz angle tunings reproducibly improved quantum noise between about 60-450 Hz.
Here are some more plots of the times that Vicky plotted above.
The first attachment is just a DARM comparison with all 4 no sqz times, OM2 cold vs hot in December vs September.
Comparing OM2 hot September vs December shows that our sensitivity at from 20-40 Hz has gotten worse since September, the MICH coherence seems lower while the jitter and SRCL coherence seem similar. The same comparison for OM2 cold shows that with OM2 cold our sensitivity has also gotten worse from 15-30 Hz.
Comparing cold vs hot, in September the MICH coherence did get worse from 60-80 Hz for cold OM2, which might explain the worse sensitivity in that region. The MICH coherence got better from 20-30 Hz where the sensitivty was better for cold OM2. The December test had better tuned MICH FF for both hot and cold OM2, so this is the better test of the impact of the curvature change.
As Gabriele pointed out with his BRUCO, 74886 there is extra coherence with DHARD Y for cold OM2 at the right frequencies to help explain the extra noise. There isn't much change in the HARD pitch coherence between these December times, but the last attachment here shows a comparison of the HARD Y coherences for hot and cold OM2 in December.
Peter asked if the difference in coherence with the HARD Yaw ASC was due to a change in the coupling or the control signal.
Here is a comparison of the control signals with OM2 hot and cold, they look very similar at the frequencies of the coherence.
Late entry, work done last tuesday. The corner station, filter cavity end station and the X-End station purge air systems were operated and tested, this work was done in preparation for the upcoming vents.
The corner station system was turned on and ran for 3 hours, then the dry air was sampled, the output at the drying towers produced a dew point of -41.3 deg C, a good reading. No other issues were noted with the compressor or the drying towers. The compressor (Kobelco) unit is to have a yearly maintenance done in the next couple of weeks.
The filter cavity end station system also ran for 3 hours, the dry air was tested, this system produced a dew point of -43.6 deg C. No real issues with the system were encountered, compressor and drying towers. However, the system requires power to run the drying tower, one automatic drain for the receiver tank, and a second automatic drain for the compressor all 115 AC power, we have an outlet with 4 dedicated plugs for such process, however the day of the test, 3 out of the 4 outlets were being used by other non vacuum systems.
The test at the X-End was terminated when the aftercooler fan failed to start, after 3 different tries. The system was powered off and troubleshooted with help from Filiberto, the fan motor was dead. A new fan motor for the aftercooler was ordered. FRS ticket opened, see 29978.
X-End issue resolved, see here 74989.
