Back to observing at 23:12 UTC

[Dave,  Jonathan, Erik, Oli]

GDS  TP screen showed ADC 4 with the timeout.

Some cards were missing from lspci,  We shut the front end off powered off the AI chasses and cycled power on the IO chassis.  After bringing the front end back, all cards appeared normal, so we turned the AI chasses back on and resumed operation.

I will prep a spare ADC in case ADC 4 permenantly fails, but ADC timeouts have signaled DAC or Adnaco failure as well.

I opened (and then marked as Resolved) an FRS ticket: FRS#29998

Looks like we have an issue with the corner station seismometer having given out and caused the 'local earthquake' and lockloss(ndscope, seismon FOM, peakmon/CPSFF).

03:35UTC Back Observing

Is this additonal nosie at low frequency due to the same non-stationarity we oberved before and we believe is related to the ESD upconversion? Probably not, here's why.

First plot compares the strain spectrum from two times with cold and hot OM2. This confirms Sheila's observation.

The second and third plots are spectrograms of GDS-CALIB_STRAIN during the two periods. Both show non-stationry noise at low frequency. The third plot shows the strain spectrogram normalized to the median of the hot OM2 data: beside the non-stationariity, it looks like the background noise is higher below 30 Hz.

This is confirmed by looking at the BLRMS in the 16-60 Hz region for the two times, as shown in the fourth plot: its higher with cold OM2

Finally, the last plot shows the correlation between the ESD RMS and the strain BLRMS, normalized to the hot OM2 state. There is still a correlation, but it appear again that the cold OM2 state has an additional background noise: when the ESD RMS is att the lower end, the strain BLRMS setlles to higher values

Here is the same comparison, without squeezing.  Using times from  74935 and 74834

This suggests that where cold OM2 seems better than hot OM2 above that is due to the squeezing (and the jitter subtraction Jenne added, which is also on in this plot for cold OM2 but not for hot OM2).  And the additional noise with cold OM2 reaches up to about 45Hz. 

After we optimized ADF demod phase in 74972, the BNS range seems better and consistently 160-165Mpc. The attached plot shows the comparison of OM2 cold/hot with/without SQZ. The OM2 cold with SQZ is measured after optimization of ADF demod phase and other measurements are same as Sheila's previous plots.

This plot supports what Sheila says in the previous alogs.

• The OM2 cold is worse below 40 Hz for both SQZ/no SQZ.  
• Without SQZ, OM2 cold and hot are almost the same above 40 Hz.
• With SQZ, OM2 cold is better between 100-600 Hz, but worse above 1 kHz. This difference could be due to SQZ and we could try to optimize SQZ around 100 Hz with OM2 hot

Echo-ing the above, and summarizing a look at OM2 with sqz in both Sept 2023 and Dec 2023 (running gps times dictionary is attached here).

If we compare the effect of squeezing -- there is higher kHz squeezing efficiency with hot OM2. We can look at either just the darm residuals dB[sqz/unsqz] (top), or do subtraction of non-quantum noise (bottom) which shows that hot OM2 improved the kHz squeezing level by ~0.5 dB at 1.7 kHz (the blue sqz blrms 5). This is consistent with summary pages: SQZ has not reached 4.5 dB since cooling OM2 74861. Possibly suggests better SQZ-OMC mode-matching with hot OM2.

Without squeezing, cold om2 has more optical gain and more low-freq non-quantum noise. Better IFO-OMC mode-matching with cold OM2.

In total, it's almost a wash for kHz sensitivity: heating OM2 loses a few % optical gain, but recovers 0.2-0.5 dB of shot noise squeezing. 

It's worth noting the consistent range increases with SQZ tuning + improvements: even in FDS, there is a non-zero contribution of quantum noise down to almost 50 Hz. For example Naoki's adjustment of sqz angle setpoint on 12/21 74972 improved range, same for Camilla's Jan sqz tuning 75151. Looking at DARM (bottom green/purple traces), these sqz angle tunings reproducibly improved quantum noise between about 60-450 Hz.

Here are some more plots of the times that Vicky plotted above. 

The first attachment is just a DARM comparison with all 4 no sqz times, OM2 cold vs hot in December vs September. 

Comparing OM2 hot September vs December shows that our sensitivity at from 20-40 Hz has gotten worse since September, the MICH coherence seems lower while the jitter and SRCL coherence seem similar.  The same comparison for OM2 cold shows that with OM2 cold our sensitivity has also gotten worse from 15-30 Hz. 

Comparing cold vs hot, in September the MICH coherence did get worse from 60-80 Hz for cold OM2, which might explain the worse sensitivity in that region.  The MICH coherence got better from 20-30 Hz where the sensitivty was better for cold OM2.  The December test had better tuned MICH FF for both hot and cold OM2, so this is the better test of the impact of the curvature change. 

As Gabriele pointed out with his BRUCO, 74886 there is extra coherence with DHARD Y for cold OM2 at the right frequencies to help explain the extra noise.  There isn't much change in the HARD pitch coherence between these December times, but the last attachment here shows a comparison of the HARD Y coherences for hot and cold OM2 in December. 

Peter asked if the difference in coherence with the HARD Yaw ASC was due to a change in the coupling or the control signal. 

Here is a comparison of the control signals with OM2 hot and cold, they look very similar at the frequencies of the coherence.

During this timeline I also cleared filter history on L2_LOCK_L (very high counts before clearing) and M0_DAMP_L (no difference after clearing) details in 74889.

The channels used for the calibration measurement injections are listed in LHO:74919.

Dave, Rahul

We lost lock this afternoon and I took this opportunity to quickly implement the R0 watchdog changes. The new thresholds are given below,

ITMX R0 chain WD rms threshold - 20k counts

ITMY R0 chain WD rms threshold - 20k counts

ETMX R0 chain WD rms threshold - 18k counts

ETMY R0 chain WD rms threshold - 18k counts

I have accepted the above changes in the SDF and posted the screenshot below.

The threshold limit for ETMs is lower than that of ITMs based on the ndscope trends for the last 30days. The safety limit for ITMs seems to be around 20k and for ETMs 18k.

A more long term safety fix will be implemented in January 2024 by making some model changes.

WP 11587 Closed.

Nice work Camilla and Ansel! Let's hope this solves the problem for good.

At 17:38UTC I tunred on the camera/CLink and restarted the HWS EX code which had been off since 74738.

At 17:46UTC I changed ITMX and ITMY to 'sem 3' and left them in this configuration so that they are not taking HWS data now. Now we need to write a script to read H1:GRD-ISC_LOCK_STATE_N state and adjust the camera mode to be sem 2 when we are in states < 580 (locking) and sem 3 when > 580 (locked). This is not trivial as camera computers are separate from EPICS.

When the excitations started, the ETMX HEPI, ISI and SUS WDs tripped and (~5 minutes later) the hardware watchdogs for both SEI and SUS. 

With all watchdogs tripped, I cleared history of the L2 LOCK L stage of ETMX as this was outputting very large numbers. M0 DAMP also seemed to be outputting very large numbers, I tried cleaning history but it seems like this was physically ust moving a LOT, plot attached.

Rahul via phone walked me though checking that ETMX is ready to be untripped: output counts changing by hundreds not 10,000's. We then reset SUS SW WD (cds > SWWD > ETMX) and untripped the SUS ETMX WD, Rahul said if it was still ringing very badly we would turn on M0 DAMP damping loops one-by-one but they already looked on so I took the GRD to DAMPED then ALIGNED. 

After we got ETMX SUS back to usual, I reset the ETMX SEI SWWD and I followed the procedure to uptrip SEI (from wiki, HEPI then ISI). Once this was done Ryan took us to maintenance mode. 

The new DARM configuration is shown in this plot is veery effective in reducing the ESD RMS, as shown in the plot below. The blue curve is one hour of normal configuration, showing the ESD total RMS as a function of time. The orange trace is during the test: the first few hundreds of seconds are still the nominal configuration, while between ~1000 and ~2000 seconds the new DARM configuration is running. The RMS is reduced well below the minimum of the normal configuration.

Unfortunately the was no clean data collected during this time to see the effect on the strain.

The MICHFF FM4 that we fit for cold OM2 74877 was actually very good for tis new DARM configuration green trace, much better than the FM1 we fit for the new DARM configuration 74817.

DARM didn't look great at 30-100Hz in either configuration, maybe this could be as the calibration is wrong.

The simulines measurements injected into the following channels: 


H1:LSC-DARM1_EXC
H1:CAL-PCALY_SWEPT_SINE_EXC
H1:SUS-ETMX_L1_CAL_EXC
H1:SUS-ETMX_L2_CAL_EXC
H1:SUS-ETMX_L3_CAL_EXC


Injections began at approximately GPS 1387035241.