After discussion with Tony we decided not to install his new PCAL Guardian nodes today, which means no DAQ restart today (my VACSTAT ini change is target-of-opportunity if a DAQ restart were to happen for other reasons)

Struggling to hold the ARMs from FSS oscillations and the wind knocking it out. The wind isn't predicted to calm down till 15:00 UTC and only for an hour before they rise even higher than they are currently from windy.com. The 2ndary microseism isn't helping either. 

As of 10:30 UTC I can hold the arms but the FSS keeps oscillating, holding it in DOWN till it passes ( It only took a few minutes)

10:47 ENGAGE_ASC_FOR_FULL_IFO lockloss

11:07 UTC RESONANCE lockloss from DRMI losing it

11:23 UTC RESONANCE lockloss again from DRMI

11:28 UTC FSS keeps oscillating

11:46 PREP_ASC lockloss from the IMC

12:00 UTC ENGAGE_ASC lockloss

The 3 minute average for the wind is just over 20 mph with gusts over 25, that combined with the secondary microseism fully above the 90th percentile relocking this morning is unlikely.

12:35 UTC LOWNOISE_COIL_DRIVERS lockloss

Jason, Ryan, Ibrahim, Elenna, Daniel, Vicky - This test had only PSL + PMC for ~2 hours, from 2024/11/11 23:47:45 UTC to 02:06:14 UTC. No FSS, ISS, or IMC.

Mixer glitch#1 @ 1415406517 - 1st PSL PMC mixer REFL glitches. Again, only PSL + PMC. No FSS, ISS, IMC.

The squeezer was running TTFSS, which locks the squeezer laser frequency to the PSL laser frequency + 160 MHz offset. With SQZ TTFSS running, for glitch #1, the squeezer witnessed glitches in SQZ TTFSS FIBR MIXER, and PMC and SHG demod error signals.

This seems to suggest the squeezer is following real PSL free-running laser frequency glitches? Since there is no PSL FSS servo actuating on the PSL laser frequency.
  -  also suggests the 35 MHz PSL (+SQZ) PMC LO VCO is not the main issue, since SQZ witnesses the PSL glitches in the SQZ FIBR MIXER.
  -  also suggests PMC PZT HV is not the issue. Without PSL FSS, any PMC PZT HV glitches should not become PSL laser frequency glitches. Caveat the cabling was not disconnected, just done from control room, so analog glitches could still propagate.

Mixer glitch #2 @ 1415408786. Trends here.

Somehow looks pretty different than glitch #1. PSL-PMC_MIXER glitches are not clearly correlated with NPRO power changes. SQZ-FIBR_MIXER sees the glitches, and SQZ-PMC_REFL_RF35 also sees glitches. But notably the SHG_RF24 does NOT see the glitches, unlike before in glitch #1.

For the crazy glitches at ~12 minutes (end of scope) - the SQZ TTFSS MIXER + PMC + SHG all see the big glitch, and there seem to be some (weak) NPRO power glitches too.

Mixer glitches #3 @ 1415409592 - Trends. Ryan and I are wondering if these are the types of glitches that bring the IMC down? But, checking the earlier IMC lockloss (tony 81199), can't tell from trends.

Here - huge PSL freq glitches that don't obviously correlate with PSL NPRO power changes (though maybe a slight step after the first round of glitches?). But these PSL glitches are clearly observed across the squeezer TTFSS + PMC + SHG signals (literally everywhere).

The scope I'm using is at /ligo/home/victoriaa.xu/ndscope/PSL/psl_sqz_glitches.yaml (sorry it runs very slow).

Thinking about the overall at tests done today, see annotated trends.

• PSL + PMC + FSS + IMC  (no ISS) ...  bad, IMC lockloss after 1:15 hours even with ISS OFF, 81198. IMC lockloss at 2024/11/11 20:37:44 UTC (1415392682).
   
• PSL + PMC  (no FSS, no ISS, no IMC), Test #1...  good? No glitches for 30 minutes. Sheila 81200.
   
• PSL + PMC + FSS  (no ISS, no IMC) ...  bad, Ref cav did not stay locked. Many glitches (visible in PMC mixer). Sheila 81200.
   
• PSL + PMC  (no FSS, no ISS, no IMC), Test #2 ...  bad this time, Tried again as Sheila suggested. After ~40 min, saw glitch #1 in PMC mixer, in turn witnessed by SQZ TTFSS. Bigger glitch #3 seen after ~1.5 hours (this thread).
  • Some glitches have NPRO power glitches, some don't.
  • The fact that squeezer TTFSS sees glitches could suggest these glitches are real and related to free-running PSL laser frequency glitches?
  • In particular - Glitch #3 (above) has similar peak-to-peak on PSL-PMC_MIXER to what unlocked the IMC in the ISS_OFF test earlier today. Glitch #3 also goes on for longer than #1,2.

Here is a look at what these locklosses actually look like.

I trended the various ETMX, ETMY and ITMX suspension channels during these transitions. The first two attachments here show a side by side set of scopes, with the left showing a successful transition from Nov 10, and the right showing a failed transition from Nov 9. It appears that in both cases, the ITMX L3 drive has a ~1 Hz oscillation that grows in magnitude until the transition. Both the good and bad times show a separate oscillation right at the transition. This occurs right at the end of state 557, so the locklosses within 1 or 2 seconds of state 558 likely are resulting from whatever the of the steps in state 557 do. The second screenshot zooms in to highlight that the successful transition has a different oscillation that rings down, whereas the unsuccessful transition fails right where this second ring up occurs.

I grabbed a quick trend of the microseism, and it looks like the ground motion approximately doubled around Sept 20 (third screenshot). I grabbed a couple of recent successful and unsuccessful transitions since Sept 20, and they all show similar behavior. A successful transition from Sept 19 (fourth attachment) does not show the first 1 Hz ring up, just the second fast ring down after the transition.

I tried to look back at a time before the DAC change at ETMX, but I am having trouble getting any data that isn't minute trend. I will keep trying for earlier times, but this already indicates an instability in this transition that our long ramp times are not avoiding.

Thanks to some help from Erik and Jonathan, I was able to trend raw data from Nov 2023 (hint: use nds2). Around Nov 8 2023, the microseism was elevated, ~ 300 counts average on the H1:ISI-GND_STS_ITMY_Z_BLRMS_100M_300M channel, compared to ~400 counts average this weekend. The attached scope compares the transition in Nov 8 2023 (left) versus this weekend Nov 10 (right). One major difference here is that we now have a new DARM offloading scheme. A year ago, it appears that this transition also involves some instability causing some oscillation in ETMX L3, but the instability now creates a larger disturbance that rings down in both the ITMX L3 and ETMX L3 channels.

Final thoughts for today:

• the lockloss occurs within 1-2 seconds of when the ramp down of IX L3 and the ramp up of EX L3 occurs, so it seems like the issue here is within the L3 control
• we have had instability issues with this transition back in March when we were commissioning it, which did include a 1 Hz instability
• The growing oscillation before the transition is about 1 Hz, whereas the larger oscillation that can coincide with the lockloss is around 3 Hz.
• Although the microseism is higher, this overall RMS on each suspension stage hasn't changed appreciably since April (see attachment). However, this plot compares NLN times. It's seems like the switchover itself is unstable, which is maybe too fast to compare
• The switchover is between IX/EY control which appears to be on the "old" configuration and the EX "new" configuration, so even if each individual configuration is stable, maybe a combo of them is unstable.

Sheila and I looked a look at these plots again. It appears that the 3 Hz oscillation has the potential to saturate L3, which might be causing the lockloss. The ringup repeatedly occurs within the first two seconds of the gain ramp, which is set to a 10 second ramp. We decided to shorten this ramp to 2 seconds. I created a new variable on line 5377 in the ISC guardian called "etmx_ramp" and set that to 2. The ramps from the EY/IX to EX L3 control are now set to this value, as well as the timer. If this is bad, this ramp variable can be changed back.

Trend shows that the ITMY HWS code stopped running. I restarted it.

Erik, Camilla. We've been seeing that the code running on h1hwsmsr1 (ITMY) kept stopping after ~1hours with a "Fatal IO error 25" (Erik said related to a display) attached.

We checked that memory is fine of h1hwsmsr1. Erik troubleshooted this back to matplotlib trying to make a plot and failing as there was no display to make the plot on. State3.py calls get_wf_new_center() from hws_gradtools.py which calls get_extrema_from_gradients() which makes a contour plot, it's trying to make this plot and thinks there's a display but then can't plot it.  This error isn't happening on h1hwsmsr (ITMX).   I had ssh'ed into h1hwsmsr1 using -Y -C options (allowing the stream image to show), but Erik found this was making the session think there was a display when there wasn't.

Fix: quitting tmux session, logging in without options (ssh controls@h1hwsmsr1),  and starting code again. The code has now been running fine for the last 18 hours.

I discovered today that LLO is indeed doing some additional digital filtering to their FSS_FASTMON channel, which would very likely explain the difference in spectra Marc shows above. Just looking at the MEDM screen for the filter bank, it shows three filters in use (called "cts2V', "NPRO", and "toMHz") while LHO is using none; parameters of these are attached in a screenshot. I'm not entirely sure what the purpose of these are, but from what I can tell there is an additional pole at 10Hz, which would explain the 1/f-looking drop in noise towards higher frequencies.