Jason, Ryan, Ibrahim, Elenna, Daniel, Vicky - This test had only PSL + PMC for ~2 hours, from 2024/11/11 23:47:45 UTC to 02:06:14 UTC. No FSS, ISS, or IMC.

Mixer glitch#1 @ 1415406517 - 1st PSL PMC mixer REFL glitches. Again, only PSL + PMC. No FSS, ISS, IMC.

The squeezer was running TTFSS, which locks the squeezer laser frequency to the PSL laser frequency + 160 MHz offset. With SQZ TTFSS running, for glitch #1, the squeezer witnessed glitches in SQZ TTFSS FIBR MIXER, and PMC and SHG demod error signals.

This seems to suggest the squeezer is following real PSL free-running laser frequency glitches? Since there is no PSL FSS servo actuating on the PSL laser frequency.
  -  also suggests the 35 MHz PSL (+SQZ) PMC LO VCO is not the main issue, since SQZ witnesses the PSL glitches in the SQZ FIBR MIXER.
  -  also suggests PMC PZT HV is not the issue. Without PSL FSS, any PMC PZT HV glitches should not become PSL laser frequency glitches. Caveat the cabling was not disconnected, just done from control room, so analog glitches could still propagate.

Mixer glitch #2 @ 1415408786. Trends here.

Somehow looks pretty different than glitch #1. PSL-PMC_MIXER glitches are not clearly correlated with NPRO power changes. SQZ-FIBR_MIXER sees the glitches, and SQZ-PMC_REFL_RF35 also sees glitches. But notably the SHG_RF24 does NOT see the glitches, unlike before in glitch #1.

For the crazy glitches at ~12 minutes (end of scope) - the SQZ TTFSS MIXER + PMC + SHG all see the big glitch, and there seem to be some (weak) NPRO power glitches too.

Mixer glitches #3 @ 1415409592 - Trends. Ryan and I are wondering if these are the types of glitches that bring the IMC down? But, checking the earlier IMC lockloss (tony 81199), can't tell from trends.

Here - huge PSL freq glitches that don't obviously correlate with PSL NPRO power changes (though maybe a slight step after the first round of glitches?). But these PSL glitches are clearly observed across the squeezer TTFSS + PMC + SHG signals (literally everywhere).

The scope I'm using is at /ligo/home/victoriaa.xu/ndscope/PSL/psl_sqz_glitches.yaml (sorry it runs very slow).

Thinking about the overall at tests done today, see annotated trends.

• PSL + PMC + FSS + IMC  (no ISS) ...  bad, IMC lockloss after 1:15 hours even with ISS OFF, 81198. IMC lockloss at 2024/11/11 20:37:44 UTC (1415392682).
   
• PSL + PMC  (no FSS, no ISS, no IMC), Test #1...  good? No glitches for 30 minutes. Sheila 81200.
   
• PSL + PMC + FSS  (no ISS, no IMC) ...  bad, Ref cav did not stay locked. Many glitches (visible in PMC mixer). Sheila 81200.
   
• PSL + PMC  (no FSS, no ISS, no IMC), Test #2 ...  bad this time, Tried again as Sheila suggested. After ~40 min, saw glitch #1 in PMC mixer, in turn witnessed by SQZ TTFSS. Bigger glitch #3 seen after ~1.5 hours (this thread).
  • Some glitches have NPRO power glitches, some don't.
  • The fact that squeezer TTFSS sees glitches could suggest these glitches are real and related to free-running PSL laser frequency glitches?
  • In particular - Glitch #3 (above) has similar peak-to-peak on PSL-PMC_MIXER to what unlocked the IMC in the ISS_OFF test earlier today. Glitch #3 also goes on for longer than #1,2.

IMC Lockloss @ 12:37:44 UTC even with the ISS turned off.

22:05:49 UTC the FSS Autolocker H1:PSL-FSS_AUTOLOCK_ON was turned off .

Here is a look at what these locklosses actually look like.

I trended the various ETMX, ETMY and ITMX suspension channels during these transitions. The first two attachments here show a side by side set of scopes, with the left showing a successful transition from Nov 10, and the right showing a failed transition from Nov 9. It appears that in both cases, the ITMX L3 drive has a ~1 Hz oscillation that grows in magnitude until the transition. Both the good and bad times show a separate oscillation right at the transition. This occurs right at the end of state 557, so the locklosses within 1 or 2 seconds of state 558 likely are resulting from whatever the of the steps in state 557 do. The second screenshot zooms in to highlight that the successful transition has a different oscillation that rings down, whereas the unsuccessful transition fails right where this second ring up occurs.

I grabbed a quick trend of the microseism, and it looks like the ground motion approximately doubled around Sept 20 (third screenshot). I grabbed a couple of recent successful and unsuccessful transitions since Sept 20, and they all show similar behavior. A successful transition from Sept 19 (fourth attachment) does not show the first 1 Hz ring up, just the second fast ring down after the transition.

I tried to look back at a time before the DAC change at ETMX, but I am having trouble getting any data that isn't minute trend. I will keep trying for earlier times, but this already indicates an instability in this transition that our long ramp times are not avoiding.

Thanks to some help from Erik and Jonathan, I was able to trend raw data from Nov 2023 (hint: use nds2). Around Nov 8 2023, the microseism was elevated, ~ 300 counts average on the H1:ISI-GND_STS_ITMY_Z_BLRMS_100M_300M channel, compared to ~400 counts average this weekend. The attached scope compares the transition in Nov 8 2023 (left) versus this weekend Nov 10 (right). One major difference here is that we now have a new DARM offloading scheme. A year ago, it appears that this transition also involves some instability causing some oscillation in ETMX L3, but the instability now creates a larger disturbance that rings down in both the ITMX L3 and ETMX L3 channels.

Final thoughts for today:

• the lockloss occurs within 1-2 seconds of when the ramp down of IX L3 and the ramp up of EX L3 occurs, so it seems like the issue here is within the L3 control
• we have had instability issues with this transition back in March when we were commissioning it, which did include a 1 Hz instability
• The growing oscillation before the transition is about 1 Hz, whereas the larger oscillation that can coincide with the lockloss is around 3 Hz.
• Although the microseism is higher, this overall RMS on each suspension stage hasn't changed appreciably since April (see attachment). However, this plot compares NLN times. It's seems like the switchover itself is unstable, which is maybe too fast to compare
• The switchover is between IX/EY control which appears to be on the "old" configuration and the EX "new" configuration, so even if each individual configuration is stable, maybe a combo of them is unstable.

Sheila and I looked a look at these plots again. It appears that the 3 Hz oscillation has the potential to saturate L3, which might be causing the lockloss. The ringup repeatedly occurs within the first two seconds of the gain ramp, which is set to a 10 second ramp. We decided to shorten this ramp to 2 seconds. I created a new variable on line 5377 in the ISC guardian called "etmx_ramp" and set that to 2. The ramps from the EY/IX to EX L3 control are now set to this value, as well as the timer. If this is bad, this ramp variable can be changed back.

• November 12th:
  • Took TFs 81247
  • Tightened loose ISS AOM RF cable 81247
  • Tested and characterized in-service TTFSS, SN LHO01: tuned notches and replaced a bad PA85 op amp 81247
• November 13th:
  • IMC stayed locked without IFO 81262
  • FSS OLG and gain change 81254
  • IMC OLG and gain change 81259. 
• Novmeber 14th:
  • PSL Beckhoff power cycle 81281
  • Replace 35MHz with Marconi: 81277
  • Tightened loose ISS AOM cable: 81280
• Novmeber 18th:
  • LSC POP un-clipped 81329
  • Issues found with IMC locked checker in ISC_LOCK's READY 81336 (maybe keeping us in DOWN longer than needed)

Updated list of things that have been checked above and attache a plot where I've split the IMC only tagged locklosses (orange) from those tagged IMC and FSS_OSCIALTION (yellow). The non IMC ones (blue) are the normal lock losses and (mostly) only once we saw before September. 

• Novmeber 14th:
  • PMC OLG measured 81283
• November 19th:
  • Scopes setup at PSL racks 81359
  • PMC+FSS only test shows glitches 81356
  • Reverted EX 28bit LIGO-DAC change back to 20-bit 81350

Commit 464b5256 in aligoNB git repo.

These results show that above 3 kHz, some of the intensity noise couples through frequency noise. There is no measurable intensity noise coupling through frequency noise at mid or low frequency.

The frequency noise projection and coupling in this plot are divided by rt2 to account for the fact that frequency noise is measured on one sensor here, but usually is controlled on two REFL sensors. This may not be correct for high frequency (roughly above 4 kHz), where CARM is gain limited.

code lives in /ligo/gitcommon/NoiseBudget/simplepyNB/Laser_noise_projections.ipynb

lockloss at 10:07 UTC from LOWNOISE_ESD_ETMX, doesn't look like an IMC lockloss

11:03 observing

This lockloss seems to have the AOM driver monitor glitching right before the lockloss(ndscope), similar to what I had noticed in the two toher locklosses from this past weekend(81037).

03:45UTC Observing

In 81073 (Tuesday 05 November) Jason/Ryan S gave the ISS more range to stop it running out of range and going unstable. But on Tuesday evening, Oli saw this type of lockloss with again 81089. Not sure if we've seen it since then. 

The channel to check in the ~second before the lockloss is H1:PSL-ISS_AOM_DRIVER_MON_OUT_DQ, I added this to the lockloss trends shown by the 'lockloss' command line tool via /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/psl_fss_imc.yaml

I checked all the "IMC" tagged locklosses since Wednesday 6th and didn't see any more of these.

This happened before we fixed the NPRO mode hopping problem, which we did on Wednesday, Nov 6th.  Not seeing any more of these locklosses since lends credence to our suspicion that the ISS was not responsible for these locklosses, the NPRO mode hopping was (NPRO mode hops cause the power output from the PMC to drop, the ISS sees this drop and does its job by lowering the diffracted power %, once the diffracted power % hits 0 the ISS unlocks and/or goes unstable).

Attaching plots from zooming in on a few locklosses:

The 10-31 zoom plot notes the framerates of the channels: ASC, REFL and NPRO_PWR are 2kHz and GS13 is 4kHz, the others are 16kHz.

• In the "NORMAL" lockloss (nearly 100% of locklosses since O4a and O4b before OFI vent):
  • The arm signals (ASC_AS_A/C) increase over 10's ms as the REFL signal decreases.
  • ~250ms later the PSL channels change and the IMC looses lock. 
• In the "IMC" lockloss (25% of locklosses since OFI vent):
  • The arm signals (ASC_AS_A/C) suddenly (<0.5ms) decrease as the REFL signal increases.
  • At the same time the PSL signals change.
  • <10 ms later the IMC looses lock.

Since September 18th we've had 21 locklosses from NLN tagged as FSS_OSCILLATION, of these 20 also had the IMC tag.   Since September 12th, we've had 49 locklosses from NLN tagged IMC, so roughly 40% of these IMC locklosses have the FSS oscillation tag, since the NPRO was swapped we don't have any tagged with FSS_OSCILLATION. 

(Reminder, the FSS_OSCILLATION tag is an old tag, intended for a different problem, but it tags times where the FSS fast mon goes above a treshold.)

Updated plot attached of NLN locklosses tagged with and without the IMC tag. 

Checked when we turned the CO2s up for 7 minutes, that this had no noticeable effect on squeezing level or sqz alignment, plot attached. But maybe 7 minutes isn't long enough to see any changes, we could try again with more time or try turning the CO2s off before Tuesday Maintenance

Checking as LLO sees the Squeezing and SQZ alignment/mode matching  may be related to changes in CO2 power (or alignment), see 72244.