Reports until 08:41, Sunday 02 February 2025
H1 SQZ (Lockloss, SQZ)
ibrahim.abouelfettouh@LIGO.ORG - posted 08:41, Sunday 02 February 2025 - last comment - 16:47, Sunday 02 February 2025(82588)
SQZ Morning Troubles (Lockloss 16:17)

SQZ Morning Troubles

After reading Tony's alog, I had the suspicion that this was the SHG power being too low, causing lock issues. I trended Sheila's SHG Power while adjusting the SHG temp to maximize. This worked to lock SQZ almost immediately - screenshot below.

However, despite SQZ locking, it looked like in trying to lock for hours the SQZ_ANG_ASC had gone awry and as the wiki put it, "ran away" I requested the reset SQZ_ANG guardian but this didn't work. Then, I saw another SQZ Angle reset guardian, called "Reset_SQZ_ANG_FDS", which immediately broke guardian. The node was in error and a bunch of settings in ZM4 abd ZM5 got changed. I reloaded the SQZ guardian and successfully got back to FREQ_DEP_SQZ. I saw there were some SDF changes that had occured so I undid them, successfully making guardian complain less and less. Then, there were 2 TCS SDF OFFSETs that were ZM4/5 M2 SAMS OFFSET and they were off by 100V, sitting at their max 200V. I recalled that when SQZ would drop out, it would notify something along the lines of "is this thing on?". I then made the assumption that because this slider was at its max, all I'd need to do was revert this change, whic happened over 3hrs ago. I did that and then we lost lock immediately.

What I think happened in order:

  1. We get to NLN, SQZ FC can't lock due to low SHG power. It had degraded from Sheila and I's set point of 2.2 to a further 1.45.
  2. Guardian maxes out its ability to attempt locking by using ZMs and SQZ Ang ASC (if those aren't the same thing) and sits there, not having FC locked and not being able to move any more.
  3. Tony gets called and realizes the issue and can only get to a certain level before LOCK_LO_FDS rails, I assumer due to the above change. This means that even if we readjust the power, the ZM sliders would have to be adjusted.
  4. I get to site, change the temp to maximize SHG power and SQZ locks but at this weird state where ZM5 and ZM4 have saturated M2 SAMS OFFSETS. I readjust but the slider difference is so violent, it causes a lockloss.

Now, I'm relocking, having reverted the M2 SAM OFFSETs that guardian made and having maximized the SHG Temp power (which interestingly had to go down contrary to yesterday). I've also attached as screenshot of the SDF change related to the AWC-SM4_M2_SAMS_OFFSET because I'm about to rever the change on the assumption it is erroneous.

What I find interesting is that lowering the SHG temp increases the SHG power first linearly, then after some arbitrary value, exponentially but only for one big and final jump (kind of like a temperature resonance if you can excuse my science fiction). You can see this in the SHG DC POWERMON trend that shows a huge final jump.

Images attached to this report
Comments related to this report
ibrahim.abouelfettouh@LIGO.ORG - 10:05, Sunday 02 February 2025 (82589)

Recovered from Lockloss and got to OBSERVING automatically - 17:58 UTC.

Only SDF was the only change I made which was re-adjusting SHG temperature to maximize its power.

Images attached to this comment
ibrahim.abouelfettouh@LIGO.ORG - 10:48, Sunday 02 February 2025 (82591)

Investigating further, I realize that once I fixed the SHG temp issue, the message "SQZ ASC AS42 not on??"  was the flashing notification, in which I should have followed: alog 71083. Instead, SDF led me to what the specific issue with ASC was. I will continue to monitor and investigate. Since range is fluctuating, I will probably go into comissioninbg to try and optimize squeezing once thermalized.

sheila.dwyer@LIGO.ORG - 11:44, Sunday 02 February 2025 (82593)

Ibrahim summarized this over the phone for me, and here's our summary of 4 things that need a fix or looking into (sometime):

  • The SQZ angle shouldn't be adjusted when there is no AS42 light.  The ASC has a check for this, which means light from the squeezer is so misaligned it isn't reaching the AS WFS or the OMC, but SQZ _ANG_ADJUST seems not to check.
  • The SQZ_MANAGER RESET_SQZ_ANG_FDS has some kind of typo that made the sqz manager go into error. (log attached, main sets a self..timer['wait'] but then in run it refers to self['wait'], this state must never have been run before).  I fixed the typo and loaded it but haven't tested it.
  • RESET_SQZ_ASC sets ZM4 +5 P + Y lock gains to 1, but that is not what they are normally set to.  We can either re-write this to check the gains and reset them, or move these gains to the servo filters. (I haven't done either of these things this morning).
  • Why did the PSAMs offsets get set to 200V at 5:49 AM local time? 
    • I looked at this, it seems that SQZ_MANAGER was just sitting in FDS_READY_IFO at the time, and the strain gauge servo settings didn't change at this time, and the two changes happened at exactly the same time so, so SDF seems like the most likely thing to have changed these.  I had trouble figuring out which SDF table these are in.  It does seem like the strain guage is at the same location that it was before all this happened, so the PSAMs should be at the same curvature.
Non-image files attached to this comment
ryan.short@LIGO.ORG - 16:47, Sunday 02 February 2025 (82598)

In regards to Sheila's final bullet point above, the change to the PSAMS offsets happened as a result of Tony's misclick (alog82585) where he brought SQZ_MANAGER to 'SET_ZM_SLIDERS_IFO [52]' as reported by the Guardian log at that time:

ryan.short@cdsws25[~]: guardctrl log -a 1422539317 -b 1422539318 SQZ_MANAGER
2025-02-02_13:48:19.274841Z SQZ_MANAGER [DOWN.run] timer['pause'] done
2025-02-02_13:48:19.445441Z SQZ_MANAGER EDGE: DOWN->SET_ZM_SLIDERS_IFO
2025-02-02_13:48:19.445441Z SQZ_MANAGER calculating path: SET_ZM_SLIDERS_IFO->SET_ZM_SLIDERS_IFO
2025-02-02_13:48:19.449362Z SQZ_MANAGER executing state: SET_ZM_SLIDERS_IFO (52)
2025-02-02_13:48:19.450448Z SQZ_MANAGER [SET_ZM_SLIDERS_IFO.enter]
2025-02-02_13:48:19.491549Z SQZ_MANAGER [SET_ZM_SLIDERS_IFO.main] ezca: H1:SUS-FC2_M1_OPTICALIGN_P_OFFSET => 252.9999999999991
2025-02-02_13:48:19.546438Z SQZ_MANAGER [SET_ZM_SLIDERS_IFO.main] ezca: H1:SUS-ZM4_M1_OPTICALIGN_P_OFFSET => -559.3071
2025-02-02_13:48:19.708738Z SQZ_MANAGER [SET_ZM_SLIDERS_IFO.main] ezca: H1:SUS-FC2_M1_OPTICALIGN_Y_OFFSET => 43.900000000000425
2025-02-02_13:48:19.792385Z SQZ_MANAGER [SET_ZM_SLIDERS_IFO.main] ezca: H1:AWC-ZM4_M2_SAMS_OFFSET => 200
2025-02-02_13:48:19.837611Z SQZ_MANAGER [SET_ZM_SLIDERS_IFO.main] ezca: H1:AWC-ZM5_M2_SAMS_OFFSET => 200

I also show this interaction in the attached ndscope.

Images attached to this comment