SUMMARY: Back To OBSERVING, but got here after going in a few circles.
H1 had a lockloss before the shift, but when I arrived H1 was at NLN, BUT SQZ had issues.
I opened up the SQZ Overview screen and could see that the SQZ_SHG guardian node was bonkos (so I had this "all node" up the whole time...it was crazy because it was frantically moving through states to get LOCKED, but could not), BUT also I saw.....
1) DIAG_MAIN
DIAG_MAIN had notifications flashing which said: "ISS Pump is off. See alog 70050." So, this is where I immediately switched focus.
2) Alog70050: "What To Do If The SQZ ISS Saturates"
Immediately followed the instructions from alog70050 which were pretty straightforward. Remember: H1's not Observing, so I jumped on the alog instructions and wanted to get back to Observing ASAP.) I took the opo_grTrans_setpoint_uW from 80 to 50, and tried to get SQZ back to FDS, but no go (SQZ Manager stuck....and SQZ_SHG still bonkos!).
At this point, I saw that there were several other sub-entries with updated instructions and notes. So I went through them and took opo_grTrans_setpoint_uW to 60 (no FDS + SQZ_SHG still crazy), and finally set opo_grTrans_setpoint_uW = 75 (but still no FDS + SQZ_SHG still crazy).
At this point, I'm assuming DIAG_SDF sent me on a wild goose chase. Soooooo, I focused on the erratic SQZ_SHG......
3) Alog search: "SQZ_SHG" ---> H1:SQZ-SHG_TEC_SETTEMP Taken to 33.9
This did the trick! And this search took me to early Feb2025 alogs from (1) RyanS alog82599 which sounded what I had and then (2) Ibrahim's alog82581 which laid out instructions for what to do for adjusting the SHG TEC Set Temperature (went from 33.7 to 33.9; see attachment #1). AND---during these adjustments the infamous SQZ_SHG finally LOCKED!!
After this it was easy and straightforward taking the SQZ Manager to FDS and get H1 back to Observing.
NOTE: I wanted to see the last time this Set Temperature was adjusted and it was Feb 17, 2025. Doing an alog search on just "SHG" + tag: SQZ took me to when it was last adjusted: By ME! During an OWL wake-up call, I adjusted this set point from ~35.1 to 33.7 on 02172025 at 1521utc/72amPST (see attachment #2).
The only SDF to ACCEPT was the H1:SQZ-SHG_TEC_SETTEMP = 33.7 (see attachment #3). BUT: Remember other changes I made (when I erroneously thought I had to adjust the OPO TEC TEMP which are not in SDF:
- opo_grTrans_setpoint_uW = 75 (was 80)
- H1:SQZ-OPO_TEC_SETTEMP = 30.937 (was 30.941)
Hindsight is 20/20, but if I addressed the "bonkos SQZ_SHG" via instructions from an alog search first, I would have saved some time! :)
During this time that the SHG guardian was cycling through it's states, that was happening because of the hard fault checker, which checks for errors on the SQZ laser, PMC transdiode, SHG demod, phase shifter.
The demod had an error because the RF level was too high, indeed this was above this threshold in this time and dropped back to normal allowing Corey to lock the squeezer.
The second screenshot shows a recent time that the SHG scaned and locked sucsesfully, in this case as the PZT scans the RF level goes up as expected when the cavity is close to resonance, and also goes above the threshold of 0dBm for a moment, causing the demod to have an error. This must have happened not at the moment when the guardian was checking this error, so that the guardian allowed it to continue to lock.
It doesn't make sense to throw an error about this RF level when the cavity is scanning, so I've commented out the demod check from the hardfault checker.
Also, looking at this hardfault checker, I noticed that it is check for a fault on the PMC trans PD. It would be prefferable to have the OMC guardian do whatever checks it needs to do on the PMC, and trust the sqz manager to correctly not ask the SHG to lock when the PMC is unlocked. Indeed, SQZ manager has a PMC checker when it is asking the SHG to lock, so I've commented out this PMC checker in the SHG guardian. This same logic applies to the check on the squeezer laser, leaving us with only a check on the SHG phase shifter in the hardfault checker.
Editing to add: I wondered why Corey got the message about the pump ISS. DIAG_MAIN has two checks for the squeezer, first that SQZ_MANAGER is in the nominal state, then second that the pump ISS is on. I added an elif to the pump ISS one, so if the sqz manager isn't in the nominal state this will be the only message that the operator sees, Ryan Short and I looked at the SQZ_MANAGER and indeed it seems that there isn't a check for the pump ISS in FREQ_DEP_SQZ.
SQZ_SHG guardian and DIAG_MAIN will need to be reloaded at the next oppurtunity.
