When I found H1 this morning following the lockloss @ 13:34 UTC, the PSL was entirely down (NPRO off, verbal alarms mentioned "noise eater out of range" immediately prior to lockloss). I was able to restart the NPRO and both amps just fine, but when I got to relocking the PMC, I noticed the high voltage was dead (see first attachment). Thinking this may have been caused by a power glitch, I trended the mains voltage channels and sure enough that looks to be the case (see second attachment).
Jenne and I have turned on the PMC HV before following a power outage (alog 65026), so I'm headed to site and will start there for recovery.
I went to the upstairs CER rack C5 and indeed found the 375V high voltage supply off, so I restarted it. Once back in the control room, I was able to relock the PMC, FSS, ISS, and IMC without issue. Fortunately, it seems the PSL was the only thing impacted by the power glitch.
Oli has started relocking H1.
Still working on relocking; ISC_LOCK is not able to complete PREP_DC_READOUT_TRANSITION due to the OMC_LOCK guardian getting stuck at FIND_CARRIER many times and not being able to find the 45MHz carrier. OMC_LOCK log output for each time it goes through this cycle is attached as a txt.
DIAG_MAIN has a commented out error message that would alert us if OMC-PZT2_MON_DC_OUT is <1 and GRD-OMC_LOCK_STATE_N is not at 3 (OFF_RESONANCE), which we are at in this case (see attachment1). Due to the low voltage output and the old DIAG_MAIN error message that the problem is due to the PZT2 high voltage probably not being restored correctly.
OMC_DCPD_SUM_OUTPUT reads ~0.0055 at this state even though it should've started increasing once we got to ENGAGE_ASC_FOR _FULL_IFO.
Currently went back to DOWN and started the relocking process again, stopping ISC_LOCK at ENGAGE_ASC_FOR _FULL_IFO, but it looks like we are still having the same issue. Looking through alogs from previous times this happened.
No email from MSR UPS, but GC UPS reported (hour corrected for PDT)
Date : 08/06/2023
Time : 06:34:41
Warning - UPS: On battery power in response to an input power problem.
Date : 08/06/2023
Time : 06:34:45
Informational - UPS: No longer on battery power.
Corner station mainsmon plot attached. End station VFD units did not trip this time.
Reset High Voltage for the OMC and we are back on our way up
18:28UTC Reached Observing