Reports until 09:24, Monday 24 February 2020
H1 CAL (CAL, DetChar)
evan.goetz@LIGO.ORG - posted 09:24, Monday 24 February 2020 - last comment - 14:02, Tuesday 25 February 2020(55257)
X-arm PCAL optical follower servo saturated since Wednesday Feb 19 2020 
Yanyan noted in the DQ shift for LHO that the x-arm PCAL lines--both high frequency lines and CW hardware injections--disappeared on Wednesday Feb 19 2020 (see summary page plot). It looks like the optical follower servo saturated on this day. The loop should be opened and closed again to regain normal operations as soon as is possible. Likely this will cause a glitch so perhaps it should be done out of observing.
Comments related to this report
jeffrey.kissel@LIGO.ORG - 11:17, Monday 24 February 2020 (55259)DetChar, INJ, OpsInfo
Link 

Thanks for catching this Evan!! 

I've added an *additional* check to the DIAG_MAIN notification guardian to look at the *error* signals of the PCAL OFS loops, e.g. H1:CAL-PCALX_OFS_ERR_OUT16. If either PCALX or Y's OFS error signal channels are less than -2.0 for more than 120 seconds, DIAG_MAIN throws the error "PCAL: PCAL [X,Y] OFS servo malfunction". We have loaded the new DIAG_MAIN guardian check, and confirmed that it now throws the correct error given this current state.

Instructions for what to do about this (in a "normal" situation) have been updated in the DIAG_MAIN page of the OPS wiki.

Note -- Sudarshan and I tried following those basic instructions, but it appears that PCALX is more broken than just a saturated OFS control loop.

Rick and Sudarshan are actively investigating while I run calibration measurements.
jeffrey.kissel@LIGO.ORG - 10:53, Tuesday 25 February 2020 (55285)CDS, DetChar, INJ, OpsInfo
Link 

"Wednesday Feb 19 2020" is a bit of a misnomer: a better summary page plot at which to determine the time of failure is the PCALX laser current here, which shows the current drop to 0.0 at 
2020-02-19 00:15 UTC, which is 
       Feb 18 2020 16:15:00 PST
i.e. Tuesday at 4pm local, i.e. only 4 hours after maintenance day closed.

This time is coincident with Keita and Fil's journey to the EX and reboot of the EX ISC IO chassis (see LHO aLOG 55171). *That* effort was a result of the CDS team attempting to mitigate sharp spectral line features earlier in the day, which thwarted some ALS WFS, (see LHO aLOG 55173).

The last sentence of that 55173 aLOG indicates what happened:
"Then we found, after driving back to the corner, that power cycling IO chassis, or maybe disconnecting AA from IO, triggered the safety system at EX to go crazy, the safety system thought that the status was fine but the laser interlock was kept open so the laser couldn't be turned on. This was manually bypassed."

The *ALS* laser system's interlock was bypassed, but the PCALX laser system's interlock was forgotten about.

The CDS/PCAL teams will further investigate if this is what happened during today's maintenance. 

Stay tuned!
jeffrey.kissel@LIGO.ORG - 14:02, Tuesday 25 February 2020 (55295)CAL, CDS, DetChar, INJ, OpsInfo
Link 

Story confirmed. PCALX Laser functionality restored after interlock was repaired today. See LHO aLOG 55291.