8.3 hours ago PCAL X saturated. I flipped the optical follower servo on and off to fix this. We should monitor the optical follower servos to make sure they aren't railed.
Notably, this switch-flipping did not knock us out of Observe, although it probably should have.
J. Kissel, T. Shaffer The exact time at which Craig flipped the switch is 1258246245.4 and flipped the switch back 1258246245.8 within the second of Nov 20 2019 00:50:27 UTC. That should be roughly when the broadband, 1/f^2 in displacement glitch occurred. See attached screen shot, which shows that flipping this switch, which is on h1sysecatx1_pl3, doesn't take us out of observing. IT SHOULD. This is under investigation. Also, we should write a DIAG_MAIN check that checks whether the OFS is saturated, for both PCALs. This is under construction.
I wrote a test in DIAG_MAIN to check this for both PCAL X & Y. I had to improve the StaticTester class to look for greater than or less than values rather than just static, a welcomed improvement for future tests.
If Operators get a "PCAL (X/Y) OFS servo malfunction" notification on DIAG_MAIN:
Toggling the OFS loop should fix it. This is found as the orange toggle switch in the respective PCAL screen, or the channel H1:CAL-PCALX_OPTICALFOLLOWERSERVOENABLE
I'm still very confused why the DIAG_SDF did not show a difference when this happend. The guard log shows a difference for sysecatx1plc3 a handful of hours before this happened, so we know it is being monitored. The snap file should have been in observe because ISC_LOCK switches them as we get to NLN. No idea why it wouldn't have flagged it.