C. Gray, K. Kawabe, J. Kissel, D. Sigg
After initial alignment was complete (2019-07-24 ~20:00 UTC), we spent a bit of time stuck with several lock losses upon reaching LOCKING_ALS. Manually running what happens in LOCKING_ALS, we narrowed down the problem to ALS DIFF, and after 45minutes of searching, I happened to lance at the ETMX ESD driver's analog voltage monitors and noticed they looked abnormal. At 2019-07-24 20:42 UTC, I hit the H1:SUS-ETMX_BIO_L3_RESET ("HV ON/OFF") button on the QUAD overview screen once to turn OFF the driver, then once again a few seconds later to turn it ON.
It's likely that the ETMX ESD driver got locked up in a funky state after all of the power and computer issues over night.
The bigger question is why (what we thought were several) warnings that should have been triggered didn't:
(1) DIAG_MAIN guardian node "used to" (our memory of the O2 past) have a check of the analog voltage monitors, and would throw a warning.
(2) The DOWN / PREP_FOR_LOCKING state should have a similar check of the analog voltage monitors, throw and error when failed, and not allow progress until fixed.
Others are investing further and will comment / fix later.
The DIAG_MAIN test would notify if H1:IOP-SUSAUX_EX_MADC4_EPICS_CH0 was between +/-16200 when the ISC_LOCK state was below 515. This has been changed to look at the DC and the quadrants' inmon channels (ex. H1:SUS-ETMX_L3_ESDAMON_DC_INMON) to check that the DC signal is within 10% of -32300, the quadrants are less than 8000, and ISC_LOCK is under 500 before notifying. This seems to be a better way to confirm that the ESD is off.