at approx 8:27UTC H1 lost lock. There were no indicators on any of the FOMs to indicate impending lock doom. Sheila has just logged on remotely as reported by the freshly restarted Verbal Alarms which did NOT report the lockloss when it happened. Now it seems that Verbal Alarms are one step behind Guardian or not reporting anything at all. DIAG_MAIN is reporting that ESD Y driver is off, but I can't remember if this is normal for this part of the lock or not.
Also, on a side note, PT424 and 524 B are in the red.
Re_Locking:
First attempt - Lost lock somewhere around PREP_TR_CARM. Verbals are no help. Restarted again.
Second attempt - going better. Verbals still dodgy/sporadic. So, I was prompted by guardian to "HELP ME!!" due to the fact that we had reached LOWNOISE_ESD_ETMY and the ESD driver was still off. I turnd it on by clicking the HV ON/OFF button on te ETMY MEDM screen. Now guardian seems to be stalled so I will attempt to move it forward manually. That worked. And it broke at Noise tunings. Shades of what I walked in on at the beginning of my shift. Will try a third attempt.
Third attempt - I believe it broke at SWITCH_TO_QPDS. Restarted the computer with Verbals and AH to no advantage.
Fourth attempt - I stopped Guardian at SRM_ASC_HIGH_POWER to manually step through NOISE_TUNINGS. The step-through process seemed to work. I may have made my mistake at manually jumping over to CLOSE_BEAM_DIVERTERS instead of proceeding with the NOISE_TUNINGS step? I made i to NLN but SDF was showing some PSL ISS diffs that I don't know why they were there. I tried reverting them. That was a bad idea. Guess I should have accepted and moved on. Lesson learned. Back to square 1. :/
stay tuned...
Next attempt got me through ACCEPTING the aforementioned ISS diffs only to be left with a waiting for IMC_LOCK node notification (see screenshot). When I attempted to request the nominal state for this node everything came loose.
12:20UTC I'm leaving H1 in NLN but not Observing. I don't know how to get the IMC_LOCK node into it's nominal state without breaking the lock that I have.
Keita and I looked into one of these locklosses from last night, and for one of them the cause seems to be related to the ESD turning off (before the lockloss). Ed got the error message about the ESD in the next lock acquisition and reset the ESD.
The last couple of problems that Ed had are related to a problem with running the guardian by hand line by line. In the code there are places where ISC_LOCK makes a request from other guardians (in NOISE_TUNNINGS there is a line nodes[IMC_LOCK] = 'ISS_DC_COUPLED') this doesn't work in the command line unless you have initialized the node manager. The easiest way to do this is to request the state from the IMC guardian using the medm screen. In this case, the ISS has to be DC couped before the IMC boost comes on later in the state, doing these things in the opposite order breaks the lock.
We will continue investigating the problem in the NOISE_TUNNINGS that caused Travis and Ed to do it by hand in the first place.