While we were working on engaging ASC (see midday mini update alog 50947), we lost lock quite suddenly. I had my locking screen open, and saw that the LSC-DARM gains had been set to zero, and the yellow ramping triangles were active. I didn't really have time to figure out what to do, and then we lost lock suddenly (which makes sense if we stopped controlling the DARM length).
In the attached ndscope screenshot you can see that the POP DC has good buildups, then the DARM1 and DARM2 gains are set to zero (this EPICS channel doesn't capture the fact that they were ramping down, just that the setpoint went to zero). After that, we lost lock according to the buildups, and then ISC_LOCK noticed and started the DOWN sequence.
This is a problem. If this is the only time it's ever happened, fine. But, we should look to see if this is what has been causing our mysterious locklosses the past few weeks.
Several locklosses are NOT this problem, so maybe it's not systemic, and was some weird fluke this one last lockloss.
- 1248455995, alog 50889. DARM gains go to zero after ISC_LOCK goes to DOWN, which happens after the POP LF buildup is gone.
- 1248347178, alog 50868. Again DARM gains go to zero after we've lost the power buildups.
While investigating this lockloss, I noticed that there was a glitch in H1:LSC-DARM_OUT_DQ at very close to the same time as the DARM gains go to zero (see attached png). The last relevant ISC_LOCK log entries around this time were "Unstalling ALS_DIFF" and "Unstalling ALS_YARM", which both happened about a second before the glitch. These occurred in the run state of PREP_ASC_FOR_FULL_IFO, which doesn't do anything except check if the nodes are stalled. My guess is that the 'node.revive' function (in NodeManager) for ALS is what changed the DARM gains to zero.
ALS is usually shuttered at ENGAGE_ASC_FOR_FULL_IFO but Sheila suggested that, since we were skipping SHUTTER_ALS to reposition the optics around this time, we could have inadvertently run some code that is not supposed to happen in that ISC_LOCK state.
