Jeff. K, Niko, Rahul
After the scheduled Tuesday maintenance, during IFO lock acquisition attempt, we found that the ETMY Guardian got stalled and got into MISALIGNING STATE (as per NIKO, ETMY also tripped this morning). In order to understand, what's going on, we looked at the channel H1:GRD-SUS_ETMY_STATE_N along with the log file (screenshots attached). Initially, Guardian was at ALIGNED (100) state and while attempting (with several requests) to jump (via 109 state) to MISALIGNED state (110), it got stuck at MISALIGNING (109) state. This can be seen in the highlighted section of the log file. During this time the Align IFO (H1:GRD-ALIGN_IFO_STATE_N) went from SET_SUS_FOR_ALS_FPMI (5) to DOWN (0) state and then to PREP_XARM_IR (10). Finally, Guardian went into SAFE and then RESET state.
This seems to be another case of FRS13563. The SUS alignment filters' ramp values gets stuck on, and the check in the run method of the MISALIGNING Guardian state was waiting for them to clear. In this case, it was the TEST banks that seemed to have gotten stuck. The wierd part is that the SWSTAT value of 39936 is True for the gain ramping, not the offset like the node just changed (pythonic check: bool(39936 & 1<<12) => True).
I've only noticed this with the SUS banks, but they just use the LIGOFilter object that the rest of the Guardians use.
Tagged CDS, FRS, and GRD. Removed AOS tag.
