J. Kissel, E. Merilh
Just before a big EQ in Alaska/Canada took us out of maintenance recovery lock acquisition attempts, we had 2 lock losses at / around the start of the CARM reduction sequence.
Grabbing some plots from the lockloss tool (though the web server appears to be down): it appears as though there's some positive feedback that triggers around this step, ringing up and causing the locklosses. At the moment, I only have POP_A_LF, and the arm powers in red (ASC-[X,Y]_TR_NSUM) showing this symptom, but it's worth looking in to, and I will. Help would be appreciated, though.
Plots are of the three lock losses, attached in chronological order:
UTC time GPS Time
2019-09-10 10:47:42 1252147680
2019-09-10 20:00:11 1252180829
2019-09-10 20:26:58 1252182436
Looking at the guardian, code history a little, it seems that we used to engage REFLBIAS FM4 before reducing the gain in the ALS path, which would give the TR_CARM loop a little bit more phase margin around 90 Hz, which might avoid locklosses like this (it seems like this is a case where the optical gain for TR_CARM was low, which might be alignment related.)
The alogs I can find about this anti-boost are 43190 (some errors where made in these states when the guardian was re-written, this alog includes debugging that), and 36686
For now I have changed the CARM to TR state so that it is engaging FM4n (the anti-boost) before ramping down the ALS analog gain, restoring it to the way it used to work. This seems to have worked, you can compare the two ndscopes from before and after the change.
Ed and Jeff K also reported that when they tried to request PRMI (while DRMI was trying to acquire), the guardian didn't change state. This was because there was a timer which would increment if DRMI triggered momentarily but didn't actually lock, and if that counter was incremented the guardian would not check for a change in the target state. That is working now.
Trying to analyze CARM_TO_TR locklosses, based on the channels listed by Sheila (/ligo/home/sheila.dwyer/Desktop/Locklosses/ lockloss -c channels_to_look_at_TR_CARM.txt select). I am attaching screenshots of the last 3 locklosses which falls under this category. This is still ongoing, as there are several things which I am trying to understand. The following 2 channels (H1:ASC-X_TR_B-NSUM_OUT_DQ, ASC-Y_TR_B-NSUM_OUT_DQ) rings up 1 second before lockloss.
