Naoki, Vicky -

Had an SHG relocking issue just now, when the squeezer briefly dropped lock at 2024 Sept 5 22:46:16 UTC, for the PMC to relock (its PZT bottomed out, routine issue).

SHG guardian went into a weird locking loophole which we had not seen before, summarized in screenshot. The SHG IR trans PD locking beatnote strength goes high when the SHG is unlocked, aka see the H1:SQZ-SHG_TRANS_RF24_DEMOD_RFMON signal. Its threshold is nominally at H1:SQZ-SHG_TRANS_RF24_DEMOD_RFMAX = 0. But the SHG_GRD has a hardfault state that brings guardian down if this threshold is exceeded, so GRD would try to LOCK, then see this error message for RF power level overload, then it would go DOWN, and its stuck.  See SHG guardian logs.

#FIXME: resolve this issue. Ideas- either remove this race condition from SHG guardian (send to IDLE if in fault?), changing RF threshold, etc.

To fix it this time, I manually changed the threshold H1:SQZ-SHG_TRANS_RF24_DEMOD_RFMAX = 5 (threshold was at 0, the demod beatnote was at 3.3), then brought SHG_GRD to LOCKED (worked fine), then reset threshold back H1:SQZ-SHG_TRANS_RF24_DEMOD_RFMAX = 0.

#TODO: decide whether this is a problem that needs fixing or just a weird one-off issue. Trending back, the RF demod power basically always goes high when unlocked, which often triggers this race condition. I'm not sure like 1) why was a problem this time, or 2)  why it is not a problem every time.