E. Capote, L. Dartez, J. Kissel, R. Short

As you know, Oli and I installed a revamped watchdog system on the ETMs yesterday (LHO:76305). 
In doing so, we arbitrarily set the watchdog threshold for all stages at 25 [um_RMS] (over the 0.1 to 10 Hz band).

Elenna and Louis reported yesterday evening that we had *one* WD trip of the PUM stage after the lock-loss of the fourth of four successful locks (see the last phrase of the last sentence in LHO:76315). Further, they were concerned because, apparently, "the watchdog reset itself."

To the first point -- that there even was a watchdog trip -- the first attachment shows
    - The ISC_LOCK guardian state (H1:GRD-ISC_LOCK_STATE_N), remeber 600 is NOMINAL_LOW_NOISE
    - The 4 stages of OSEM watchdog "is it tripped?" channels (H1:SUS-ETMX_??_WDMON_STATE, with ?? = M0, R0, L1, and L2 for Main Chain and Reaction Chain top stages, the UIM and PUM stages)
    - The 4 stages of OSEM watchdog "is the trigger actively suggesting it should be tripped?" channels (H1:SUS-ETMX_??_WDMON_CURRENTTRIG)
    - The PUM (L2) stage BLRMS trigger signals themselves, now for the first time in physical units! (H1:SUS-ETMX_L2_WD_OSEMAC_RMSLP_??_OUT16)

One can see that lock losses are typically kicking the PUM, up to 10-20 [um_RMS], but otherwise, the quiescent BLRMS is around 0.5 [um_RMS].

To the second point -- we're still investigating, but the second attachment shows that the PUM WD is reset (H1:SUS-ETMX_L2_WDMON_STATE goes from 2 to 1) at 2024-03-13 06:44:32 UTC.

In general, the guardian systems has been build on the axiom of "GUARDIAN SHALL NOT TOUCH WATCHDOGS," so we're quite suspicious of that this was not human-related. The human action that seems to be coincident with the reset on *this* lock acquisition attempt, as opposed to the three attempts immediately prior is that the ISC_LOCK guardian was taken to "INIT." It's not a smoking gun, but it's at least a bread crumb. We'll keep looking.

Finally, the third attachment shows an overall assessment of typical maximum values of the BLRMS for all stages. Here's a summary of the assessment in tabular form:

Stage        Quiescent Average Value        Maximum Transient Value
                    [um_RMS]                     [um_RMS]
M0                    0.35                         6.35
L1                    0.32                        18.34
L2                    0.44                        29.02

(where out of laziness and worry of too crowded a plot, I assume R0 has the same behavior.)

In the mean time, the watchdog is working exactly as expected, and let's bump up the PUM WD threshold to 30 [um_RMS].