Alerted by H1_MANAGER that it needed intervention. There was a large earthquake coming through that had tripped the ISIs for the ITMs and ETMs. Also, ETMY stages M0 and R0 had tripped (not yet sure if that is just due to the ISI tripping).
There was also a notification on verbals from 08:51UTC, a minute after everything tripped, that says "ETMY hardware watchdog trip imminent", but when I checked the hardware watchdog screen everything looked normal with no countdowns. Once it looked like the worst of the earthquake had passed and the ISI values were all within range, I reset R0 and M0 for ETMY and reset all four ISI watchdogs. We are still in LARGE_EQ mode so we haven't started relocking yet, but it looks like we are close to leaving and we should be good to go for relocking.
It looks like ETMY M0 and R0 tripped due to barely going over the watchdog limit, and that it was due to the ISI stages tripping (ndscope). Similar to what I did in 81668, I'll up the thresholds for M0 and R0 by a bit. Thankfully it looks like the thresholds we established for the other stages were all good guesses and are still good.
Stage | Original WD threshold | Max BLRMS reached after lockloss | New WD threshold |
M0 | 100 | 106 | 150 |
R0 | 120 | 122 | 175 |
L1 | 170 | 134 | 170 (unchanged) |
L2 | 270 | 168 | 270 (unchanged) |