J. Kissel, T. Schaffer 

During the brief period we were down and attempting to lock the IFO this afternoon, TJ's robot -- we'll call him Jeeves today -- was announcing that the SUS SRM and SUS OMC's Software Watchdogs were tripping. The software watchdogs have not tripped. This watchdog noise has been going on for quite some time, and I wanted to get to the bottom of it. Here're my conclusions:

(1) Neither the SUS SRM or SUS OMC are in danger.
(2) The robot is claiming a "trip," but it is merely reporting that the EPICs records
H1:IOP-SUS_$(OPTIC)_WD_RMS
H1:IOP-SUS_%(OPTIC)_DK_SIGNAL
are indicating that the BLRMS of top mass OSEMs is briefly surpassing threshold. Recall that we need 300 [sec] = 5 [min] of these indicators at solid "bad" = "red" before *any* action is taken by the software watchdog. Thus, the robot is claiming a much worse thing has happened than what has really happened.
(3) The suspensions' top masses are moving "so much" because of ISC control fed to the SUS and those stages of the SUS (for the SRM, it's occasionally blasts from the angular controls, and for the OMC it's more consistent blasting in all L, P, and Y DOFs -- but also for angular control). I imagine that because the ground motion is rather large right now with the wind, the top masses are moving more from residual ground motion AND the ISC request is larger trying to account for it.
(4) The threshold for the software watchdog is set right where it should be, given how long and consistently the threshold must saturate before any action is taken. The USER watchdogs, which watch the same RMS signal (just not calibrated into volts as the software watchdog has been), are at 20% / 75% of their threshold for SRM / OMC (respectively) for tripping during this time. I don't argue that these have been set any more carefully, but in any case, the USER WD should trip first, and in this case it will.
(5) We could probably treat the suspensions a little nicer by being a little smarter about when we turn on the ASC control, or putting software limits on the control signals, but they're big girls -- they can handle it.

I suggest the only thing we change is the EPICs record that Jeeves watches to 
H1:IOP-SUS_OPTIC_DK_WD_OUT
to report that the software watchdogs have tripped. This record tells you that 5 minutes has passed in which the OSEMs have surpassed their threshold, and has triggered the SEI system's countdown until shutdown which will be 5 minutes later.

I attach screenshots of the ISC Lock Guardian's state, the RMS values of the trigger signals for both the USER and SW WDs for both SRM and OMC, as well as a trend of the ISC requested drive to that respective suspension. These trends, and my knowledge of the design of each of these watchdogs, are what have convinced me of the above conclusions.