J. Kissel 5.6 Mag Earthquake near Japan at 2015-05-10 21:25:46 UTC. For quite some time, when an ISI's watchdog trips, it runs the equivalent of a "down" script via front-end code (see E1300685, installed circa Fall 2013). One of the many things the front-end down script does is to force the isolation feedback loop filter banks' gain to zero, 0.0. However, in the era of the Celerier system of SEI guardians, the ISI nodes are using some Jedi-Level-9 combination of the python tools in /opt/rtcds/userapps/release/isi/common/guardian/isigiardianlib/ to run the platform's isolation stages, and somewhere along the journey to "FULLY_ISOLATED," these filter banks are set to 0.1, and then 1.0 (and I don't think ever to zero, 0.0). Thus, when a watchdog trips, the new Set Point Monitor -- who thinks it's the only thing in control of these values -- sends a notification up the ladder that these gains to zero (0.0) have been changed by something other the guardian, i.e. the front end's down script. No big deal here, it just caught my eye while Elli and Nutsinee were asking me "Can I reset the SEI system... now? How about now?" after the earthquake mentioned above, and I recommended bringing the SEI manager to DAMPED (such they could regain alignment, but be robust while the EQ rings down). I presume we'd never notice, since rarely if every request this interim DAMPED state of the SEI systems these days, but in the interest of non-expert clarity, it would be good to sort this one out. I attach a screenshot of the relevant MEDM screens in this state. Also note ETMX HPI tripped which caused the ISI and SUS to go down a few seconds later, via horizontal actuators (see second attachment), and it looks to a pretty darn quick signal causing the trip. SYS_DIAG was reporting that the EX Tidal was near the edge of its range...
This is, as you say, not a big deal, and pretty easy to work around in guardian. We should modify the ISI guardians to reset the isolation gains to zero after a watchdog trip, thereby duplicating what is done in the front end watchdog reset code. Watchdog trips will then be followed by a brief SPM notifcation blip in the ISI guardians, which will go away as soon as the guardian itself sets the gain to zero, thereby achieving parity between the guardian SPM setpoint and the actual front end value. We should file this with the integration tracker.
This situation will keep happening during these kinds of watchdog trips until we make these changes to the ISI guardians.
General note: SPM notifications don't necessarily mean you need to do anything unusual or out of the ordinary. In this case the ISI guardians were telling you that something was a little strange, but overall the state was actually fine. The notification would have gone away as soon as you reset the watchdog and the ISI guardians started isolating the platforms (and changing the isolation gains). You didn't need to do anything special to deal with this situation.