WP8329

I added a test into the DIAG_CRIT Guardian node to monitor the seismic configuration nodes while we are observing.

Quick recap of how we get to Observing, from a Guardian perspective:

The node "IFO" checks every other node, that is not in an exclude list, for its OK channel to be True (ie. H1:GRD-{node name}_OK). If all of the nodes that are monitored have their OK channel True, then the operator can click the intention bit to flag us into Observing. Besides the normal nodes used for regular operation of the machine, there are three diagnostic nodes that are monitored by the IFO node: DIAG_SDF, DIAG_EXC, and DIAG_CRIT. These nodes monitor for SDF differences, front end excitations, and any critical tests defined. Up until this point, DIAG_CRIT has not contained any tests, simply because we have not found the need for it.

In recent discussions with Jamie about the seismic configuration nodes, he expressed concern that these nodes were vital to the operation of the machine, but were not being monitored by the system, only the operator. These nodes were removed from the IFO node monitoring before the start of O3 because their state can change based on the environment (not autonomously...yet). A proposed solution to this lack of coverage was to put a test in DIAG_CRIT to monitor the health of these nodes. And here we are.

The test looks at the ACTIVE Guardian defined channel for all of the seismic configuration nodes. This channel is True when the following are True:

OP = EXEC (Not Paused or Stopped)
MODE in [AUTO or MANAGED]
LOAD_STATUS = DONE
WORKER != INIT
not in ERROR
Connection status is OK

If any of these are False for the seismic configuration nodes, DIAG_CRIT will display a message and we will not be able to Observe.