OBSERVATION channels now handled by IFO top node

The additional status channels added to guardian in the recent 1.3.1 release have enabled us to now handle all OBSERVATION status logic in the GRD-IFO top node.  The following binary channels (0=False, 1=True) indicate the overall OBSERVATION state of the observatory:

• H1:GRD-IFO_READY: The interferometer is fully locked in it's nominal low noise state, and all relevant guardian nodes are reporting OK status.
• H1:GRD-IFO_INTENT: The operator has requested the OBSERVE state, indicating that the observatory is ready for science observation.
• H1:GRD-IFO_OK: The observatory is in full scientific OBSERVATION mode.

Behavior of the IFO top node

The IFO top node is monitoring the status of every other guardian node in the full system.  The primary tasks of the top node is to wait until all other nodes are individually reporting OK status.  If any node is not reporting OK status, the top node H1:GRD-IFO_READY bit reports False/0(zero).  Once all nodes are reporting OK, the READY bit flips to True/1.

The top node has two requestable states: "OBSERVE" and "COMMISSION".  These states represent the intent of the human observatory staff.  When the "OBSERVE" is requested (and the top node is not "stalled") the H1:GRD-IFO_INTENT bit will be True/1.  Otherwise it will be False/0.

Once both the READY and INTENT bits are both True, the system will proceed to the OBSERVE state, and the H1:GRD-IFO_OK bit will flip to True/1.  This will indicate that the observatory is in OBSERVATION/SCIENCE mode.

Operational mode: MANAGED or AUTO

The top node can also be in one of two operational modes: MANAGED or AUTO.

In MANAGED mode, the on-shift operator is expected to re-confirm intent *after* the system has transitioned to READY and before the system will flip the OK bit.   Observation intent is expressed by re-requesting the "OBSERVE" state.

In AUTOMATIC mode, which is intended for unattended operation of the facility, no reconfirmation of intent will be required after transitioning to READY.  Once the "OBSERVE" state has been requested, the OK bit will flip as soon as the READY bit flips.  In other words, in AUTO mode, if the OBSERVE state is requested, we will go straight to observation mode as soon as the IFO locks and all guardian nodes report OK status.

Operator interface, with example

The OBSERVATION_OVERVIEW MEDM screen has been updated to expose all of the above behavior in an intuitive way (very similar to how it was in O1 and O2).  This screen can be viewed independently, but it is also primarily embedded into the top of the GUARD_OVERVIEW screen.

To illustrate the above behavior, we examine the OBSERVATION_OVERVIEW screen using an "IFO2" node acting as a stand-in for the IFO top node, monitoring a single "TEST" node acting as a stand-in for the full guardian hierarchy.

When the interferometer is not in it's final configuration (e.g. TEST node not OK), the screen looks like this:

The READY, INTENT, and OBSERVE boxes are all orange, indicating that their values are False.

Note that we are in "MANAGED" mode: the box on the left is purple and the "MANAGED OPERATION" button is highlighted.   This means that the operator will always need to confirm OBSERVE before final observation.

Once the instrument reaches the nominal low noise state (e.g. TEST_OK = True), the READY bit will flip to True, and the READY box will turn green:

Note that the INTENT box is now yellow, indicating that the operator must re-confirm the intent to go to OBSERVE.  Once the operator confirms intent by clicking the "OBSERVE" button, the OK bit flips and we are in the full OBSERVE state:

If we lose lock, all status bits are reset to False (the INTENT box is yellow to indicate that the INTENT bit was forcibly unset by lock loss because of being in MANAGED mode):

In AUTO mode (left box blue and "AUTOMATIC OPERATION" button highlighted), if we lose lock, the READY bit flips but the INTENT bit remains set:

Once nominal low noise is regained and the READY bit flips, the OK bit also flips immediately and the system goes straight back into OBSERVE: