Successful upgrade of SEI guardian code today to achieve full SEI deployment

Yesterday, we:

• installed new guardian core (v918) and cdsutils (v237)
• HPI guardian nodes deployed for all active HPI systems: HAM 2,3 and BSC 1,2,3,9,10
• upgraded ISI single-stage code which includes monitoring of T240 saturation state: all ISI stage nodes were restarted
• upgraded BSC chamber manager nodes to include handling of BSC HPI: all BSC managers restarted
• deployed HAM chamber managers for HAM 2,3 that coordinates HAM ISI and HPI

In other words, we now have full SEI guardian for both HAMs and BSCs.  This is a fairly big milestone, as now essentially all seismic systems are fully automated (modulo the output HAMs which are still undergoing installation).

SEI guardian operation

The most important new UI aspect of these improvements is that all interface to the seismic systems for each chamber is now done through the chamber managers, e.g. SEI_HAM3, SEI_ITMY, etc.  They are the manager nodes that coordinate the activity of the HPI and ISI in each chamber.  All HPI and ISI nodes should now be operating in managed mode, which means that there activity is controlled by their respective chamber manager.  Managed nodes are purple.  Here is a shot from the guardian overview screen, showing the BS manager SEI_BS in blue, and the one HPI and two ISI nodes in purple:

All interaction with the seismic systems should now go through the managers.  Here's the graph of the SEI_BS manager, which is identical to the graphs of the rest of the BSC chamber managers:

The dark blue states above are the requestable states, e.g. DAMPED or FULLY_ISOLATED.

In general, hte managed subordinate nodes (i.e. the ISI and HPI nodes) should not be touched while they're in managed mode (i.e. purple) unless you know what you're doing.  Otherwise you run the risk of interfering with and potentially confusing the manager, and therefore distrupting the automation.  That said, the managers are designed to be fairly robust against noodling with their subordinates (see alog 11077 for more information about manager behavior).  Probably the most important thing to keep in mind, that might help you out of a jam or to reset things:

If any of the subordinates are out of wack, for whatever reason, e.g. they are not in the correct MODE or STATE or REQUEST etc., command the chamber manager to go to the INIT state

The manager INIT state will:

• reset all subordinates into MANAGED mode (i.e. reacquire them to being managed)
• check the STATEs of the subordinates to determine the overall state of the system
• REQUEST the appropriate states of the subordinates to recover the system back to a known configuration

SEI guardian code structure

All SEI guardian code is installed in:

$USERAPPS_DIR/isi/common/guardian

At the top level of this directory are the main guardian nodes for HPI, each ISI stage, and the chamber managers.  These modules all load their code from the SEI guardian library:

$USERAPPS_DIR/isi/common/guardian/isiguardianlib

This library includes the primary packages for HPI, ISI_STAGE, HAM_MANAGER, and BSC_MANAGER.  These load support modules/packages also part of isiguardianlib.  For example, the ISI_ITMY_ST1 guardian is defined by the ISI_ITMY_ST1.py module:

from isiguardianlib.ISI_STAGE
import * prefix = 'ISI-ITMY_ST1

The first line loads all code from the isiguardianlib.ISI_STAGE package, and the second line sets the channel access prefix.  All the SEI modules are structured in this way.