jameson.rollins@LIGO.ORG - posted 18:09, Wednesday 25 June 2014 (12486)
guardian core upgraded; all nodes restarted
This afternoon I upgraded the guardian core to r1020, and cdsutils to r250. There are a couple important new guardian features that I wanted to pull in:
- There is a new NONE pseudo-state request. When the request is NONE, guardian does not calculate a path away from the current state, or change states one the current state is complete. Jump transitions are honored, though. This could also be considered a "follow" mode of operation, where jump transitions can be used to track the state of the system without controlling it. This is also now the new default REQUEST state on startup.
- More robust handling and recovery from loss of channels. If, during the course of state execution, there is a channel access error (connection or write), the guardian will go into a "connection error" mode. This will show up in the node as an error (red background) but it will continue to attempt to execute the current state method until the error clears. This allows guardian to ride out front end or subordinate node restarts and recover somewhat gracefully.
- Other minor improvements and bug fixes.
I then restarted all nodes with "guardctrl restart --all". All SUS nodes had been shut down for the moment, and all SEI nodes are in watchdog trip states, so this didn't do too much. Some nodes that don't specify initial request states came up with the new NONE request. I set manually reset the requests for those nodes back to what they were before the restart. No major issues:
node s m n vers request state message
---- - - - ---- ------- ----- -------
ALS_COMM * E ! 1020 DOWN DOWN not ready for handoff
ALS_DIFF * E - 1020 DOWN DOWN -
ALS_XARM * E - 1020 UNLOCKED UNLOCKED -
ALS_YARM * E - 1020 UNLOCKED UNLOCKED -
H1ECATC1PLC2 * E - 1020 OP OP -
H1ECATX1PLC2 * E - 1020 OP OP -
H1ECATY1PLC2 * E - 1020 OP OP -
H1ISCEX * E - 1020 OP OP -
H1ISCEY * E - 1020 OP OP -
H1LSC * E - 1020 OP OP -
H1LSCAUX * E - 1020 OP OP -
HPI_BS * E ! 1020 ROBUST_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
HPI_ETMX * M - 1020 ROBUST_ISOLATED ROBUST_ISOLATED -
HPI_ETMY * M - 1020 ROBUST_ISOLATED ROBUST_ISOLATED -
HPI_HAM2 * E ! 1020 ROBUST_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
HPI_HAM3 * E ! 1020 ROBUST_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
HPI_ITMX * E ! 1020 ROBUST_ISOLATED WATCHDOG_TRIPPED_DAMPING WATCHDOG TRIP: DAMPING_ONLY (3)
HPI_ITMY * M ! 1020 ROBUST_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
IMC * E ! 1020 DOWN FAULT PSL periscope PD error (low light?)
ISI_BS_ST1 * E ! 1020 HIGH_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
ISI_BS_ST2 * E ! 1020 HIGH_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
ISI_ETMX_ST1 * M - 1020 HIGH_ISOLATED HIGH_ISOLATED -
ISI_ETMX_ST2 * M - 1020 HIGH_ISOLATED HIGH_ISOLATED -
ISI_ETMY_ST1 * M - 1020 HIGH_ISOLATED HIGH_ISOLATED -
ISI_ETMY_ST2 * M - 1020 HIGH_ISOLATED HIGH_ISOLATED -
ISI_HAM2 * E ! 1020 HIGH_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
ISI_HAM3 * E ! 1020 HIGH_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
ISI_HAM4 o - - - - - -
ISI_HAM5 o - - - - - -
ISI_HAM6 o - - - - - -
ISI_ITMX_ST1 * E ! 1020 HIGH_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
ISI_ITMX_ST2 * E ! 1020 HIGH_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
ISI_ITMY_ST1 * M - 1020 HIGH_ISOLATED DAMPED -
ISI_ITMY_ST2 * M ! 1020 HIGH_ISOLATED WATCHDOG_TRIPPED_FULL_SHUTDOWN WATCHDOG TRIP: FULL_SHUTDOWN (4)
LSC * E - 1020 NONE INIT -
SEI_BS * E ! 1020 ISOLATED_DAMPED WATCHDOG_TRIPPED_DACKILL WATCHDOG TRIP: DACKILL
SEI_ETMX * E - 1020 FULLY_ISOLATED FULLY_ISOLATED -
SEI_ETMY * E - 1020 FULLY_ISOLATED FULLY_ISOLATED -
SEI_HAM2 * E ! 1020 ISOLATED WATCHDOG_TRIPPED_DACKILL WATCHDOG TRIP: DACKILL
SEI_HAM3 * E ! 1020 ISOLATED WATCHDOG_TRIPPED_DACKILL WATCHDOG TRIP: DACKILL
SEI_ITMX * E ! 1020 FULLY_ISOLATED WATCHDOG_TRIPPED_DACKILL WATCHDOG TRIP: DACKILL
SEI_ITMY * E ! 1020 FULLY_ISOLATED WATCHDOG_TRIPPED_SUBORDINATE WATCHDOG TRIP: SUBORDINATE
SUS_BS o - - - - - -
SUS_ETMX o - - - - - -
SUS_ETMY o - - - - - -
SUS_ITMX o - - - - - -
SUS_ITMY o - - - - - -
SUS_MC1 o - - - - - -
SUS_MC2 o - - - - - -
SUS_MC3 o - - - - - -
SUS_OMC o - - - - - -
SUS_PR2 o - - - - - -
SUS_PR3 o - - - - - -
SUS_PRM o - - - - - -
SUS_SR2 o - - - - - -
SUS_SR3 o - - - - - -
SUS_SRM * E - 1020 DAMPED DAMPED -
SUS_TMSX o - - - - - -
SUS_TMSY o - - - - - -