Reports until 08:39, Thursday 16 October 2014
H1 SEI
hugh.radkins@LIGO.ORG - posted 08:39, Thursday 16 October 2014 - last comment - 10:34, Thursday 16 October 2014(14478)
All Seismic Guardians Restarted

At the request of Jameson, I had done the output HAMs on Monday and this morning I completed all the others.  None of the platforms were disturbed so this was completely non-invasive.  In general though, the restarted node switched to EXEC with a request of NONE.  Returning the mode to MANAGED and the request to whatever put things back with no fuss.  The Chamber manager typically lost communication with the subordinate but I always restarted it last and all finished in good shape.  I say "In general...and ...typically..." because the behaviour was not universally consistent.

Comments related to this report
jameson.rollins@LIGO.ORG - 09:23, Thursday 16 October 2014 (14479)

The SEI guardian systems *should* be completely robust against a "blind" restart of all nodes in a particular chamber, e.g.:

guardctrl restart {SEI,ISI,HPI}_<chamber>*

If the manager comes back up before the subordinates, that's ok because it will recover from the transient NO CONNECT error condition once the subordinate does come back up.

The subordinates also do come up in EXEC, but the manager will handle setting them to MANAGED once they resolved their current state.

If such a fully blind restart of all nodes in a chamber doesn't come up cleanly, please let me know and we can try to fix the problem.

hugh.radkins@LIGO.ORG - 10:00, Thursday 16 October 2014 (14481)

Jameson--Are you saying restart all the guardians for a chamber in a single command?  I did each alone.

jameson.rollins@LIGO.ORG - 10:34, Thursday 16 October 2014 (14483)

I'm saying that you can, with the command I included above, and that everything should still come up fine.

I've been meaning to look into a way to make the single-chamber restart a little easier.