TITLE: 10/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY:
Strange instabilities causing locklosses were the order of the evening and then the Guardian went South. Stefan was the only commissioner here at the outset and he decided he had had enough. Nutsinee is here for Owl shift but there doesn't seem to be much reason to stay now. Heading home.
LOG:
01:26 Switched ISI blends to EQ v2. EQ Z axis reached up to ≈1 µm/s. ISC_LOCK Guardian set to DOWN waiting for ringdown.
03:43 Switched ISI config back to nominal state - WINDY
4:52 Guardian had an error party - Connection errors 1st, ALS_YARM, then ALS_DIFF and then ISC_LOCK
ISC_LOCK, ALS_DIFF, and IFO were all showing connection errors because they lost connection to the ALS_YARM guardian, which had unceremoniously died:
As the error indicates, the worker process apparently died without explanation, which is not at all nice.
I restarted the ALS_YARM node with guardctrl and it came back up fine. The rest of the nodes recovered their connections soon after. As of right now all nodes appear to be funtioning normally.
This "worker exited unexpectedly" error isn't one I've seen much at all, so I'm very curious what could have caused it.
Logged an FRS (6338) for this. Jamie was able to get everything back before ~2am, but since we canceled the OWL shift (in this pre-ER10 epoch, operators are informed to NOT wake up help in the middle of the night), instead of being down 4hrs it was more like 10hrs.
Since this issue was fixed, the above FRS can now be closed.