aLIGO LHO Logbook

H1 General

corey.gray@LIGO.ORG - posted 14:18, Sunday 22 December 2019 - last comment - 10:03, Monday 23 December 2019(54053)

H1 Drops Out Of OBSERVING....but why??

At 21:31:21utc, H1 was dropped out of OBSERVING. I did a quick scan and did not see anything obvious. There have been several Guardian Nodes which are NOT green: ISI (for a TRamp, Violins (from violin activities), Inj Trans (it's waiting to go back to Success state), etc, but these remained as they have been. A few notes:

The Intent Bit was able to go back to OBSERVING shortly after dropping out,
Was not able to catch this drop in time to bring up the list of "Not OK" Guardian Nodes--so no light shd there.
Trended Diff Count for dozens of SDF nodes, but saw nothing change from 0 for all of the ones I trended.
- Previous examples I remember of quick changes which dropped us out of Observing were related to TCS & Slow Controls/Beckhoff. Then there is also Squeezer, but that is usually for when the Squeezer drops out of lock. None of these were smoking guns on this drop.

Will keep the Guardian and SDF medm windows open in case there's another drop---so I can hopefully catch what flashes and sheds ligth as to the culprit here.

21:33:04utc Went back to OBSERVING

Comments related to this report

keita.kawabe@LIGO.ORG - 10:03, Monday 23 December 2019 (54064)

Link

You can always look at the guardian log of any node using guardctrl command (see below).

In this case, TCS ETMX ring heater Beckhof was unresponsive for a few seconds though I don't know why.

~$ guardctrl log --after '2019/12/22 21:20:00 UTC' --before '2019/12/22 21:36:00 UTC' IFO
2019-12-22_21:31:21.524583Z IFO [OBSERVE.run] USERMSG 0: TCS_ETMX_RH_PWR: has notification
2019-12-22_21:31:21.534630Z IFO [OBSERVE.run] USERMSG 5: waiting for node: TCS_ETMX_RH_PWR
2019-12-22_21:31:21.568637Z IFO JUMP target: DROP_OBSERVE
2019-12-22_21:31:21.569218Z IFO [OBSERVE.exit]
2019-12-22_21:31:21.569218Z IFO STALLED
2019-12-22_21:31:21.630609Z IFO JUMP: OBSERVE->DROP_OBSERVE
2019-12-22_21:31:21.631284Z IFO calculating path: DROP_OBSERVE->OBSERVE
2019-12-22_21:31:21.631284Z IFO new target: WAITING_FOR_NODES
2019-12-22_21:31:21.644872Z IFO executing state: DROP_OBSERVE (5)
2019-12-22_21:31:21.715324Z IFO [DROP_OBSERVE.run] USERMSG 0: TCS_ETMX_RH_PWR: has notification
2019-12-22_21:31:21.716172Z IFO [DROP_OBSERVE.run] USERMSG 1: VIOLIN_DAMPING: has notification
2019-12-22_21:31:21.719072Z IFO [DROP_OBSERVE.run] USERMSG 2: SEI_ITMX: has notification
2019-12-22_21:31:21.720278Z IFO [DROP_OBSERVE.run] USERMSG 3: ISC_LOCK: has notification
2019-12-22_21:31:21.720707Z IFO [DROP_OBSERVE.run] USERMSG 4: ISI_ITMX_ST2: has notification
2019-12-22_21:31:21.772147Z IFO JUMP target: WAITING_FOR_NODES
2019-12-22_21:31:21.772828Z IFO [DROP_OBSERVE.exit]
2019-12-22_21:31:21.772828Z IFO STALLED
2019-12-22_21:31:21.822498Z IFO JUMP: DROP_OBSERVE->WAITING_FOR_NODES
2019-12-22_21:31:21.823140Z IFO calculating path: WAITING_FOR_NODES->OBSERVE
2019-12-22_21:31:21.823140Z IFO new target: READY
2019-12-22_21:31:21.824140Z IFO executing state: WAITING_FOR_NODES (20)
2019-12-22_21:31:21.898601Z IFO [WAITING_FOR_NODES.run] USERMSG 0: TCS_ETMX_RH_PWR: has notification
2019-12-22_21:31:21.899459Z IFO [WAITING_FOR_NODES.run] USERMSG 1: VIOLIN_DAMPING: has notification
2019-12-22_21:31:21.903659Z IFO [WAITING_FOR_NODES.run] USERMSG 2: SEI_ITMX: has notification
2019-12-22_21:31:21.904770Z IFO [WAITING_FOR_NODES.run] USERMSG 3: ISC_LOCK: has notification
2019-12-22_21:31:21.905071Z IFO [WAITING_FOR_NODES.run] USERMSG 4: ISI_ITMX_ST2: has notification
2019-12-22_21:31:21.908626Z IFO [WAITING_FOR_NODES.run] USERMSG 5: waiting for node: TCS_ETMX_RH_PWR
2019-12-22_21:31:23.380593Z IFO JUMP target: READY
2019-12-22_21:31:23.381379Z IFO [WAITING_FOR_NODES.exit]
2019-12-22_21:31:23.382120Z IFO STALLED
2019-12-22_21:31:23.453002Z IFO JUMP: WAITING_FOR_NODES->READY
2019-12-22_21:31:23.454064Z IFO calculating path: READY->OBSERVE
2019-12-22_21:31:23.454512Z IFO new target: OBSERVE
2019-12-22_21:31:23.455691Z IFO executing state: READY (50)
2019-12-22_21:31:23.522106Z IFO [READY.run] USERMSG 0: VIOLIN_DAMPING: has notification
2019-12-22_21:31:23.527690Z IFO [READY.run] USERMSG 1: SEI_ITMX: has notification
2019-12-22_21:31:23.529655Z IFO [READY.run] USERMSG 2: ISC_LOCK: has notification
2019-12-22_21:31:23.530093Z IFO [READY.run] USERMSG 3: ISI_ITMX_ST2: has notification
2019-12-22_21:33:03.893311Z IFO REQUEST: OBSERVE
2019-12-22_21:33:03.894002Z IFO STALL cleared
2019-12-22_21:33:03.894002Z IFO calculating path: READY->OBSERVE
2019-12-22_21:33:03.961278Z IFO EDGE: READY->OBSERVE
2019-12-22_21:33:04.018526Z IFO calculating path: OBSERVE->OBSERVE
2019-12-22_21:33:04.018526Z IFO executing state: OBSERVE (100)
2019-12-22_21:33:04.032461Z IFO [OBSERVE.run] USERMSG 0: VIOLIN_DAMPING: has notification
2019-12-22_21:33:04.036775Z IFO [OBSERVE.run] USERMSG 1: SEI_ITMX: has notification
2019-12-22_21:33:04.037957Z IFO [OBSERVE.run] USERMSG 2: ISC_LOCK: has notification
2019-12-22_21:33:04.038344Z IFO [OBSERVE.run] USERMSG 3: ISI_ITMX_ST2: has notification

~$ guardctrl log --after '2019/12/22 21:20:00 UTC' --before '2019/12/22 21:36:00 UTC' TCS_ETMX_RH_PWR
2019-12-22_21:31:21.391292Z CA.Client.Exception...............................................
2019-12-22_21:31:21.392061Z     Warning: "Virtual circuit unresponsive"
2019-12-22_21:31:21.392607Z     Context: "h1ecatx1.cds.ligo-wa.caltech.edu:5064"
2019-12-22_21:31:21.393068Z     Source File: ../tcpiiu.cpp line 947
2019-12-22_21:31:21.393504Z     Current Time: Sun Dec 22 2019 13:31:21.391261577
2019-12-22_21:31:21.393926Z ..................................................................
2019-12-22_21:31:21.394372Z TCS_ETMX_RH_PWR [NOMINAL.run] USERMSG 0: CONNECTION ERRORS. see SPM DIFFS for dead channels
2019-12-22_21:31:21.463499Z TCS_ETMX_RH_PWR EZCA CONNECTION ERROR. attempting to reestablish...
2019-12-22_21:31:21.463930Z TCS_ETMX_RH_PWR CERROR: State method raised an EzcaConnectionError exception.
2019-12-22_21:31:21.463930Z TCS_ETMX_RH_PWR CERROR: Current state method will be rerun until the connection error clears.
2019-12-22_21:31:21.463930Z TCS_ETMX_RH_PWR CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-12-22_21:31:23.267678Z TCS_ETMX_RH_PWR connections reestablished

Edit later: If you're in front of the workstation when something happens, you don't have to use command line, you can open MEDM screen for whatever node (in this case IFO node by pressing "GRD IFO" button at the top of the guardian overview screen) and then press the "log" button to see what's going on in real time.

Images attached to this comment