Displaying report 1-1 of 1.
Reports until 02:35, Wednesday 11 September 2019
H1 CDS (CDS, GRD)
corey.gray@LIGO.ORG - posted 02:35, Wednesday 11 September 2019 - last comment - 09:00, Wednesday 11 September 2019(51880)
h1boot1 Causing Guardian Node To Have EZCA CONNECTION Issues

[Edit:  Originally thought this was a Guardian Issue, but we later found out it was due to the h1boot1 computer.]

New behavior to me tonight occurred when attempting an INITIAL ALIGNMENT.  Everything was fine up until INIT ALIGN wanted to run the MICH BRIGHT step.  This step runs a DOWN command for ALIGN_IFO, but then in an early step of the DOWN code it repeatedly gets an connection error (for PRC1_P).  After this, ALIGN_IFO goes into a FAULT state & a continuous loop.  Below is the log:

2019-09-11_09:24:03.821637Z ALIGN_IFO REQUEST: MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822289Z ALIGN_IFO calculating path: DOWN->MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822719Z ALIGN_IFO new target: PREP_FOR_MICH
2019-09-11_09:24:06.469708Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:24:06.824697Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:24:06.830620Z ALIGN_IFO REQUEST: DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO calculating path: DOWN->DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO new target: DOWN
2019-09-11_09:24:08.924272Z ALIGN_IFO [DOWN.main] USERMSG 0: EZCA CONNECTION ERROR: Did not observe effect of writing value to switch channel ASC-PRC1_P_SW1R within EZCA_TIMEOUT (2.0s).
2019-09-11_09:24:08.949899Z ALIGN_IFO EZCA CONNECTION ERROR. attempting to reestablish...
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: State method raised an EzcaConnectionError exception.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: Current state method will be rerun until the connection error clears.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-09-11_09:26:16.909062Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:17.160830Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:29.469452Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:29.721182Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:42.038308Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:42.290377Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT

....

I then gave up on ALIGN_IFO (I took INIT ALIGN to DOWN a while ago), and moved back to ISC_LOCK and ran a DOWN, but now it looks like it's having the same issue.  For isc lock it's listing a connection error...with a different channel & can't perform a down and is stuck in a loop.

Not sure what I can do at this point.......will keep investigating Guardian Land.  :-/

Marking this DOWN TIME as CORRECTIVE MAINTENANCE since I can no longer run an alignment...or even return to locking apparently.

Comments related to this report
corey.gray@LIGO.ORG - 02:45, Wednesday 11 September 2019 (51881)

Here is the error I get with ISC LOCK when I try to run a DOWN:  (and this was after taking OP to STOP, letting it complete, and then taking OP to EXEC)

H1:LSC-REFLBIAS_SW2 => 0
2019-09-11_09:38:48.097459Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => ON: FM9, FM3
2019-09-11_09:38:50.274110Z ISC_LOCK [DOWN.main] USERMSG 3: EZCA CONNECTION ERROR: Did not observe effect of writing value to switch channel SUS-ETMX_M0_LOCK_P_SW1R within EZCA_TIMEOUT (2.0s).
2019-09-11_09:38:50.277456Z ISC_LOCK EZCA CONNECTION ERROR. attempting to reestablish...
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: State method raised an EzcaConnectionError exception.
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: Current state method will be rerun until the connection error clears.
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-09-11_09:38:50.618404Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:50.870580Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:51.122586Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:51.374299Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:51.626122Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DSOFT_P => OFF: INPUT, OFFSET
2019-09-11_09:38:51.888774Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DSOFT_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:52.145125Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CSOFT_P => OFF: INPUT, OFFSET
2019-09-11_09:38:52.396948Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CSOFT_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:52.401265Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW1 => 256
2019-09-11_09:38:52.401855Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW2 => 80
2019-09-11_09:38:52.402348Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => OFF: FM1, FM2, FM3, FM4, FM5, FM6, FM7, FM8, FM9, FM10
2019-09-11_09:38:52.402773Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW1 => 0
2019-09-11_09:38:52.403250Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW2 => 0
2019-09-11_09:38:52.403250Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => ON: FM9, FM3
2019-09-11_09:38:54.839541Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:55.091397Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:55.343344Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:55.595399Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_Y => OFF: INPUT, OFFSET
 

corey.gray@LIGO.ORG - 02:54, Wednesday 11 September 2019 (51882)

Sent out texts of help to Jenne & Jamie.

Jamie luckily was in Europe so it's morning time for him.  He's helping me look through the issues currently.

thomas.shaffer@LIGO.ORG - 09:00, Wednesday 11 September 2019 (51906)

Just for future reference, this was an issue with h1boot1, not guardian.

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=51883

Displaying report 1-1 of 1.