[Edit: Originally thought this was a Guardian Issue, but we later found out it was due to the h1boot1 computer.]
New behavior to me tonight occurred when attempting an INITIAL ALIGNMENT. Everything was fine up until INIT ALIGN wanted to run the MICH BRIGHT step. This step runs a DOWN command for ALIGN_IFO, but then in an early step of the DOWN code it repeatedly gets an connection error (for PRC1_P). After this, ALIGN_IFO goes into a FAULT state & a continuous loop. Below is the log:
2019-09-11_09:24:03.821637Z ALIGN_IFO REQUEST: MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822289Z ALIGN_IFO calculating path: DOWN->MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822719Z ALIGN_IFO new target: PREP_FOR_MICH
2019-09-11_09:24:06.469708Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:24:06.824697Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:24:06.830620Z ALIGN_IFO REQUEST: DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO calculating path: DOWN->DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO new target: DOWN
2019-09-11_09:24:08.924272Z ALIGN_IFO [DOWN.main] USERMSG 0: EZCA CONNECTION ERROR: Did not observe effect of writing value to switch channel ASC-PRC1_P_SW1R within EZCA_TIMEOUT (2.0s).
2019-09-11_09:24:08.949899Z ALIGN_IFO EZCA CONNECTION ERROR. attempting to reestablish...
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: State method raised an EzcaConnectionError exception.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: Current state method will be rerun until the connection error clears.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-09-11_09:26:16.909062Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:17.160830Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:29.469452Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:29.721182Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:42.038308Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:42.290377Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
....
I then gave up on ALIGN_IFO (I took INIT ALIGN to DOWN a while ago), and moved back to ISC_LOCK and ran a DOWN, but now it looks like it's having the same issue. For isc lock it's listing a connection error...with a different channel & can't perform a down and is stuck in a loop.
Not sure what I can do at this point.......will keep investigating Guardian Land. :-/
Marking this DOWN TIME as CORRECTIVE MAINTENANCE since I can no longer run an alignment...or even return to locking apparently.
Here is the error I get with ISC LOCK when I try to run a DOWN: (and this was after taking OP to STOP, letting it complete, and then taking OP to EXEC)
H1:LSC-REFLBIAS_SW2 => 0
2019-09-11_09:38:48.097459Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => ON: FM9, FM3
2019-09-11_09:38:50.274110Z ISC_LOCK [DOWN.main] USERMSG 3: EZCA CONNECTION ERROR: Did not observe effect of writing value to switch channel SUS-ETMX_M0_LOCK_P_SW1R within EZCA_TIMEOUT (2.0s).
2019-09-11_09:38:50.277456Z ISC_LOCK EZCA CONNECTION ERROR. attempting to reestablish...
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: State method raised an EzcaConnectionError exception.
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: Current state method will be rerun until the connection error clears.
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-09-11_09:38:50.618404Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:50.870580Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:51.122586Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:51.374299Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:51.626122Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DSOFT_P => OFF: INPUT, OFFSET
2019-09-11_09:38:51.888774Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DSOFT_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:52.145125Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CSOFT_P => OFF: INPUT, OFFSET
2019-09-11_09:38:52.396948Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CSOFT_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:52.401265Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW1 => 256
2019-09-11_09:38:52.401855Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW2 => 80
2019-09-11_09:38:52.402348Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => OFF: FM1, FM2, FM3, FM4, FM5, FM6, FM7, FM8, FM9, FM10
2019-09-11_09:38:52.402773Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW1 => 0
2019-09-11_09:38:52.403250Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW2 => 0
2019-09-11_09:38:52.403250Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => ON: FM9, FM3
2019-09-11_09:38:54.839541Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:55.091397Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:55.343344Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:55.595399Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_Y => OFF: INPUT, OFFSET
Sent out texts of help to Jenne & Jamie.
Jamie luckily was in Europe so it's morning time for him. He's helping me look through the issues currently.
Just for future reference, this was an issue with h1boot1, not guardian.
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=51883