Sheila and Elenna noticed strange behavior with the ALIGN_IFO node early this afternoon where they were doing the 'OFFLOAD_PRC_ALIGN' state and mistakenly requested 'DOWN' and 'INIT' before the offloading was finished, and the Guardian jumped mid-offload, leaving things in a weird state.
This is not ideal, as the offloading states in this node are meant to be "protected" (redirect=False) and the Guardian should not leave these states unless they return True, even with a redirect request. I've highlighted in the Guardian log below that the initial requests for the 'PRX_LOCKED' and 'DOWN' states were correctly ignored, but a request for 'INIT' prompts an "INIT redirect," which then waits for one second for worker completion, which finishes, then terminates the worker. This restarts the code, which starts the node at 'INIT.'
Is INIT a special state which overrides a protected state? Why did the worker crash in this case? These are questions still under investigation.
ryan.short@cdsws25[~]: guardctrl log ALIGN_IFO -a 1433187576 -b 1433187587
2025-06-05_19:39:18.240855Z ALIGN_IFO [OFFLOAD_PRC_ALIGNMENT.main] ezca: H1:SUS-RM2_M1_LOCK_Y => OFF: INPUT
2025-06-05_19:39:18.241603Z ALIGN_IFO [OFFLOAD_PRC_ALIGNMENT.main] ezca: H1:SUS-RM2_M1_LOCK_Y_GAIN => 0
2025-06-05_19:39:18.241978Z ALIGN_IFO [OFFLOAD_PRC_ALIGNMENT.main] ezca: H1:SUS-RM2_M1_OPTICALIGN_Y_TRAMP => 10
2025-06-05_19:39:18.242418Z ALIGN_IFO [OFFLOAD_PRC_ALIGNMENT.main] ezca: H1:SUS-RM2_M1_OPTICALIGN_Y_OFFSET => -836.7946
2025-06-05_19:39:18.242518Z ALIGN_IFO [OFFLOAD_PRC_ALIGNMENT.main] waiting for ramps to finish...
2025-06-05_19:39:22.706911Z ALIGN_IFO REQUEST: PRX_LOCKED
2025-06-05_19:39:22.708302Z ALIGN_IFO calculating path: OFFLOAD_PRC_ALIGNMENT->PRX_LOCKED
2025-06-05_19:39:22.709109Z ALIGN_IFO new target: DOWN
2025-06-05_19:39:22.709931Z ALIGN_IFO GOTO REDIRECT IGNORED: redirect=False for state OFFLOAD_PRC_ALIGNMENT
2025-06-05_19:39:25.447181Z ALIGN_IFO REQUEST: DOWN
2025-06-05_19:39:25.448398Z ALIGN_IFO calculating path: OFFLOAD_PRC_ALIGNMENT->DOWN
2025-06-05_19:39:25.448771Z ALIGN_IFO GOTO REDIRECT IGNORED: redirect=False for state OFFLOAD_PRC_ALIGNMENT
2025-06-05_19:39:27.193260Z ALIGN_IFO REQUEST: INIT
2025-06-05_19:39:27.194773Z ALIGN_IFO INIT redirect
2025-06-05_19:39:27.194773Z ALIGN_IFO REDIRECT requested, timeout in 1.000 seconds
2025-06-05_19:39:27.197640Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.269261Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.328470Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.395177Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.448378Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.516500Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.581382Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.638191Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.709034Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.773263Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.828997Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.881440Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:27.949812Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:28.014862Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:28.077071Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:28.132765Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:28.203338Z ALIGN_IFO REDIRECT wait for worker completion...
2025-06-05_19:39:28.204112Z ALIGN_IFO REDIRECT timeout reached. worker terminate and reset...
2025-06-05_19:39:28.214558Z ALIGN_IFO worker terminated
2025-06-05_19:39:28.227354Z ALIGN_IFO W: initialized
2025-06-05_19:39:28.254897Z ALIGN_IFO W: EZCA v1.4.0
2025-06-05_19:39:28.254897Z ALIGN_IFO W: EZCA CA prefix: H1:
2025-06-05_19:39:28.254897Z ALIGN_IFO W: ready
2025-06-05_19:39:28.256840Z ALIGN_IFO worker ready
2025-06-05_19:39:28.261318Z ALIGN_IFO EDGE: OFFLOAD_PRC_ALIGNMENT->INIT
2025-06-05_19:39:28.262024Z ALIGN_IFO calculating path: INIT->DOWN
2025-06-05_19:39:28.262914Z ALIGN_IFO new target: DOWN
2025-06-05_19:39:28.273539Z ALIGN_IFO executing state: INIT (0)
2025-06-05_19:39:28.274460Z ALIGN_IFO [INIT.enter]
2025-06-05_19:39:28.275951Z Warning: Duplicate EPICS CA Address list entry "10.101.0.255:5064" discarded
2025-06-05_19:39:28.316393Z ALIGN_IFO REQUEST: PRX_LOCKED
2025-06-05_19:39:28.322564Z ALIGN_IFO calculating path: INIT->PRX_LOCKED