Displaying report 1-1 of 1.
Reports until 13:53, Wednesday 26 June 2024
H1 OpsInfo
thomas.shaffer@LIGO.ORG - posted 13:53, Wednesday 26 June 2024 (78685)
Small change to ALS guardians to avoid last night's issue

Last night the ALS_XARM node was stuck in a loop of tryin to go to increase flashes but Beckhoff was reporting some errors (PDH, PLL, others) so the node would jump to fault state (see Oli and Ryan C's alog78669). There were two issues I found: the unstall decorator that re-requested the node to jump out of FAULT only looks for Beckhoff error messages which might be temporarily not present, and the FAULT state has a jump transition in it to allow us to go to UNLOCKED. I have three possible solutions for this:

  1. Line 208 of ALS_ARM.py forces a jump transition to UNLOCKED while in the FAULT state if the target state is unlocked. Based off of a comment in the code there, I believe the intent was to allow us to still go to unlocked if ALS is shuttered. This isn't desireable because I can't really see another state that would be the target, so it will always jump out of this FAULT state. Changing this line from looking at the target state to the request state should allow it to stay in fault unless we want it to go to unlocked.
  2. Make the unstall node decorator smarter looking at more than just if there are any node notifications. In this case the notifications would only come from Beckhoff error messages, so this isn't good enough at times since the messages can flash by or temporarily go away.
  3. Create a state just before Increase_Flashes that does a check of the ALS Beckhoff errors and will wait here until they are cleared. I worry that this will not be a clear enough indicator that we are in an error state though. It also could create a way to get stuck here and force more user intervention.

For now we'll try option #1 and see how that works. If we need to make it more sophisitcated we can test it out. The code with this change has been saved and will be reloaded when we drop out of Observing.

Images attached to this report
Displaying report 1-1 of 1.