- 4:06utc Verbal Alarm notes we are OUT of OBSERVING.
- Observation Bit said there were (3) NOT OK Guardian Nodes: SDF, ISC_LOCK, & ALIGN_IFO.
- Accepted Diffs [hindsight: should have held off on this...assumed I would not be able to fix Squeezer]
- ALIGN_IFO pointed to the Squeezer with its messages (also the SDF diffs were also Squeezer related)
- SQZ_MANAGER was in PREP_IFO_SQZ
- SQZ_LO_LR was in DOWN
- Took SQZ_LO_LR to it's nominal LOCKED_OMC & I think it eventually made it there (without me needing to select intermediate step). So this was GOOD!
- Tried taking SQZ_MANAGER to it's nominal SQZ_ASC_ON (from PREP_IFO_SQZ), but it would not go....think it was stuck in an intermediate step. I then selected the LOCKING_3MHZ_OMC state and then it went all the way to SQZ_ASC_ON: it was GOOD!
- Now only "not ok" node was the SDF, due to me ACCEPTING the diffs earlier. So I accepted the new diffs and then....
- 4:11utc Back to OBSERVING
attached are photos of the Guardian Nodes which I took action on.
Does ISC_LOCK not take SQZ_MANAGER to its nominal state anymore? Or did ISC_LOCK requested SQZ_MANAGER nominal state and failed? What's in Inject_squeezing state in the ISC_LOCK?
If ISC_LOCK requested SQZ_MANAGER to go to nominal state but failed for got stuck, can I have more details of how it fails next time this happens? Including guardian log, screenshots, would be helpful for improving squeezer automation.
ISC_LOCK takes the SQZ _Manager to its nominal state during normal locking, but with what happened tonight (would we call this the "Squeezer losing lock" or something else?), the ISC_LOCK & ALIGN_IFO were transitioned to "NOT OK" states, when the Squeezer did what it did.
Luckily I was able to get the SQZ Guardian nodes mentioned above to their nominal state. Otherwise, ISC_LOCK & ALIGN_IFO would have remained "Not OK" and would prevent OBSERVING. Alternatively, if one is comfortable with changing the ISC_LOCK & ALIGN_IFO scripts, I reckon there must be a way to break this remaining connection they have to the squeezer.
Anyway, when I looked at SQZ_MANAGER and SQZ_LO_LR, they were not actively doing anything according to their logs. SQZ_LO_LR was down and staying there. After I (1) tried getting it to its nominal state, and then (2) getting SQZ_MANAGER to its nominal state, then ISC_LOCK & ALIGN_IFO were back to being OK.
Since this was easy to recover from, maybe it's good to have it go down in this way? Mainly because recovery was fairly straightforward? And we also get a heads up that Squeezer's state has changed.
Oh, and good idea about posting more info. Attached is screenshot showing all the logs of pertinent guardian nodes from this event. Now that I look at them, it looks like ISC_LOCK & ALIGN_IFO probably did not take us out...but they point the way to SQZ since they give the messages when they were listed as "not ok" (but their logs don't show anything until the very end when I go back to Observing).
I reckon it's the SDF diffs from the squeezer which took us out of OBSERVING.
Hope this helps!
Oh, and the other thing from the screenshots is that you can see ISC_LOCK definitely does not do anything to the SQZ_MANAGER when this happened. (but ISC_LOCK & align_ifo were listed as NOT OK....and this would not have let us return to Observing until we fixed their situation with the SQZ or edited their scripts).