As Jim had almost relocked the IFO, we had an epics freeze in the gaurdian state RESONANCE. ISC_LOCK had an epics connection error.
What is the right thing for the operator to do in this situation?
Are these epics freezes becoming more frequent again?
screenshot attached.
epics freezes never fully went away completely and are normally only a few seconds in duration. This morning's SUS ETMX event lasted for 22 seconds which exceeded Guardian's timeout period. To get the outage duration, I second trended H1:IOP-SUS_EX_ADC_DT_OUTMON. Outages are on a computer basis, not model basis, so I have put the IOP Duotone output EPICS channel into the frame as EDCU channels (access via channel access over the network). When these channels are unavailable, the DAQ sets them to be zero.
For this event the time line is (all times UTC)
| 16:17:22 | DAQ shows EPICS has frozen on SUS EX |
| 16:17:27 | Guardian attempts connection |
| 16:17:29 | Guardian reports error, is retrying |
| 16:17:43 | Guardian timesout |
| 16:17:45 | DAQ shows channel is active again |
The investigation of this problem is ongoing, we could bump up the priority if it becomes a serious IFO operations issue.
To be clear, it sounds like there was a lockloss during acquisition that was caused by some kind of EPICS drop out. I see how a lockloss could occur during the NOMINAL lock state just from an EPICS drop out. guardian nodes might go into error, but that shouldn't actually affect the fast IFO controls at all.
Sorry, I meant that I can not see how a guardian EPICS dropout could cause a lock loss during the nominal lock state.