Today I saw, via my monitor programs, a freeze up of EPICS data from LSC and ASC between the times 20:27:25 and 20:27:40 UTC, a duration of 15 seconds. My strip tool showed flat-line for this time. I was able to get the load-average on a variety of corner station front ends around this time, they were all elevated. Looking at the Guardian ISC_DRMI node logs and checking writes against the DAQ data I see that systems other than LSC, ASC appear to be frozen. For example ISC_DRMI attempted to change the H1:SUS-IM4_M1_LOCK_P_GAIN during this period. The modifications which were attempted during the freeze time did not show up in the DAQ data.

To see if IPC was playing a role in this, I started a third monitor on h1susauxb123 (no IPC). At 17:23 as I was writing this alog we got another event, which was seen by all three ASC, LSC and SUSAUXB123.

Investigation continues.