We have had a timing at EY, which in turn caused a Dolphin glitch stopping the SUS, SEI and ISC models.
Dmesg on all three machines shows ADC timeouts. All IO Chassis are visible, so it looks like we just need to restart all the models.
I've opened FRS13623
To document the system before I started recovery, I captured all the front end system and Dolphin logs.
I then restarted the models in the order h1susey, h1seiey, h1iscey.
Finally did a global DIAG_RESET and cleared the DAQ CRC errors. All looks nominal.
I've added front end log details to the FRS (they suggest a gradual degradation of EY over a period of 8 minutes, initiated by h1seiey). I've marked the ticket with 3 hours observation time lost, and closed it as resolved (restarting models restored H1).
At the moment there is no obvious cause for the glitch, there was no EY activity at the time.