Reports until 21:33, Friday 20 September 2019
david.barker@LIGO.ORG - posted 21:33, Friday 20 September 2019 - last comment - 11:00, Saturday 21 September 2019(52049)
EY timing glitch

We have had a timing at EY, which in turn caused a Dolphin glitch stopping the SUS, SEI and ISC models.

Dmesg on all three machines shows ADC timeouts. All IO Chassis are visible, so it looks like we just need to restart all the models.

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 21:42, Friday 20 September 2019 (52050)

I've opened FRS13623

david.barker@LIGO.ORG - 21:49, Friday 20 September 2019 (52051)

To document the system before I started recovery, I captured all the front end system and Dolphin logs.

I then restarted the models in the order h1susey, h1seiey, h1iscey.

Finally did a global DIAG_RESET and cleared the DAQ CRC errors. All looks nominal.

david.barker@LIGO.ORG - 11:00, Saturday 21 September 2019 (52056)

I've added front end log details to the FRS (they suggest a gradual degradation of EY over a period of 8 minutes, initiated by h1seiey). I've marked the ticket with 3 hours observation time lost, and closed it as resolved (restarting models restored H1).

At the moment there is no obvious cause for the glitch, there was no EY activity at the time.