david.barker@LIGO.ORG - posted 06:12, Tuesday 09 April 2024 - last comment - 10:48, Wednesday 10 April 2024(77040)
h1omc0 channel hop caused dolphin glitch of SUS and LSC
Oli, Erik, Dave:
Around 04:30 this morning a ADC channel hop on h1iopomc0 caused all of the omc0 models to stop running, and also caused a corner station Dolphin glitch which DACKILLED h1susb123, h1sush34, h1sush56 and h1lsc0.
h1omc0 dmesg:
[Tue Apr 9 04:34:05 2024] rts_cpu_isolator: LIGO code is done, calling regular shutdown code
[Tue Apr 9 04:34:05 2024] h1iopomc0: ERROR - A channel hop error has been detected, waiting for an exit signal.
[Tue Apr 9 04:34:05 2024] h1omcpi: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue Apr 9 04:34:05 2024] h1omc: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
Initially I thought this was an IO Chassis issue, so we power cycled h1omc0 rather than a restart all of its models (my confusion was because this front end only has one Adnaco, and is running the low-noise ADC). This brought h1omc0 back up and running.
We restarted the models on h1lsc0, which cleared the DACKILL.
Oli put the SUS and SEI for BSC1,2,3 and HAM3,4,5,6 into safe, I SWWD bypassed the SEI IOPs and we restarted the models on h1susb123, h1sush34, h1sush56. All came back with no problems.
I cleared the SWWDs, did a DIAG_RESET and cleared the DAQ CRCs.
Handing over to Oli for IFO recovery.
CDS Overview after DIAG_RESET run on all front ends:
Time of OMC crash:
04:32:45 PDT
11:32:45 UTC
1396697583 GPS
FRS30939