david.barker@LIGO.ORG - posted 10:27, Tuesday 02 October 2018 - last comment - 14:31, Tuesday 02 October 2018(44289)
Corner station models crash at 05:03PDT this morning
Most models which are Dolphin'ed computers in the corner station crashed at 05:03:44 PDT (12:03:44 UTC) Oct 02 2018.
For most computers, all the user models stopped with ADC timeout errors. For some, not all models crashed.
here is the CDS overview before we started the recovery process.
Rolf asked for some information on the cpu cycle count through the glitch. I've two plots of the cpu meter from h1iopsusb123, h1iopsush2a, h1ioplsc0, h1iopasc0. The first plot is in dataviewer and gives the mean round the glitch, the second is from ndscope is is raw data (cpu_meter + cpu_meter_max). A summary from nds_query giving raw numbers: Data for 5 seconds starting at GPS: 1222517040 Channel type nWords units H1:FEC-28_CPU_METER int_4 80 Channel: H1:FEC-28_CPU_METER type: int_4 nWords: 80 0 5 5 5 5 5 5 5 5 8 5 5 5 5 5 5 5 5 16 5 5 5 5 5 5 5 5 24 5 5 5 5 5 5 5 5 32 5 5 5 5 5 5 5 5 40 5 5 5 5 5 5 5 5 48 5 20228 20228 20228 20228 20228 20228 20228 56 20228 20228 20228 20228 20228 20228 20228 20228 64 20228 6 6 6 6 6 6 6 72 6 6 6 6 6 6 6 6Jonathan, Dave:
Recovery process:
Took h1psl0 out of the Dolphin fabric before restarting the models. Disabled its IXS600 switch port and pulled the IX cable at the rear of the computer. This should insulate the PSL from any further corner station Dolphin issues. We will re-attach when the system is more stable.
Replaced h1asc0's Dolphin cable (this machine reported a Dolphin Gen1 issue). Powered h1asc0 down, replaced 2m cable with new 1m cable temporarily ran directly from IX600 switch C0 port 8 to cpu.
Systematically restarted all models on all computers with errors.
When h1iopseib1 was starting, the computer froze. SSH session was unresponsive, local console was dark. Disabled IXS600 switch port and power cycled computer.
There were DAC errors on h1seib2 and h1seih16, so these IOP models were also restarted.
By the end of the day, h1seib3 was the only Dolphin'ed corner machine which did not get anything restarted on it.
I've started a wiki page to overview these recent front end issues: https://cdswiki.ligo-wa.caltech.edu/wiki/FrontEndCrash
Note that there are expected IPC errors on h1psliss and h1lsc following the disconnection of h1psl0 from the Dolphin network.
h1psliss receives two IPC channels from h1lsc (TRX and TRY normalised photodiode signals). Used for ISS third loop, not needed in the short term.
h1lsc receives one IPC channel from h1psldbb (FF_JITTER). Used to add to OMC-DARM signal. Has not had a useful signal for some time.