Rolf asked for some information on the cpu cycle count through the glitch. I've two plots of the cpu meter from h1iopsusb123, h1iopsush2a, h1ioplsc0, h1iopasc0. The first plot is in dataviewer and gives the mean round the glitch, the second is from ndscope is is raw data (cpu_meter + cpu_meter_max).
A summary from nds_query giving raw numbers:
Data for 5 seconds starting at GPS: 1222517040
Channel type nWords units
H1:FEC-28_CPU_METER int_4 80
Channel: H1:FEC-28_CPU_METER type: int_4 nWords: 80
0 5 5 5 5 5 5 5 5
8 5 5 5 5 5 5 5 5
16 5 5 5 5 5 5 5 5
24 5 5 5 5 5 5 5 5
32 5 5 5 5 5 5 5 5
40 5 5 5 5 5 5 5 5
48 5 20228 20228 20228 20228 20228 20228 20228
56 20228 20228 20228 20228 20228 20228 20228 20228
64 20228 6 6 6 6 6 6 6
72 6 6 6 6 6 6 6 6
Images attached to this comment
david.barker@LIGO.ORG - 14:28, Tuesday 02 October 2018 (44300)
Took h1psl0 out of the Dolphin fabric before restarting the models. Disabled its IXS600 switch port and pulled the IX cable at the rear of the computer. This should insulate the PSL from any further corner station Dolphin issues. We will re-attach when the system is more stable.
Replaced h1asc0's Dolphin cable (this machine reported a Dolphin Gen1 issue). Powered h1asc0 down, replaced 2m cable with new 1m cable temporarily ran directly from IX600 switch C0 port 8 to cpu.
Systematically restarted all models on all computers with errors.
When h1iopseib1 was starting, the computer froze. SSH session was unresponsive, local console was dark. Disabled IXS600 switch port and power cycled computer.
There were DAC errors on h1seib2 and h1seih16, so these IOP models were also restarted.
By the end of the day, h1seib3 was the only Dolphin'ed corner machine which did not get anything restarted on it.
I've started a wiki page to overview these recent front end issues: https://cdswiki.ligo-wa.caltech.edu/wiki/FrontEndCrash
david.barker@LIGO.ORG - 14:31, Tuesday 02 October 2018 (44301)
here is the CDS overview before we started the recovery process.
Rolf asked for some information on the cpu cycle count through the glitch. I've two plots of the cpu meter from h1iopsusb123, h1iopsush2a, h1ioplsc0, h1iopasc0. The first plot is in dataviewer and gives the mean round the glitch, the second is from ndscope is is raw data (cpu_meter + cpu_meter_max). A summary from nds_query giving raw numbers: Data for 5 seconds starting at GPS: 1222517040 Channel type nWords units H1:FEC-28_CPU_METER int_4 80 Channel: H1:FEC-28_CPU_METER type: int_4 nWords: 80 0 5 5 5 5 5 5 5 5 8 5 5 5 5 5 5 5 5 16 5 5 5 5 5 5 5 5 24 5 5 5 5 5 5 5 5 32 5 5 5 5 5 5 5 5 40 5 5 5 5 5 5 5 5 48 5 20228 20228 20228 20228 20228 20228 20228 56 20228 20228 20228 20228 20228 20228 20228 20228 64 20228 6 6 6 6 6 6 6 72 6 6 6 6 6 6 6 6Jonathan, Dave:
Recovery process:
Took h1psl0 out of the Dolphin fabric before restarting the models. Disabled its IXS600 switch port and pulled the IX cable at the rear of the computer. This should insulate the PSL from any further corner station Dolphin issues. We will re-attach when the system is more stable.
Replaced h1asc0's Dolphin cable (this machine reported a Dolphin Gen1 issue). Powered h1asc0 down, replaced 2m cable with new 1m cable temporarily ran directly from IX600 switch C0 port 8 to cpu.
Systematically restarted all models on all computers with errors.
When h1iopseib1 was starting, the computer froze. SSH session was unresponsive, local console was dark. Disabled IXS600 switch port and power cycled computer.
There were DAC errors on h1seib2 and h1seih16, so these IOP models were also restarted.
By the end of the day, h1seib3 was the only Dolphin'ed corner machine which did not get anything restarted on it.
I've started a wiki page to overview these recent front end issues: https://cdswiki.ligo-wa.caltech.edu/wiki/FrontEndCrash
Note that there are expected IPC errors on h1psliss and h1lsc following the disconnection of h1psl0 from the Dolphin network.
h1psliss receives two IPC channels from h1lsc (TRX and TRY normalised photodiode signals). Used for ISS third loop, not needed in the short term.
h1lsc receives one IPC channel from h1psldbb (FF_JITTER). Used to add to OMC-DARM signal. Has not had a useful signal for some time.