At 21:00:16 most of the front end systems showed multiple IPC errors. This corresponds to the lock loss.
Around this time the DAQ restarted itself. The DAQ then did a second restart at 21:36.
At the time of writing (23:15) the DAQ has been running for 1hr 39mins.
The CDS overview before any recovery is attached. Note that it appears that only models with Dolphin IPC receivers are showing IPC errors, for corner stations and end stations.
There were two NDS issues; NDS1 was frozen with an uptime of 2132 (time of second DAQ crash) and NDS0 was running for 1 second longer than DC0.
For immediate recovery I issued a DIAG_RESET to verify all the IPC errors were from transients and not currently active (this was verified) and I restarted both NDS1 (to unstick it) and NDS0 (to ensure it was synced with DC0).
Here is a time machine capture of the IPC errors at 21:15 (after the 21:00 crash and when DAQ was back) and 21:40 (after the 21:36 crash, when DAQ was back and before I issued the DIAG_RESET at 21:42)
Note that all types of IPC are erroring (SHMEM, Dolphin, X-arm, Y-arm). The errors are geographically consistent (EX has only X-arm, EY has only Y-arm, CS has both).
In the morning I'll do a channel-by-channel scan to see if they all errored at the same time.
Dolphin dis_diag error scan
I ran a dis_diag scan on the Dolphined frontends. The only two FEs which showed errors were h1lsc0 and h1asc0, but the last scan was ran at 11am Tuesday and we then rebooted these two machines at 1pm Tuesday for the IO Chassis work, so these could be errors from that time.
The errors have been cleared, I'll rescan in the morning
h1lsc0 Uncorrectable Link Errors: 9
h1asc0 Uncorrectable Link Errors: 9
Test stand DAQ restarted at the same times plus two seconds for the same reason: data starvation from front ends. It's a separate network, isolated from CDS. The common component is the timing system. A timing discontinuity from the IOP models would explain all systems, including IPC errors and interruption of DAQ data.
Test stand daq log
Nov 02 21:00:19 x1daqdc0 daqd[30510]: Dropped data from shmem or received 0 dcus; gps now = 1383019237, 0; was = 1383019228, 13; dcu count = 5
Nov 02 21:00:19 x1daqdc0 daqd[30510]: expected gps = 1383019228
Nov 02 21:00:19 x1daqdc0 daqd[30510]: expected cycle = 14
Nov 02 21:00:19 x1daqdc0 daqd[30510]: expected nano = 62500013
Nov 02 21:00:19 x1daqdc0 daqd[30510]: first 5 dcuids seen
Nov 02 21:00:19 x1daqdc0 daqd[30510]: saw dcu 166 - gps: 1383019237 nano: 0 cycle: 0
Nov 02 21:00:19 x1daqdc0 daqd[30510]: saw dcu 161 - gps: 1383019237 nano: 0 cycle: 0
Nov 02 21:00:19 x1daqdc0 daqd[30510]: saw dcu 36 - gps: 1383019237 nano: 0 cycle: 0
Nov 02 21:00:19 x1daqdc0 daqd[30510]: saw dcu 34 - gps: 1383019237 nano: 0 cycle: 0
Nov 02 21:00:19 x1daqdc0 daqd[30510]: saw dcu 35 - gps: 1383019237 nano: 0 cycle: 0
Nov 02 21:00:20 x1daqdc0 kernel: dqprod[30534]: segfault at 7f11acedb9d0 ip 00007f12313ac2e8 sp 00007f116b9bac40 error 4 in libpthread-2.28.so[7f12313a9000+f000]
Nov 02 21:00:20 x1daqdc0 kernel: Code: 00 00 41 56 41 55 41 54 55 53 48 83 ec 40 64 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 48 85 ff 0f 84 1b 01 00 00 48 89 fb <8b> bf d0 02
Nov 02 21:00:20 x1daqdc0 systemd[1]: rts-daqd.service: Main process exited, code=killed, status=11/SEGV
Nov 02 21:00:20 x1daqdc0 systemd[1]: rts-daqd.service: Failed with result 'signal'.
Nov 02 21:36:08 x1daqdc0 daqd[5453]: Dropped data from shmem or received 0 dcus; gps now = 1383021386, 0; was = 1383021377, 13; dcu count = 5
Nov 02 21:36:08 x1daqdc0 daqd[5453]: expected gps = 1383021377
Nov 02 21:36:08 x1daqdc0 daqd[5453]: expected cycle = 14
Nov 02 21:36:08 x1daqdc0 daqd[5453]: expected nano = 62500013
Nov 02 21:36:08 x1daqdc0 daqd[5453]: first 5 dcuids seen
Nov 02 21:36:08 x1daqdc0 daqd[5453]: saw dcu 166 - gps: 1383021386 nano: 0 cycle: 0
Nov 02 21:36:08 x1daqdc0 daqd[5453]: saw dcu 161 - gps: 1383021386 nano: 0 cycle: 0
Nov 02 21:36:08 x1daqdc0 daqd[5453]: saw dcu 36 - gps: 1383021386 nano: 0 cycle: 0
Nov 02 21:36:08 x1daqdc0 daqd[5453]: saw dcu 34 - gps: 1383021386 nano: 0 cycle: 0
Nov 02 21:36:08 x1daqdc0 daqd[5453]: saw dcu 35 - gps: 1383021386 nano: 0 cycle: 0
Nov 02 21:36:09 x1daqdc0 kernel: dqprod[5475]: segfault at 7f04e0edb9d0 ip 00007f05640162e8 sp 00007f04a55b9c40 error 4 in libpthread-2.28.so[7f0564013000+f000]
Nov 02 21:36:09 x1daqdc0 kernel: Code: 00 00 41 56 41 55 41 54 55 53 48 83 ec 40 64 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 48 85 ff 0f 84 1b 01 00 00 48 89 fb <8b> bf d0 0
Nov 02 21:36:09 x1daqdc0 systemd[1]: rts-daqd.service: Main process exited, code=killed, status=11/SEGV
Nov 02 21:36:09 x1daqdc0 systemd[1]: rts-daqd.service: Failed with result 'signal'.
h1daqdc0:
Nov 02 21:00:15 h1daqdc0 daqd[315938]: Dropped data from shmem or received 0 dcus; gps now = 1383019233, 0; was = 1383019226, 13; dcu count = 19
Nov 02 21:00:15 h1daqdc0 daqd[315938]: expected gps = 1383019226
Nov 02 21:00:15 h1daqdc0 daqd[315938]: expected cycle = 14
Nov 02 21:00:15 h1daqdc0 daqd[315938]: expected nano = 62500013
Nov 02 21:36:04 h1daqdc0 daqd[327260]: Dropped data from shmem or received 0 dcus; gps now = 1383021382, 0; was = 1383021375, 13; dcu count = 19
Nov 02 21:36:04 h1daqdc0 daqd[327260]: expected gps = 1383021375
Nov 02 21:36:04 h1daqdc0 daqd[327260]: expected cycle = 14
Nov 02 21:36:04 h1daqdc0 daqd[327260]: expected nano = 62500013
Timing master experiences trouble with GPS lock. 512 in the plot below indicates that the GPS module isn't locked to GPS.
