Tony, Jim, Erik, Dave:
We had a timing error which caused DACKILLs on h1susb123, h1sush34, h1sush56 and DAC_FIFO errors on h1sush2a.
There was no obvious cause of the timing error which caused the Dolphin glitch, we noted that h1calcs was the only model with a DAQ CRC error (see attached).
After diag-reset and crc-reset only the SUS dackill and fifo error persisted.
We restarted all the models on h1susb123, h1sush2a, h1sush34, h1sush56 after bypassing the SWWD SEI systems for BSC1,2,3 and HAM2,3,4,5,6.
SUS models came back OK, we removed the SEI SWWD bypasses and handed the system over to Tony.
H1SUSH2A DACs went into error 300 ms before the CRC SUM increased on h1calcs.
The DAQ (I believe) reports CRC errors 120 ms after a dropped packet, leaving 180 ms. unaccounted for.
FRS31532 created for this issue. I has been closed as resolved-by-restarting.
Model restart logs from this morning:
Sat06Jul2024
LOC TIME HOSTNAME MODEL/REBOOT
01:12:22 h1susb123 h1iopsusb123
01:12:33 h1sush2a h1iopsush2a
01:12:39 h1sush34 h1iopsush34
01:12:43 h1susb123 h1susitmy
01:12:47 h1sush2a h1susmc1
01:12:56 h1sush56 h1iopsush56
01:12:57 h1susb123 h1susbs
01:12:59 h1sush34 h1susmc2
01:13:01 h1sush2a h1susmc3
01:13:10 h1sush56 h1sussrm
01:13:11 h1susb123 h1susitmx
01:13:13 h1sush34 h1suspr2
01:13:15 h1sush2a h1susprm
01:13:24 h1sush56 h1sussr3
01:13:25 h1susb123 h1susitmpi
01:13:27 h1sush34 h1sussr2
01:13:29 h1sush2a h1suspr3
01:13:38 h1sush56 h1susifoout
01:13:52 h1sush56 h1sussqzout