Jonathan, Dave:
In the past week we have had several occasions of DAQ-CRC error counts, primarily on h1seiex and also on h1susex. Here is what we have discovered so far:
All CRC errors are accurately reported in the DAQ data concentrator's log file
Errors appear at random times, and at random times within the second
When a front end has errors, for each model, its CRC counter increments by the same number every time
The number the CRC increments by is not the same across models. If it differs, it increments over the model position in the rtsystab file (e.g. IOP < USER1 < USER2)
The error, as seen by the data concentrator, is that the model's gps time is one second in the past
If, during the period of gps mismatch, a second boundary is crossed, both dcu_gps and gps increment within the same 16Hz cycle
The data concentrator marks the DCU DAQ-STATUS internally as 0x4000 (timing error).
This error propagates to the frame writer, which marks all channels from the model with this status
The error does not always show up on the EPICS STATUS PV (perhaps gets reset to zero before being reported?)
h1seiex:
number of CRC events since 11jun2019: 10
model order in mx_stream (num crc_errors each time): h1iopseiex(7), h1hpietmx(8), h1isietmx(8)
Error sequence: (hpi, isi) + 7*(iop, hpi, isi)
h1susex:
number of CRC events since 11jun2019: 2
model order in mx_stream (num crc_errors each time): h1iopsusex(8), h1susetmx(8), h1sustmsx(8), h1susetmxpi(9)
Error sequence: etmxpi + 8(iop, etmx, tmsx, etmxpi)
Attached plot of full data at the time of this morning's h1susex CRC glitch shows that for 8 cycles (0.5 seconds) the DAQ data is repeated. It is easy to see in a slowly varying channel (lower plot) and more hidden in the upper plot.
We reviewed the code and understand the internal mechanism of the error. There are several possible causes:
the front end models are slow in getting data into shared memory*
mx_stream is slow in reading the DAQ data from shared_memory
mx_stream is slow in processing the data (it calculates the data crc)
mx_stream is slow in sending the data
the network is slow in getting the data to the data_concentrator
* this is highly unlikely, however the mx_stream is triggered by the IOP being ready, and of the 4 computers at EX the two with problems have iop watchdogs.
Outstanding questions are;
Why is this happening only at EX?
Why did it only start last week?
Why only in h1seiex (10) and h1susex (2)?
We are reviewing our options for mitigation and/or further diagnostics.
Jonathan, Greg, Dave:
We have verified that both the full frames and the HofT frames are correctly marking the DAQ status of all impacted channels as bad (0x4000 = timing error).
Opened FRS13093 to cover this