Currently, iscex reports maximum CPU times of 29µs in an average 1 second period with an absolute maximum of 36µs. iscey reports average maxima of 28µs and an absolute maximum of 33µs. The fiber delay is 21.5µs for the x-arm and 21.7µs for the y-arm. The absolute worst case delay to the corner is then 58µs and 55µs for iscex and iscey, respectively. This is still well below the cycle time of 61µs. However, the corner lsc model still reports IPC errors. Something doesn't add up. The attached plot shows the IPC error counts in the corner lsc, their timestamps and the iscex/y CPU times.The IPC errors are latched when when they occur. This has the unfortunate side effect of making the first twelve plots in the attachment essentially useless, except indicating that "errors are happening".
In contrast, the lsc uses an average maxima of 35µs with an absolute maximum of 40µs. Both susetmx and susetmy report around 10 IPC errors per second in each and every second.
This shows the error rate in the end station sus together with lsc CPU maximum times.
There are some additional delays 1) Propagation delay for every RFM card the message transits. Vendor reports 0.5 mu-sec per adapter. LHO has 6 cards on each arm (so up to 3 mu-sec). In one direction, it may traverse very few cards, while the other direction can have the maximal number. Removing the unneeded card on the end-station SEI machine may help at LHO. 2) PCIe bus contention. On the current front-end computers, we already see bus contention issues with receiving all the ADC interrupt alerts in a timely manner. Delays of a few mu-sec can happen. This may affect traffic to/from the RFM cards as well. - Testing on newer front-end computers shows both reduced real-time loop time (faster processor) and the variance (better/faster bus controller). This is also a likely path forward.