Dan, Sheila
The attached screenshot shows a one second trend of the beckhoff channel that is the fast shutter trigger readback, along with two RCG channels, the first is AS_C sum, this PD should be the source for the fast shutter trigger, and ASAIR_LF, which should see similar power fluctuations.
The beckhoff channel seems to be 0.2 seconds ahead of the RCG channels. This timing difference sometimes makes it appear as though the fastshutter shut first, causing the lockloss, as in the second attached screenshot.
We would like to prove definteively that the shutter is only shutting when it should, but we don't have a readback that we trust.
Dave and I looked into this somewhat and we don't think that the timestamps of the EtherCAT systems are used at all by the frame builder. As a matter of fact the internal clock of these machines was about 3 seconds off. Since it is hard to believe that the EtherCAT systems violate causality, the most likely candidates are frame builder, data concentrator or maybe the EPICS gateway. This would also mean that all data transported through EPICS to the frame builder could be affected. More investigations are pending.