h1ecatc1 C1_PLC1_Device3 is under reporting its slave count (actual = 156, configured = 291).
We lost the units around 13:20 UTC (06:20 PDT)
Connection between corner chassis 4 L0 and M0 has failed. The first red block in the topology screenshot is corner chassis 4 M0 (EK1100).
Patrick and Corey are working the problem.
The time for the error Dave mentions above coincides with the H1 Lockloss (and issues with IMC) at 13:21utc (6:21amPDT).
Patrick has tracked this down to the Corner Chasis 4 in the CER and says this may need to get pulled. Fil has been notified and will work with Patrick.
For something like this which might have taken H1 down, should this be a Verbal Alarm? (only way an operator would know about this issue is to happen to catch the red box on the CDS Overview (which we both didn't notice).
[Attached is the RED box on the CDS Overview.]
FRS Ticket submitted: #13049.
https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13049
So far down time is 3.5hrs, but now I need to try locking.
Side Note: Alog would not let me link the FRS ticket to the text of "13049" above. ![]()
EK1100 EtherCAT Coupler failed and replaced. Unit was scanned and communication errors have cleared. We did noticed a buzzing sound coming from one of the terminals. Tried various combinations of powering off different components to isolate noisy terminal(s), but decided to reinstall unit to get IFO back up. Corner 4 Slow Controls Chassis S1107450 F. Clara, P. Thomas
On the CDS Overview MEDM, if there is a Beckhoff hardware error the slow controls banner will turn RED for better visibility. I simulated a C1PLC1 error to demonstrate (see attachment).
All in all this cost us about 5 hours. See e.g. https://ldas-jobs.ligo.caltech.edu/~detchar/summary/day/20190612/ .
This FRS ticket (13049) is now CLOSED/RESOLVED.