Reports until 08:19, Wednesday 12 June 2019
H1 CDS
david.barker@LIGO.ORG - posted 08:19, Wednesday 12 June 2019 - last comment - 15:12, Monday 17 June 2019(49837)
Corner station Beckhoff hardware error

h1ecatc1 C1_PLC1_Device3 is under reporting its slave count (actual = 156, configured = 291). 

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 08:28, Wednesday 12 June 2019 (49838)

We lost the units around 13:20 UTC (06:20 PDT)

 

Images attached to this comment
patrick.thomas@LIGO.ORG - 08:34, Wednesday 12 June 2019 (49839)
Connection between corner chassis 4 L0 and M0 has failed. The first red block in the topology screenshot is corner chassis 4 M0 (EK1100).
Images attached to this comment
david.barker@LIGO.ORG - 08:30, Wednesday 12 June 2019 (49840)

Patrick and Corey are working the problem.

corey.gray@LIGO.ORG - 08:33, Wednesday 12 June 2019 (49841)

The time for the error Dave mentions above coincides  with the H1 Lockloss (and issues with IMC) at 13:21utc (6:21amPDT).

Patrick has tracked this down to the Corner Chasis 4 in the CER and says this may need to get pulled.  Fil has been notified and will work with Patrick.

corey.gray@LIGO.ORG - 08:49, Wednesday 12 June 2019 (49843)

For something like this which might have taken H1 down, should this be a Verbal Alarm?  (only way an operator would know about this issue is to happen to catch the red box on the CDS Overview (which we both didn't notice).

[Attached is the RED box on the CDS Overview.]

Images attached to this comment
corey.gray@LIGO.ORG - 09:49, Wednesday 12 June 2019 (49845)

FRS Ticket submitted:  #13049.  

https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13049

So far down time is 3.5hrs, but now I need to try locking.

Side Note:  Alog would not let me link the FRS ticket to the text of "13049" above.  indecision

filiberto.clara@LIGO.ORG - 10:10, Wednesday 12 June 2019 (49847)
EK1100 EtherCAT Coupler failed and replaced. Unit was scanned and communication errors have cleared.

We did noticed a buzzing sound coming from one of the terminals. Tried various combinations of powering off different components to isolate noisy terminal(s), but decided to reinstall unit to get IFO back up.

Corner 4 Slow Controls Chassis S1107450

F. Clara, P. Thomas
david.barker@LIGO.ORG - 10:15, Wednesday 12 June 2019 (49848)
david.barker@LIGO.ORG - 13:34, Wednesday 12 June 2019 (49854)

On the CDS Overview MEDM, if there is a Beckhoff hardware error the slow controls banner will turn RED for better visibility. I simulated a C1PLC1 error to demonstrate (see attachment).

Images attached to this comment
keita.kawabe@LIGO.ORG - 01:09, Saturday 15 June 2019 (49935)

All in all this cost us about 5 hours. See e.g. https://ldas-jobs.ligo.caltech.edu/~detchar/summary/day/20190612/ .

corey.gray@LIGO.ORG - 15:12, Monday 17 June 2019 (50005)

This FRS ticket (13049) is now CLOSED/RESOLVED.