Reports until 09:37, Saturday 22 April 2023
H1 CDS
david.barker@LIGO.ORG - posted 09:37, Saturday 22 April 2023 - last comment - 11:19, Monday 24 April 2023(68915)
Beckhoff communication error to CER

At 04:56 it appears we lost communication to the CER Beckhoff system. Timing MEDMs are attached.

Images attached to this report
Comments related to this report
jenne.driggers@LIGO.ORG - 09:46, Saturday 22 April 2023 (68916)
Link

Gabriele, DanielS, Jenne

Indeed it looks like the power supply has failed.  Alog 68911 indicated it was very much not healthy, and Richard pointed out in a comment that plans were already in place to replace it early next week, but it seems that it didn't last that long. 

Fil and Fernando are discussing whether it'll be possible to replace it this weekend.

Images attached to this comment
david.barker@LIGO.ORG - 09:44, Saturday 22 April 2023 (68917)
Link

Corner station ethercat device3 Slavecountactual went from 310 to 1 at 04:55:54 PDT

Images attached to this comment
marc.pirello@LIGO.ORG - 11:15, Saturday 22 April 2023 (68918)
Link

Supply replaced per WP11147

Power looks good, old supply fan bushing was completely shot.  Turning off the lights and heading home.

david.barker@LIGO.ORG - 11:27, Saturday 22 April 2023 (68919)
Link

Device3 Slavecountactual increased from 1 to 301, but still 9 shy of 310 needed. We are contacting Daniel and Patrick to see what is needed to complete the recovery.

patrick.thomas@LIGO.ORG - 12:00, Saturday 22 April 2023 (68920)
Link 

Logging into 10.105.0.26 I first saw that the Anybus X-gateway terminals were not in OP, so I toggled the requested state of the EtherCAT device. This brought these back but then I noticed that the Corner Chassis 4 L10 EL9410 was in error (see first two screenshots), so I tried toggling some more. Now that terminal is back, but a few of the terminals after it are still in error (see last screenshot, yellow above terminals instead of green).
Images attached to this comment
patrick.thomas@LIGO.ORG - 12:04, Saturday 22 April 2023 (68921)
Link 

Toggling isn't bringing these back. Next step would be to power cycle corner chassis 4. If that fails, pull the chassis.
gabriele.vajente@LIGO.ORG - 13:27, Saturday 22 April 2023 (68922)
Link

Power cycled Corner 4 at 1:25 pm

patrick.thomas@LIGO.ORG - 13:44, Saturday 22 April 2023 (68923)
Link 

Power cycling appears to have brought things back online.
marc.pirello@LIGO.ORG - 11:19, Monday 24 April 2023 (68949)
Link

Supply replaced was S1201930, this supply had a failed fan.  We will replace the fan and put this supply into the spares inventory.  I have it recorded that thie old supply was sourcing 4A at 24V.

Supply installed was S1201903.  This supply is on mezanine rack C2, U25-U27 LHS powering ISC C2 Slow Controls with 24V.  This supply seems to be delivering 8A, unclear why this is.