J. Warner, J. Kissel, M. Pirello
2025-04-06 Power outage: LHO:83753

Among the things that did not recover nicely from the 2025-04-06 power outage was the aLIGO BSC ISI Interface Chassis (D1002432) for Corner 1 of the ISI ITMY. Page 1 of the wiring diagram for a generic BSC SEI (HPI and ISI) system D0901301 shows that ST1 and ST2 CPS, ST1 L4Cs, and ST2 GS13s of a given corner (Corner 1, Corner 2, and Corner 3) are all read out by the same chassis. In the instance of this drawing / rack for ISI ITMY is SEI-C4, and this corner 1 chassis lives in SEI-C4 U40.

This morning, Jim identified that this particular interface chassis failed -- after hearing that Tony tried to re-isolate the platform repeated last night and couldn't -- by opening the ISI ITMY overview screen and found that all of these corner one sensors' signals were reading out a "noisy zero" i.e. 0 +/- a few counts. See attached time-machine screenshot, where the corner 1 sensors are higlighted with yellow arrows showing less than 2 counts.

Jim, Marc, and I went out to the CER SEI-C4 and found that power cycling the chassis would not help.
There happened to be a spare chassis resting on top of the existing corner 1 chassis, so we just made the swap.
We suspect that chassis power regulator didn't survive the current surge of the rack's DC power supplies losing power and coming back (perhaps unequally), but we'll do a post-mortem later.

Taken out Interface chassis S1201320.
Replaced with Interface chassis S1203892.

The attached images show the top of the SEI-C4 rack BEFORE and AFTER.

Slow start on vent tasks today - besides the lack of water onsite this morning, and the particularly nasty power outage which are making things not come back up very well, we popped into HAM2 to keep moving on a few of the next steps.  Corey has captured detailed pictures of components and layouts, and Camilla and I have logged all of the LSC and ASC PD serial numbers and cables numbers.  We removed all connections at these PD boxes and Daniel is out making the RF cable meter length measurements.  OPS were realigning all suspensions to a golden DRMI time they chose as a reference for any fall back times.  Jason and Ryan are troubleshooting other PSL items that are misbehaving.

We are gearing up to take a side road on today's plan to look for flashes out of HAM2 to convince ourselves that the PSL PZT and alignment restoration of some suspensions are somewhat correct.

In preparation for disconnecting cables in HAM1, I turned off the following DC interface chassis:

LSC RF PD DC interface chassis in ISC R4 (REFL-A, POP-A among others), LSC RF PD DC interface chassis in ISC R1 (REFL-B among others),  and ASC WFS DC interface chassis in ISC R4 (REFL_A, REFL_B among others).

Daniel will perform TDR to measure RF in-vac cable length from outside.

J. Kissel, M. Pirello
2025-04-06 Power outage: LHO:83753

Among the things that did not recover nicely from the 2025-04-06 power outage was the Timing Comparator D1001370 that lives in ISC-C2 U40 (see component C261 on pg 3 of D1900511-v9). The symptom was that its time-synchronizing FPGA was caught in a bad state, and the timing fanout in the CER Beckhoff status for the comparator was reporting that H1:SYS-TIMING_C_FO_A_PORT_13_NODE_UPLINKUP was in error (a channel value of zero instead of one). 

We didn't know any of this at the start of the investigation.
At the time of investigation start, we only new of an error by following through the automatically generated "SYS" screens (see attached guide),
   SITEMAP > SYS > Timing > Corner A Button [which had a red status light] 
   > TIMING C_FO_A screen Port 13, dynamically marked as a "C" for comparator [whose number was red, and the status light was red] 
   > Hitting the "C" opens the subscreen for TIMING C_FO_A NODE 13, which shows that red "Uplink Down" message in the middle right
The screenshot shows the NODE 13 screen both in the "now" fixed green version state, and a time-machined "broken" version.

Going out to the CER, we found that status light for Digital Port 13 == Analog Port 14 on the timing fanout (D080534; ISC-C3 U11) was blinking. 
Marc tried moving the comms cable to analog port 16, because "sometimes these things have bad ports." That didn't work, so we moved it back to analog port 14.

That port's comms fiber cable was not labeled, so we followed it physical to find its connection to the SQZ timing comparator (again in ISC-C2 U40, thankfully "right next door"), to find it's "up" status light also blinking.
Marc suggested that the comparators may lose sync, so we power cycled it. This chassis doesn't have a power switch, so we simply disconnected and reconnected its +/-18 V power cable.
After waiting ~2 minutes, all status lights turned green.

#FIXEDIT

I've been trying to recover seismic systems this morning after the power outage. So far, I've gotten HEPI pump stations back up, all the ISI are damping. Worked with Patrick to get both BRS are back up and running, but their heating was off for a while and it will take some time to get back to a good state.

ETMY and HAM8 however have some issue with the binary read backs for their coil drivers. I've check the racks and the overtemp relay lights are all good, but the binary read backs indicate a number of the overtemp relays are tripped. Not sure what the cause is yet, but fixing these isn't a priority today. Working with Dave, I power cycled both the binary in and coil drivers for HAM8, but that didn't fix the readbacks. Similar to this alog, we just put in test offsets to get the bit word clean and will wait for Fil to be available to try opening the expansion chassis. These tables should be able to isolate now, not that they are needed. Dave has filed and FRS and the test offset shows up in SDF. 

As of 12:16 here is the latest CDS not-working list (some cleared from last list, some added).

Slow controls DEV4 EX chassis terminal missing

NCALEX

Diode Room Beckhoff (h1pslctrl0)

BRS EX and EY

HWS

EX Mains Mon

Weather stations (EX and EY)

PWRCS and PWREX

R. Short, J. Oberling, P. Thomas, J. Hanks

We restarted the PSL after yesterday's power outage.  Some notes:

• The LVEA Control Box is on a bench-top power supply that had to be turned on manually, the system would not turn on until this was done
• The PSL restarted mostly without issue after that
  • For some reason we lost our system calibrations file (power meters, PDs, etc.), it was using calibrations from November 2024, shortly after the final NPRO swap.  More below.
• Communication between PSL Beckhoff and EPICS is still broken, Patrick and Jonathan are currently investigating

Once the PSL Beckhoff was restarted we noticed that the output power seemed unusually low, we traced this to the calibration settings being out of date.  When running a newer version of the software we have to remember to grab the persistent settings file (port_851.bootdata from C:\TwinCAT\3.1\Boot\PLC), this holds all of the trip points, sensor calibrations, and a running tally of operating hours.  Not sure why the system lost this information now, I don't think we've ever seen this happen with a system restart and I currently have no explanation.  We were able to get the PD and LD monitor calibration settings from my alog from February when I changed the pump diode operating currents, and were able to grab the operating hours using ndscope (we looked at what they were reading when the power outage happened and set the operating hours back to that point + 1 hour (since the system had been running for ~1 hour at that point)).  One thing to note, however, the persistent operating hour data is now completely wrong.  The software tracks operating hours 2 ways: a user-updatable value and a locked value.  The former allows us to change operating hours when we install a new component (like swapping chillers or installing a new NPRO/Amplifier after a failure), while the latter tracks total uptime of said components (i.e. total number of hours an NPRO, any NPRO, has been running in the system).  It's these latter operating hours that are completely bogus, as we have no way to update these if the persisten settings file is lost (as I said, it's a locked value).

For future reference I've attached a screenshot of the current system settings table; the operating hours that are now wrong are in column labeled OPHRS A.

Mon Apr 07 10:07:39 2025 Fill completed in 7min 36secs

BSC8 annulus volume has been vented with nitrogen.  Randy has been notified that door bolt removal can proceed.

Came in to find all IFO systems down. Working through recovery now.

SUSB123 power supply seemed to have tripped off. ITMs and BS OSEM counts were all sitting very low around 3000. Once power supply was flipped back on, OSEM counts returned to normal for ITMX and BS, but now ITMY coil drivers filter banks are fflashing ROCKER SWITCH DEATH. Jeff and Richard are back in the CER cycling the coil drivers to hopefully fix that.

Also power for ISIITMY is down and being worked on to bring back.

Most of vent work is currently on hold and focused on getting systems back online

The counts were pretty low over the weekend, peaking at ~ 30 counts of 0.3s and 10 counts for 0.5s.

By no means a complete list and in no particular order:

FMCS EPICS

CER Timing Fanout 14th port

Diode room PSL Beckhoff

HEPI pump controller, corner station

Slow controls DEV4 terminal errror

h1cdsrfm long range dolphin

DTS

ncalex

BRS EX, EY

HWS

EX mains mon

TITLE: 04/07 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 18mph Gusts, 10mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.29 μm/s
QUICK SUMMARY:

The CDS team is currently working on getting the control room screens and other systems back up from the power outage last night.

Today we have a vent coordination meeting at 15:30UTC.

Plans for today:

• FAC Clean BSC8 door and area
• IAS HAM1 stack table survey
• IAS HAM1 support tube ends?
• HAM1 contamination control checks
• Corey HAM1 pics
• ISI HAM1 offload HEPI setup, disconnect actuators
• HAM1 - COAX Cable length measurement (disconnect at component)
• VAC Start Feedthru Swap
• HAM4-5 Cable tray work - Ken, some Crane usage
• OPS - trend ITM OpLevs and PIT/YAW WIT channels to look for vent shift

Richard powered up h1seih23 from the front panel, it is now running.

We power cycled h1psl0, it is back.

Erik reconfigured EX Dolphin front ends to communicate with port 8 of the EY Dolphin switch to get them to get the models going. CDSRFM at EX points to port 7 of EY's switch.

All models are running, only IPC errors are long-range-dolphin.

Should someone from operations start activating the SEI and SUS systems?

Today's activities:

- HAM1's Y+ and Y- doors have been taken off (thanks to Tyler, Randy for the help in the door removal, and to Tony and Corey for breaking the bolts earlier)
- All 5 HAM1/HAM2 viewport covers have been installed
- The 16.5" flange was taken off from the top of HAM1 - this will be the basis of the new vacuum interlock gauge
- A high number of leak checking was done, in detail:
 - Y-manifold: Y+ X+ side flanges (3 pcs.); Turbo flange (1 pc.)
 - BSC8: Y+ X+ corner (3 pcs.); Y- X+ corner (3 pcs.); X+ HEPI bellows (2 pcs.); X+ collar 16.5" flanges (2 pcs.)
 - BSC8-BSC1 spool: Y+ X+ flanges (3 pcs.)
 - BSC1: Y+ X+ side flanges (3 pcs.); X+ collar flanges (2 pcs.); X+ HEPI bellows (2 pcs.); X+ door flanges (5 pcs.); Y- X+ side flanges (3 pcs.)
 - BSC2: Y+ X+ side flanges (3 pcs.)
 - BSC3: Y+ HEPI bellows (2 pcs.); Y+ X- side flanges (3 pcs.); Y+ collar flanges (2 pcs.); Y+ door flanges (5 pcs.); Y+ X+ side flanges (3 pcs.)
 - HAM2-HAM3 spool: vent port (1 pc.); X- flanges (11 pcs.); Y- side flanges (2 pcs.); X+ Y- flanges (5 pcs.)
 - HAM3: Y- door flanges (5 pcs.)