R. Short, J. Oberling, P. Thomas, J. Hanks

We restarted the PSL after yesterday's power outage.  Some notes:

• The LVEA Control Box is on a bench-top power supply that had to be turned on manually, the system would not turn on until this was done
• The PSL restarted mostly without issue after that
  • For some reason we lost our system calibrations file (power meters, PDs, etc.), it was using calibrations from November 2024, shortly after the final NPRO swap.  More below.
• Communication between PSL Beckhoff and EPICS is still broken, Patrick and Jonathan are currently investigating

Once the PSL Beckhoff was restarted we noticed that the output power seemed unusually low, we traced this to the calibration settings being out of date.  When running a newer version of the software we have to remember to grab the persistent settings file (port_851.bootdata from C:\TwinCAT\3.1\Boot\PLC), this holds all of the trip points, sensor calibrations, and a running tally of operating hours.  Not sure why the system lost this information now, I don't think we've ever seen this happen with a system restart and I currently have no explanation.  We were able to get the PD and LD monitor calibration settings from my alog from February when I changed the pump diode operating currents, and were able to grab the operating hours using ndscope (we looked at what they were reading when the power outage happened and set the operating hours back to that point + 1 hour (since the system had been running for ~1 hour at that point)).  One thing to note, however, the persistent operating hour data is now completely wrong.  The software tracks operating hours 2 ways: a user-updatable value and a locked value.  The former allows us to change operating hours when we install a new component (like swapping chillers or installing a new NPRO/Amplifier after a failure), while the latter tracks total uptime of said components (i.e. total number of hours an NPRO, any NPRO, has been running in the system).  It's these latter operating hours that are completely bogus, as we have no way to update these if the persisten settings file is lost (as I said, it's a locked value).

For future reference I've attached a screenshot of the current system settings table; the operating hours that are now wrong are in column labeled OPHRS A.

Mon Apr 07 10:07:39 2025 Fill completed in 7min 36secs

BSC8 annulus volume has been vented with nitrogen.  Randy has been notified that door bolt removal can proceed.

Came in to find all IFO systems down. Working through recovery now.

SUSB123 power supply seemed to have tripped off. ITMs and BS OSEM counts were all sitting very low around 3000. Once power supply was flipped back on, OSEM counts returned to normal for ITMX and BS, but now ITMY coil drivers filter banks are fflashing ROCKER SWITCH DEATH. Jeff and Richard are back in the CER cycling the coil drivers to hopefully fix that.

Also power for ISIITMY is down and being worked on to bring back.

Most of vent work is currently on hold and focused on getting systems back online

The counts were pretty low over the weekend, peaking at ~ 30 counts of 0.3s and 10 counts for 0.5s.

By no means a complete list and in no particular order:

FMCS EPICS

CER Timing Fanout 14th port

Diode room PSL Beckhoff

HEPI pump controller, corner station

Slow controls DEV4 terminal errror

h1cdsrfm long range dolphin

DTS

ncalex

BRS EX, EY

HWS

EX mains mon

TITLE: 04/07 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 18mph Gusts, 10mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.29 μm/s
QUICK SUMMARY:

The CDS team is currently working on getting the control room screens and other systems back up from the power outage last night.

Today we have a vent coordination meeting at 15:30UTC.

Plans for today:

• FAC Clean BSC8 door and area
• IAS HAM1 stack table survey
• IAS HAM1 support tube ends?
• HAM1 contamination control checks
• Corey HAM1 pics
• ISI HAM1 offload HEPI setup, disconnect actuators
• HAM1 - COAX Cable length measurement (disconnect at component)
• VAC Start Feedthru Swap
• HAM4-5 Cable tray work - Ken, some Crane usage
• OPS - trend ITM OpLevs and PIT/YAW WIT channels to look for vent shift

Richard powered up h1seih23 from the front panel, it is now running.

We power cycled h1psl0, it is back.

Erik reconfigured EX Dolphin front ends to communicate with port 8 of the EY Dolphin switch to get them to get the models going. CDSRFM at EX points to port 7 of EY's switch.

All models are running, only IPC errors are long-range-dolphin.

Should someone from operations start activating the SEI and SUS systems?

From UPS emails it appears the site has had mutiple power issues starting 18:06 PDT extending to 18:20

Today's activities:

- HAM1's Y+ and Y- doors have been taken off (thanks to Tyler, Randy for the help in the door removal, and to Tony and Corey for breaking the bolts earlier)
- All 5 HAM1/HAM2 viewport covers have been installed
- The 16.5" flange was taken off from the top of HAM1 - this will be the basis of the new vacuum interlock gauge
- A high number of leak checking was done, in detail:
 - Y-manifold: Y+ X+ side flanges (3 pcs.); Turbo flange (1 pc.)
 - BSC8: Y+ X+ corner (3 pcs.); Y- X+ corner (3 pcs.); X+ HEPI bellows (2 pcs.); X+ collar 16.5" flanges (2 pcs.)
 - BSC8-BSC1 spool: Y+ X+ flanges (3 pcs.)
 - BSC1: Y+ X+ side flanges (3 pcs.); X+ collar flanges (2 pcs.); X+ HEPI bellows (2 pcs.); X+ door flanges (5 pcs.); Y- X+ side flanges (3 pcs.)
 - BSC2: Y+ X+ side flanges (3 pcs.)
 - BSC3: Y+ HEPI bellows (2 pcs.); Y+ X- side flanges (3 pcs.); Y+ collar flanges (2 pcs.); Y+ door flanges (5 pcs.); Y+ X+ side flanges (3 pcs.)
 - HAM2-HAM3 spool: vent port (1 pc.); X- flanges (11 pcs.); Y- side flanges (2 pcs.); X+ Y- flanges (5 pcs.)
 - HAM3: Y- door flanges (5 pcs.)