aLIGO LHO Logbook

H1 CDS

david.barker@LIGO.ORG - posted 20:14, Sunday 06 April 2025 - last comment - 14:58, Friday 25 April 2025(83758)

Recovery from power outage

Richard powered up h1seih23 from the front panel, it is now running.

We power cycled h1psl0, it is back.

Erik reconfigured EX Dolphin front ends to communicate with port 8 of the EY Dolphin switch to get them to get the models going. CDSRFM at EX points to port 7 of EY's switch.

All models are running, only IPC errors are long-range-dolphin.

Should someone from operations start activating the SEI and SUS systems?

Images attached to this report

Comments related to this report

jordan.vanosky@LIGO.ORG - 23:09, Sunday 06 April 2025 (83762)

Link

After the power outage, I went out to the site ~10pm Pacific to check status of purge air skid. Kobelco, water pump and drying towers were all running upon arrival. I verified the dew point was still ok (-41.8C measured at HAM1) and verified the drying towers switched normally. Note: Purge air was only connected to HAM1 since doors are off, corner volume is vented but not actively purging with no doors off.

While there I also restarted the controller for 3 of the FCT ion pumps and reset solenoid valve on aux cart connected to GV5 annulus. All looks ok. Left site ~10:40pm.

david.barker@LIGO.ORG - 20:15, Sunday 06 April 2025 (83759)

Link

h1guardian1 did not reboot, here is the guardian MEDM

Images attached to this comment

david.barker@LIGO.ORG - 20:49, Sunday 06 April 2025 (83760)

Link

Systems currently down as of 20:49:

FMCS/BACNET

Corner Station HEPI Pump Controller

PSL Diode Room slow controls

CP1 overfill

NCAL

BRS

HWS

DTS test stand

david.barker@LIGO.ORG - 21:30, Sunday 06 April 2025 (83761)

Link

To continue running vac alarms overnight I have bypassed the currently active alarms for the next 24 hours.

Bypass will expire:
Mon Apr 7 09:28:54 PM PDT 2025
For channel(s):
H0:FMC-CS_CY_H2O_PUMPSTAT
H0:FMC-CS_CY_H2O_SUP_DEGF
H0:FMC-CS_FIRE_PUMP_1
H0:FMC-CS_FIRE_PUMP_2
H0:FMC-EX_CY_H2O_PUMPSTAT
H0:FMC-EX_CY_H2O_SUP_DEGF
H0:FMC-EY_CY_H2O_PUMPSTAT
H0:FMC-EY_CY_H2O_SUP_DEGF
H1:CDS-DTS_ENV_TEMPERATURE_DEGF
H1:CDS-STAT_NUMBER_OF_ERRORS

gerardo.moreno@LIGO.ORG - 14:58, Friday 25 April 2025 (84134)VE

Link

Regarding the 4UHV ion pump controller, the autostart setting is enabled now, thus the unit should power ON after a power glitch. See some of the instructions to access the general menu.

Images attached to this comment

H1 CDS

Link

david.barker@LIGO.ORG - posted 18:25, Sunday 06 April 2025 - last comment - 19:54, Sunday 06 April 2025(83753)

Power event, most front ends down, Beckhoff slow controls down, DAQ down.

From UPS emails it appears the site has had mutiple power issues starting 18:06 PDT extending to 18:20

Images attached to this report

Comments related to this report

david.barker@LIGO.ORG - 19:36, Sunday 06 April 2025 (83754)

Link

Richard is on his way to the site. Erik and I remotely brought h1susauxb123 (edc) back online, which got the DAQ to restart itself so we are trending vacuum channels again. Data gap for VE data is 18:05 - 18:40.

Current CDS issues:

h1seih23 ipmi not working, computer is still down.

IX Dolphin switch at EX is not working correctly. h1susex, h1seiex, h1iscex are down. Erik is trying a workaround.

h1psl0 is down, it didn't actually power off, we are not sure we should get that going again.

Images attached to this comment

david.barker@LIGO.ORG - 19:41, Sunday 06 April 2025 (83755)

Link

MSR UPS messages:

Name : ups-msr-0
Location : LHO MSR
Contact : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 04/06/2025
Time: 18:20:42
Code: 0x0002
Critical - System: Warmstart.

Name : ups-msr-0
Location : LHO MSR
Contact : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 04/06/2025
Time: 18:20:50
Code: 0x0101
Informational - UPS: Restored the local network management interface-to-UPS communication.

david.barker@LIGO.ORG - 19:51, Sunday 06 April 2025 (83756)

Link

GC OSB UPS messages (times out by one hour)

Name : lho-gc-osb-ups
Location : LHO OSB Mechanical Room (rm 160)
Contact : lho-gc-root@ligo-wa....
Serial Number : 5A1315T04169
Device Serial Number : PS1316131451
Date : 04/06/2025
Time : 17:06:10
Code : 0x0235
Critical - UPS: A site wiring fault exists.

Date : 04/06/2025
Time : 17:06:10
Code : 0x020F
Critical - UPS: An input voltage or frequency problem prevents switching to bypass mode.

Date : 04/06/2025
Time : 17:06:26
Code : 0x010A
Informational - UPS: No longer on battery power.

Date : 04/06/2025
Time : 17:06:44
Code : 0x0236
Informational - UPS: A site wiring fault no longer exists.

Date : 04/06/2025
Time : 17:06:44
Code : 0x0210
Informational - UPS: An input voltage or frequency problem no longer prevents switching to bypass mode.

david.barker@LIGO.ORG - 19:54, Sunday 06 April 2025 (83757)

Link

GC LSB (one hour out)

Time : 17:18:14
Code : 0x0109
Warning - UPS: On battery power in response to an input power outage.

Critical - UPS: An out of tolerance input voltage or frequency , prevents switching to bypass mode.

Time : 17:18:33
Code : 0x0210
Informational - UPS: An out of tolerance input voltage or frequency no longer prevents switching to bypass mode

Date : 04/06/2025
Time : 17:18:36
Code : 0x010A
Informational - UPS: No longer on battery power.

LHO VE

Link

david.barker@LIGO.ORG - posted 10:37, Sunday 06 April 2025 (83752)

Sun CP1 Fill

Sun Apr 06 10:06:44 2025 INFO: Fill completed in 6min 40secs

Images attached to this report

LHO VE

Link

david.barker@LIGO.ORG - posted 10:25, Saturday 05 April 2025 (83750)

Sat CP1 Fill

Sat Apr 05 10:05:42 2025 INFO: Fill completed in 5min 38secs

Images attached to this report

H1 CDS

Link

david.barker@LIGO.ORG - posted 09:17, Saturday 05 April 2025 - last comment - 10:36, Saturday 05 April 2025(83747)

h1daqnds1 having issues serving data, please use h1daqnds0 for now while we work on this

Note that the web trends use nds0 and are updating.

Comments related to this report

david.barker@LIGO.ORG - 10:20, Saturday 05 April 2025 (83748)

Link

FRS33784 ticket created for this issue.

david.barker@LIGO.ORG - 10:23, Saturday 05 April 2025 (83749)

Link

Jonathan restarted rts-daqd on h1daqnds1 at 09:37 Sat 05apr2025PDT which cleared the immediate issue.

It appears that nds1 had a connection burst of over 300 connections requests at 23:30 Fri 04apr2025PDT which caused the issue. The daqd process continued to run after that but was no longer accepting any connection requests until the restart this morning.

david.barker@LIGO.ORG - 10:36, Saturday 05 April 2025 (83751)

Link

I rebooted nuc5 at 10:28 to get the Seismic BLRMS FOM running again. I think the other FOMs can wait till Monday.

LHO General

Link

tyler.guidry@LIGO.ORG - posted 17:18, Friday 04 April 2025 - last comment - 09:06, Saturday 05 April 2025(83745)

Potable Supply Valved Out to All Buildings

I received a call from Oli at 11:46 this morning informing me that water pressure at the building had fallen. I logged into the FMCS remotely and found that the potable holding tank had been completely drained and only one RO was making water. Before leaving home to come out to site, I contacted Randy and had him manually disable the domestic pressure skid and force the second RO skid on. At this point, because of the known leak, Richard and I have decided that the RO system will replenish the potable holding tank but that the water supply will remain valved out so as not to continue to tax the entire water treatment plant. 

C&E Trenching who had already been slated to come make the repairs next week has been contacted. After explaining the current situation, they have graciously agreed to move us up in their schedule and will be on site Monday Morning to being the work. Once the repair is made, water should be restored.

T. Guidry

Comments related to this report

david.barker@LIGO.ORG - 09:06, Saturday 05 April 2025 (83746)

Link

RO went into alarm Friday 23:53 04apr2025 PDT. I have temporarily removed it from the alarms system and restarted the service 09:04 sat 05apr2025.

LHO VE

Link

janos.csizmazia@LIGO.ORG - posted 15:38, Friday 04 April 2025 (83744)

2025 April vent - VAC diary

Today's activities:

- All the turbos have been valved out and switched off
- The corner was vented through the Y-manifold vent valve, with 3-4 Torr/min speed - it took 3.5 hrs
- As a side effect, the current of GV5's AIP went up, so now this Annulus volume is being pumped with an aux cart
- Another aux cart is pumping the RGA in the corner (between HAM5 and HAM4)

LHO VE

Link

janos.csizmazia@LIGO.ORG - posted 18:30, Thursday 03 April 2025 - last comment - 01:31, Monday 07 April 2025(83738)

2025 April vent - VAC diary

Today's activities:

- HAM1's Y+ and Y- doors have been taken off (thanks to Tyler, Randy for the help in the door removal, and to Tony and Corey for breaking the bolts earlier)
- All 5 HAM1/HAM2 viewport covers have been installed
- The 16.5" flange was taken off from the top of HAM1 - this will be the basis of the new vacuum interlock gauge
- A high number of leak checking was done, in detail:
 - Y-manifold: Y+ X+ side flanges (3 pcs.); Turbo flange (1 pc.)
 - BSC8: Y+ X+ corner (3 pcs.); Y- X+ corner (3 pcs.); X+ HEPI bellows (2 pcs.); X+ collar 16.5" flanges (2 pcs.)
 - BSC8-BSC1 spool: Y+ X+ flanges (3 pcs.)
 - BSC1: Y+ X+ side flanges (3 pcs.); X+ collar flanges (2 pcs.); X+ HEPI bellows (2 pcs.); X+ door flanges (5 pcs.); Y- X+ side flanges (3 pcs.)
 - BSC2: Y+ X+ side flanges (3 pcs.)
 - BSC3: Y+ HEPI bellows (2 pcs.); Y+ X- side flanges (3 pcs.); Y+ collar flanges (2 pcs.); Y+ door flanges (5 pcs.); Y+ X+ side flanges (3 pcs.)
 - HAM2-HAM3 spool: vent port (1 pc.); X- flanges (11 pcs.); Y- side flanges (2 pcs.); X+ Y- flanges (5 pcs.)
 - HAM3: Y- door flanges (5 pcs.)

Comments related to this report

gerardo.moreno@LIGO.ORG - 01:31, Monday 07 April 2025 (83763)VE

Link

(Travis, Tyler, Randy, Jordan, Janos, Gerardo)

Something to note during the removal of the -Y door for HAM1, a hair (see attached photo) was found embeded on one of the annulus o-rings, good eye Travis. Maybe we are due for a contamination control refresh.

Images attached to this comment