Displaying reports 1761-1780 of 83107.Go to page Start 85 86 87 88 89 90 91 92 93 End
Reports until 20:14, Sunday 06 April 2025
H1 CDS
david.barker@LIGO.ORG - posted 20:14, Sunday 06 April 2025 - last comment - 14:58, Friday 25 April 2025(83758)
Recovery from power outage

Richard powered up h1seih23 from the front panel, it is now running.

We power cycled h1psl0, it is back.

Erik reconfigured EX Dolphin front ends to communicate with port 8 of the EY Dolphin switch to get them to get the models going. CDSRFM at EX points to port 7 of EY's switch.

All models are running, only IPC errors are long-range-dolphin.

Should someone from operations start activating the SEI and SUS systems?

Images attached to this report
Comments related to this report
jordan.vanosky@LIGO.ORG - 23:09, Sunday 06 April 2025 (83762)
Link

After the power outage, I went out to the site ~10pm Pacific to check status of purge air skid. Kobelco, water pump and drying towers were all running upon arrival. I verified the dew point was still ok (-41.8C measured at HAM1) and verified the drying towers switched normally. Note: Purge air was only connected to HAM1 since doors are off, corner volume is vented but not actively purging with no doors off.

While there I also restarted the controller for 3 of the FCT ion pumps and reset solenoid valve on aux cart connected to GV5 annulus. All looks ok. Left site ~10:40pm.

david.barker@LIGO.ORG - 20:15, Sunday 06 April 2025 (83759)
Link

h1guardian1 did not reboot, here is the guardian MEDM

Images attached to this comment
david.barker@LIGO.ORG - 20:49, Sunday 06 April 2025 (83760)
Link

Systems currently down as of 20:49:

FMCS/BACNET

Corner Station HEPI Pump Controller

PSL Diode Room slow controls

CP1 overfill

NCAL

BRS

HWS

DTS test stand

david.barker@LIGO.ORG - 21:30, Sunday 06 April 2025 (83761)
Link

To continue running vac alarms overnight I have bypassed the currently active alarms for the next 24 hours.

Bypass will expire:
Mon Apr  7 09:28:54 PM PDT 2025
For channel(s):
    H0:FMC-CS_CY_H2O_PUMPSTAT
    H0:FMC-CS_CY_H2O_SUP_DEGF
    H0:FMC-CS_FIRE_PUMP_1
    H0:FMC-CS_FIRE_PUMP_2
    H0:FMC-EX_CY_H2O_PUMPSTAT
    H0:FMC-EX_CY_H2O_SUP_DEGF
    H0:FMC-EY_CY_H2O_PUMPSTAT
    H0:FMC-EY_CY_H2O_SUP_DEGF
    H1:CDS-DTS_ENV_TEMPERATURE_DEGF
    H1:CDS-STAT_NUMBER_OF_ERRORS
 

gerardo.moreno@LIGO.ORG - 14:58, Friday 25 April 2025 (84134)VE
Link

Regarding the 4UHV ion pump controller, the autostart setting is enabled now, thus the unit should power ON after a power glitch.  See some of the instructions to access the general menu.

Images attached to this comment
H1 CDS
david.barker@LIGO.ORG - posted 18:25, Sunday 06 April 2025 - last comment - 19:54, Sunday 06 April 2025(83753)
Power event, most front ends down, Beckhoff slow controls down, DAQ down.

From UPS emails it appears the site has had mutiple power issues starting 18:06 PDT extending to 18:20

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 19:36, Sunday 06 April 2025 (83754)
Link

Richard is on his way to the site. Erik and I remotely brought h1susauxb123 (edc) back online, which got the DAQ to restart itself so we are trending vacuum channels again. Data gap for VE data is 18:05 - 18:40.

Current CDS issues:

h1seih23 ipmi not working, computer is still down.

IX Dolphin switch at EX is not working correctly. h1susex, h1seiex, h1iscex are down. Erik is trying a workaround.

h1psl0 is down, it didn't actually power off, we are not sure we should get that going again.

Images attached to this comment
david.barker@LIGO.ORG - 19:41, Sunday 06 April 2025 (83755)
Link

MSR UPS messages:

Name     : ups-msr-0
Location : LHO MSR
Contact  : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 04/06/2025
Time: 18:20:42
Code: 0x0002
Critical - System: Warmstart.

 

Name     : ups-msr-0
Location : LHO MSR
Contact  : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 04/06/2025
Time: 18:20:50
Code: 0x0101
Informational - UPS: Restored the local network management interface-to-UPS communication.

david.barker@LIGO.ORG - 19:51, Sunday 06 April 2025 (83756)
Link

GC OSB UPS messages (times out by one hour)

Name : lho-gc-osb-ups
Location : LHO OSB Mechanical Room (rm 160)
Contact  : lho-gc-root@ligo-wa....
Serial Number : 5A1315T04169
Device Serial Number : PS1316131451
Date : 04/06/2025
Time : 17:06:10
Code : 0x0235
Critical - UPS: A site wiring fault exists.

Date : 04/06/2025
Time : 17:06:10
Code : 0x020F
Critical - UPS: An input voltage or frequency problem prevents switching to bypass mode.

Date : 04/06/2025
Time : 17:06:26
Code : 0x010A
Informational - UPS: No longer on battery power.

Date : 04/06/2025
Time : 17:06:44
Code : 0x0236
Informational - UPS: A site wiring fault no longer exists.

Date : 04/06/2025
Time : 17:06:44
Code : 0x0210
Informational - UPS: An input voltage or frequency problem no longer prevents switching to bypass mode.

 

david.barker@LIGO.ORG - 19:54, Sunday 06 April 2025 (83757)
Link

GC LSB (one hour out)

Time : 17:18:14
Code : 0x0109
Warning - UPS:  On battery power in response to an input power outage.

Critical - UPS: An out of tolerance input voltage or frequency , prevents switching to bypass mode.

Time : 17:18:33
Code : 0x0210
Informational - UPS: An out of tolerance input voltage or frequency no longer prevents switching to bypass mode

Date : 04/06/2025
Time : 17:18:36
Code : 0x010A
Informational - UPS: No longer on battery power.

 

LHO VE
david.barker@LIGO.ORG - posted 10:37, Sunday 06 April 2025 (83752)
Sun CP1 Fill

Sun Apr 06 10:06:44 2025 INFO: Fill completed in 6min 40secs

 

Images attached to this report
LHO VE
david.barker@LIGO.ORG - posted 10:25, Saturday 05 April 2025 (83750)
Sat CP1 Fill

Sat Apr 05 10:05:42 2025 INFO: Fill completed in 5min 38secs

 

Images attached to this report
H1 CDS
david.barker@LIGO.ORG - posted 09:17, Saturday 05 April 2025 - last comment - 10:36, Saturday 05 April 2025(83747)
h1daqnds1 having issues serving data, please use h1daqnds0 for now while we work on this

Note that the web trends use nds0 and are updating.

Comments related to this report
david.barker@LIGO.ORG - 10:20, Saturday 05 April 2025 (83748)
Link

FRS33784 ticket created for this issue.

david.barker@LIGO.ORG - 10:23, Saturday 05 April 2025 (83749)
Link

Jonathan restarted rts-daqd on h1daqnds1 at 09:37 Sat 05apr2025PDT which cleared the immediate issue.

It appears that nds1 had a connection burst of over 300 connections requests at 23:30 Fri 04apr2025PDT which caused the issue. The daqd process continued to run after that but was no longer accepting any connection requests until the restart this morning.

david.barker@LIGO.ORG - 10:36, Saturday 05 April 2025 (83751)
Link

I rebooted nuc5 at 10:28 to get the Seismic BLRMS FOM running again. I think the other FOMs can wait till Monday.

LHO General
tyler.guidry@LIGO.ORG - posted 17:18, Friday 04 April 2025 - last comment - 09:06, Saturday 05 April 2025(83745)
Potable Supply Valved Out to All Buildings 
I received a call from Oli at 11:46 this morning informing me that water pressure at the building had fallen. I logged into the FMCS remotely and found that the potable holding tank had been completely drained and only one RO was making water. Before leaving home to come out to site, I contacted Randy and had him manually disable the domestic pressure skid and force the second RO skid on. At this point, because of the known leak, Richard and I have decided that the RO system will replenish the potable holding tank but that the water supply will remain valved out so as not to continue to tax the entire water treatment plant. 

C&E Trenching who had already been slated to come make the repairs next week has been contacted. After explaining the current situation, they have graciously agreed to move us up in their schedule and will be on site Monday Morning to being the work. Once the repair is made, water should be restored.

T. Guidry
Comments related to this report
david.barker@LIGO.ORG - 09:06, Saturday 05 April 2025 (83746)
Link

RO went into alarm Friday 23:53 04apr2025 PDT. I have temporarily removed it from the alarms system and restarted the service 09:04 sat 05apr2025.

LHO VE
janos.csizmazia@LIGO.ORG - posted 15:38, Friday 04 April 2025 (83744)
2025 April vent - VAC diary 
Today's activities:

- All the turbos have been valved out and switched off
- The corner was vented through the Y-manifold vent valve, with 3-4 Torr/min speed - it took 3.5 hrs
- As a side effect, the current of GV5's AIP went up, so now this Annulus volume is being pumped with an aux cart
- Another aux cart is pumping the RGA in the corner (between HAM5 and HAM4)
H1 General
oli.patane@LIGO.ORG - posted 14:46, Friday 04 April 2025 (83743)
Ops Day Shift End

TITLE: 04/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: Done early for today! The corner is vented! It's possible that not everything got finished today due to a lack of water pressure causing people to leave early, but at least all of the most important tasks have been completed, if not all of them.

LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

Start Time System Name Location Lazer_Haz Task Time End
15:37 FAC Randy, Mitchell LVEA n Craning 16:52
15:54 FAC Kim LVEA n Tech clean 17:42
16:09 VAC Jordan, Gerardo, Janos LVEA n Corner vent 21:41
16:57 ISC Rahul, Siva, Mayank OpticsLab y(local) ISS Array (Siva out 18:31) 20:03
17:29 FAC Mitchell MY n Looking for parts 17:57
17:30 ISC Jennie OpticsLab y(local) ISS Array 18:30
17:54 PCAL Tony PCAL Lab y(local) Measurement 18:32
17:58 FAC Mitchell LVEA n Installing HAM1 ISI installation fixture 19:02
19:14 VAC Janos EY n Picking up equipment 20:07
20:32 PCAL Tony PCAL Lab y(local) Getting ready for weekend lab mode 20:40
20:38 FAC Tyler LVEA n Kickin tires with the VAC guys 20:49
LHO VE
david.barker@LIGO.ORG - posted 10:21, Friday 04 April 2025 - last comment - 10:22, Friday 04 April 2025(83741)
Fri CP1 Fill

Fri Apr 04 10:06:13 2025 INFO: Fill completed in 6min 10secs

 

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 10:22, Friday 04 April 2025 (83742)
Link

For the duration of the vent I have removed PT132_MOD2 gauge from the alarm system.

Displaying reports 1761-1780 of 83107.Go to page Start 85 86 87 88 89 90 91 92 93 End