Displaying reports 4741-4760 of 83192.Go to page Start 234 235 236 237 238 239 240 241 242 End
Reports until 09:07, Monday 21 October 2024
H1 CDS
david.barker@LIGO.ORG - posted 09:07, Monday 21 October 2024 - last comment - 10:27, Monday 21 October 2024(80791)
Site-wide power glitches, reboots of h1susaux[h2, ex]

We had two site-wide power gltiches at 04:15:48 and 04:52:39 PDT Mon 21oct2024.

Both were seen in CS and EX MAINSMON channels. EX saw voltage drop for all three phases, CS saw this for two phases. Voltage drop persisted for about 3 cycles and then recovered.

h1susauxh2 and h1susauxex rebooted on the first glitch at 04:15, their models restarted around 04:17. No other model restarted at these times. There were no IPC errors and no long cpu-runs. We currently don't know why these two susaux machines rebooted by themselves.

 

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 09:28, Monday 21 October 2024 (80793)

Opened FRS32411

Notes from Fil:

The HAM6 PZT had to be restarted

Work on Beckhoff DAC power monitors

david.barker@LIGO.ORG - 09:33, Monday 21 October 2024 (80794)

Both GC and CDS UPS units switched to battery backup power for both of these times. The GC unit sent email notifications, the CDS unit did not.

GC was on battery for 5 seconds, CDS for only 1 second.

ibrahim.abouelfettouh@LIGO.ORG - 10:27, Monday 21 October 2024 (80796)

FMC-EX_MAINS Channels seeing the outage at the reported times.

Outage 1: 4:15:48

Outage 2: 4:52:39

Images attached to this comment
LHO General
ibrahim.abouelfettouh@LIGO.ORG - posted 07:40, Monday 21 October 2024 - last comment - 08:07, Monday 21 October 2024(80789)
OPS Day Shift Start

TITLE: 10/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 3mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

IFO is in MAINTENANCE due to NPRO and AMPs being disabled likely because of some power fault. Seems like the OWL shift was quite busy and that there are some PSL and LASER_PWR issues - will try to figure out what was going on.

Comments related to this report
ibrahim.abouelfettouh@LIGO.ORG - 08:07, Monday 21 October 2024 (80790)

Dave, Ibrahim

A significant power glitch on all 3 phases occured at 4:15:48 Local. PSL power issues Ryan C had were due to this (alog 80782). Plots below.

Images attached to this comment
H1 General (ISC, PSL)
ryan.crouch@LIGO.ORG - posted 04:58, Monday 21 October 2024 - last comment - 09:17, Monday 21 October 2024(80782)
OWL assistance

H1 called for help after "Not ready for 15 minutes" at 11:38 UTC. IMC_LOCK and LASER_PWR are in fault. My medms aren't opening on my no machine unless I do guardmedm in the console. IMC lock is saying the PMC and FSS can't lock, LASER_PWR is in fault, unable to get to 2Ws "IMC has no power". Following the IMC locking issues wiki page I've tried DOWN initing ISC_LOCK, IMCLOCK, and LASER_PWR, the FSS noise eater is green.

Comments related to this report
ryan.crouch@LIGO.ORG - 05:26, Monday 21 October 2024 (80783)

The MCs and IMs alignment all looks fine. The AMPs and NPRO all went to disabled about an hour ago.

ryan.crouch@LIGO.ORG - 05:54, Monday 21 October 2024 (80784)

The NPRO and AMPs are all disabled as of 11:15 UTC

Images attached to this comment
ryan.crouch@LIGO.ORG - 05:44, Monday 21 October 2024 (80786)PSL

Running PSL weekly scripy yielded the following:
Laser Status:
    NPRO output power is 0.1373W (nominal ~2W)
    AMP1 output power is -0.4512W (nominal ~70W)
    AMP2 output power is 0.1128W (nominal 135-140W)
    NPRO watchdog is RED
    AMP1 watchdog is RED
    AMP2 watchdog is RED
    PDWD watchdog is GREEN

PMC:
    It has been locked 0 days, 0 hr 1 minutes
    Reflected power = -0.1761W
    Transmitted power = -0.02753W
    PowerSum = -0.2036W

FSS:
    It has been locked for 0 days 0 hr and 0 min
    TPD[V] = -0.01628V

ISS:
    The diffracted power is around 3.0%
    Last saturation event was 0 days 1 hours and 26 minutes ago


Possible Issues:
    NPRO power is low
    AMP1 power is low
    AMP2 power is low
    NPRO watchdog is inactive
    AMP1 watchdog is inactive
    AMP2 watchdog is inactive
    FSS TPD is low
    NPRO error, see SYSSTAT.adl

Images attached to this comment
ryan.crouch@LIGO.ORG - 05:57, Monday 21 October 2024 (80787)

After talking to Jason, there's not much to do about this right now. More investigation needs to be done to see what happened to the laser, potentially needs to be swapped.

richard.mccarthy@LIGO.ORG - 06:57, Monday 21 October 2024 (80788)CDS, PSL

There power issues on the Hanford site that could have easily glitched our power and caused the npro to trip.

ryan.short@LIGO.ORG - 09:17, Monday 21 October 2024 (80792)PSL

Fil re-enabled the PMC high voltage after arriving on site, then I brought the PSL back up with little issue. One hiccup I ran into was with the new PDWD for the amplifiers; when the NPRO shut off this morning, the regular power watchdogs tripped the system off, but the PDWD did not. This meant I had to disable the PDWD before I was able to reset the system and turn the NPRO back on. After that, there were no issues recovering the PSL.

LHO General
ryan.short@LIGO.ORG - posted 22:01, Sunday 20 October 2024 (80781)
Ops Eve Shift Summary

TITLE: 10/21 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Two locklosses this shift, one from the PSL glitching, each followed by fully automatic relocks. Otherwise, it's been a fairly quiet evening.

H1 General (Lockloss, PSL)
ryan.short@LIGO.ORG - posted 20:36, Sunday 20 October 2024 - last comment - 21:37, Sunday 20 October 2024(80779)
Lockloss @ 03:25 UTC

Lockloss @ 03:25 UTC - link to lockloss tool

The NPRO/FSS had been glitching with high frequency for about a minute prior to the lockloss, so I feel confident saying it's the cause here.

Images attached to this report
Comments related to this report
ryan.short@LIGO.ORG - 21:37, Sunday 20 October 2024 (80780)

H1 back to observing at 04:23 UTC. Fully automatic relock.

H1 General (Lockloss)
ryan.short@LIGO.ORG - posted 19:21, Sunday 20 October 2024 (80778)
Lockloss @ 01:09 UTC

Lockloss @ 01:09 UTC - link to lockloss tool

No obvious cause; doesn't look like the FSS, and I don't see evidence of an ETM glitch or DARM wiggle.

H1 returned to observing at 02:18 UTC after a fully automatic relock.

LHO General
ibrahim.abouelfettouh@LIGO.ORG - posted 16:32, Sunday 20 October 2024 (80777)
OPS Day Shift Summary

TITLE: 10/20 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 21:31 UTC

Mostly quiet shift with one fussy lock acquisition.

Lock Acquisition had no special problems but rather, a slew of small annoyances (unknown cause Lockloss alog 80773):

Other:

LOG:

None

LHO General
ryan.short@LIGO.ORG - posted 16:00, Sunday 20 October 2024 (80776)
Ops Eve Shift Start

TITLE: 10/20 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 16mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY: H1 has been observing for about 1.5 hours.

H1 PSL (PSL)
ibrahim.abouelfettouh@LIGO.ORG - posted 11:58, Sunday 20 October 2024 - last comment - 14:57, Sunday 20 October 2024(80774)
PSL Weekly Report - Weekly FAMIS 26307

Closes FAMIS 26307. Last checked in alog 80649. Service mode error, see SYSSTAT.adl is the only new difference between last week's check.


Laser Status:
    NPRO output power is 1.827W (nominal ~2W)
    AMP1 output power is 64.44W (nominal ~70W)
    AMP2 output power is 137.4W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 1 days, 22 hr 40 minutes
    Reflected power = 24.82W
    Transmitted power = 100.8W
    PowerSum = 125.7W

FSS:
    It has been locked for 0 days 0 hr and 4 min
    TPD[V] = 0.8048V

ISS:
    The diffracted power is around 2.4%
    Last saturation event was 0 days 0 hours and 5 minutes ago


Possible Issues:
    AMP1 power is low
    PMC reflected power is high
    Service mode error, see SYSSTAT.adl

Comments related to this report
ryan.short@LIGO.ORG - 14:57, Sunday 20 October 2024 (80775)

Looks like I had left the PSL computer in service mode after work on Tuesday; fortunately this doesn't actually effect anything operationally, so no harm done. I've just taken the computer out of service mode.

I also added 100mL of water into the chiller since Ibrahim had gotten the warning from Verbal today.

Displaying reports 4741-4760 of 83192.Go to page Start 234 235 236 237 238 239 240 241 242 End