We had two site-wide power gltiches at 04:15:48 and 04:52:39 PDT Mon 21oct2024.

Both were seen in CS and EX MAINSMON channels. EX saw voltage drop for all three phases, CS saw this for two phases. Voltage drop persisted for about 3 cycles and then recovered.

h1susauxh2 and h1susauxex rebooted on the first glitch at 04:15, their models restarted around 04:17. No other model restarted at these times. There were no IPC errors and no long cpu-runs. We currently don't know why these two susaux machines rebooted by themselves.

TITLE: 10/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 3mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

IFO is in MAINTENANCE due to NPRO and AMPs being disabled likely because of some power fault. Seems like the OWL shift was quite busy and that there are some PSL and LASER_PWR issues - will try to figure out what was going on.

H1 called for help after "Not ready for 15 minutes" at 11:38 UTC. IMC_LOCK and LASER_PWR are in fault. My medms aren't opening on my no machine unless I do guardmedm in the console. IMC lock is saying the PMC and FSS can't lock, LASER_PWR is in fault, unable to get to 2Ws "IMC has no power". Following the IMC locking issues wiki page I've tried DOWN initing ISC_LOCK, IMCLOCK, and LASER_PWR, the FSS noise eater is green.

TITLE: 10/21 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Two locklosses this shift, one from the PSL glitching, each followed by fully automatic relocks. Otherwise, it's been a fairly quiet evening.

• 23:30 - H1 observing for 1.5 hours
• 01:09 - Lockloss - alog80778
• 02:18 - Observing
• 03:25 - Lockloss - alog80779
• 04:23 - Observing

Lockloss @ 03:25 UTC - link to lockloss tool

The NPRO/FSS had been glitching with high frequency for about a minute prior to the lockloss, so I feel confident saying it's the cause here.

Lockloss @ 01:09 UTC - link to lockloss tool

No obvious cause; doesn't look like the FSS, and I don't see evidence of an ETM glitch or DARM wiggle.

H1 returned to observing at 02:18 UTC after a fully automatic relock.

TITLE: 10/20 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 21:31 UTC

Mostly quiet shift with one fussy lock acquisition.

Lock Acquisition had no special problems but rather, a slew of small annoyances (unknown cause Lockloss alog 80773):

• 1 PSL FSS glitch (losing lock from MOVE_SPOTS) and 1 other time from a lower pre-DRMI state
• Guardian stuck at DRMI_LOCKED_CHECK_ASC for over 30 mins due to unconverged SRC1,2 Y signals (despite touching)
  • Ended up inducing LL and it seems that the initial alignment done prior was no longer relevant (I'm guessing due to wind jump of 15mph during Lock acquisition)
  • Ran another initial alignment.
  • Lock Acquisition was fully auto thereafter.
• Wind hovering around 30mph (rough border for ALS and DRMI locking issues statistically)

Other:

• SQZ Tuning: Range was low due to SQZ angle being sub-optimal so went into comissioning (14:41-14:47 UTC) to adjust the opo temp (no improvement) and then SQZ_ANG_ADJUST, which improved the range by ~10MPc (I know!)
• PSL Chiller Check: Verbal alarm twice to check it. When Ryan S arrived, he showed me how to refill it, which he did this time (100ml) (alog 80775).

LOG:

None

TITLE: 10/20 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 16mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY: H1 has been observing for about 1.5 hours.

• Wind speeds moderate, microseism just dropped below the 50th percentile
• All other systems nominal

Closes FAMIS 26307. Last checked in alog 80649. Service mode error, see SYSSTAT.adl is the only new difference between last week's check.

Laser Status:
    NPRO output power is 1.827W (nominal ~2W)
    AMP1 output power is 64.44W (nominal ~70W)
    AMP2 output power is 137.4W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 1 days, 22 hr 40 minutes
    Reflected power = 24.82W
    Transmitted power = 100.8W
    PowerSum = 125.7W

FSS:
    It has been locked for 0 days 0 hr and 4 min
    TPD[V] = 0.8048V

ISS:
    The diffracted power is around 2.4%
    Last saturation event was 0 days 0 hours and 5 minutes ago

Possible Issues:
    AMP1 power is low
    PMC reflected power is high
    Service mode error, see SYSSTAT.adl