TITLE: 10/20 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 16mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY: H1 has been observing for about 1.5 hours.

• Wind speeds moderate, microseism just dropped below the 50th percentile
• All other systems nominal

Closes FAMIS 26307. Last checked in alog 80649. Service mode error, see SYSSTAT.adl is the only new difference between last week's check.

Laser Status:
    NPRO output power is 1.827W (nominal ~2W)
    AMP1 output power is 64.44W (nominal ~70W)
    AMP2 output power is 137.4W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 1 days, 22 hr 40 minutes
    Reflected power = 24.82W
    Transmitted power = 100.8W
    PowerSum = 125.7W

FSS:
    It has been locked for 0 days 0 hr and 4 min
    TPD[V] = 0.8048V

ISS:
    The diffracted power is around 2.4%
    Last saturation event was 0 days 0 hours and 5 minutes ago

Possible Issues:
    AMP1 power is low
    PMC reflected power is high
    Service mode error, see SYSSTAT.adl

Lockloss due to unknown cause. Not PSL this time. Still investigating.

Sun Oct 20 10:13:53 2024 INFO: Fill completed in 13min 49secs

TITLE: 10/20 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 09:46 UTC (5 hr lock).

Range is a bit low so will see what that's about.

H1 called for assistance at 09:42 UTC from the NLN timer expiring, by the time I logged in 4 minutes later we were Observing 09:46 UTC. There was a high state lockloss, at LASERNOISE_SUPRESSION.

TITLE: 10/20 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Fairly quiet shift with one lockloss followed by an automatic relock. H1 has now been observing for over 2 hours.

• 23:30 - H1 observing for 3 hours
• 01:32 - Lockloss - alog80767
• 02:47 - Observing

Lockloss @ 01:32 UTC - link to lockloss tool

No obvious cause; looks like a sizeable ETMX glitch about a half second before the lockloss.

TITLE: 10/19 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Its been a windy day, a few short locks today. Relocking has been easy, we've been locked for just over 3 hours.
LOG: No log

• 19:02 UTC lockloss
  • NPRO and FSS glitch during RESONANCE and CARM_TO_ANALOG
  • 20:21 UTC Observing

TITLE: 10/19 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 8mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY: H1 has been locked and observing for almost 3 hours.

• Wind speeds moderate, microseism trending down towards the 50th percentile
• All other systems nominal

Ryan C taking over for the last 4 hours but here's what happened in the first 4:

After issues getting the IMC locked during SRC_ALIGN, we manged to automatically get to NLN, Observing at 16:04 UTC.

SQZ Tuning: Dropped out of OBSERVING 17:14-17:26 UTC

• Adjusted temperature first to no avail.
• Ran SQZ Scan and managhed to recover ~8MPc of range

Calibration Sweep: Dropped out of OBSERVING 18:34 UTC - Lockloss 19:05 UTC

• Ran Broadband successfully
• Lockloss happened during last stages of simulines but since FSS was glitching pre-lockloss, the script automatically turned the sweep off.
• alog 80761

Lockloss (alog 80760): FSS Caused lockloss during simulines.

Timing glitch followup (Dave):  While yesterday's EY time glitch had no consequences. Dave said that it has been glitching by quite erratic numbers lately and has recommended another power supply swap (like the EX one recently).

Fast Shutter Guardian Glitch Followup: Dave found that the guardian fast shutter malfunction yesterday was caused because there was a 5s delay in data-grabbing but why that happened is unknown. He will post an alog about this soon.

Ran the usual calibration sweep following the wiki. IFO was thermalized (locked for 2.5 hrs but ~3hrs since MAX_POWER. Monitor Attached. Times are in GPS.

Broadband

Start: 1413398319

End: 1413398630

Simulines

Start: 1413398808

End (Lockloss): 1413399944

No files written on the screen due to abort from lockloss. Here's the error message:

2024-10-19 19:05:22,079 | ERROR | IFO not in Low Noise state, Sending Interrupts to excitations and main thread.
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:LSC-DARM1_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L3_CAL_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:CAL-PCALY_SWEPT_SINE_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L2_CAL_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L1_CAL_EXC
2024-10-19 19:05:22,080 | ERROR | Aborting main thread and Data recording, if any. Cleaning up temporary file structure.
ICE default IO error handler doing an exit(), pid = 3195655, errno = 32
PDT: 2024-10-19 12:05:26.374801 PDT
UTC: 2024-10-19 19:05:26.374801 UTC
GPS: 1413399944.374801

PSL Caused Lockloss during last part of Simulines Calibration.

IMC and ASC lost lock within 5ms of one another. (plot attached).

Sat Oct 19 10:12:16 2024 INFO: Fill completed in 12min 12secs

When Ibrahim relocked the IFO, the fast shutter guardian was stuck in the state "Check Shutter" 

This was because it apparently got hung up getting the data using cdsutils getdata. 

The first two attachments show a time when the shutter triggered and shows up in the HAM6 GS13s, and the guardian passes.  The next time shows our most recent high power lockloss, where the shutter also triggered and shows up at a similar level in the GS13s, but the guardian doesn't move on. 

The guardian log screenshot shows both of these times, it seems that it was still waiting for data and so the test neither passed nor failed. 

To get around this and go to observing, we manualed to HIGH_ARM_POWER. 

Vicky points out that TJ has sovled this problem for other guardians using his timeout utils, this guardian may need that added.  

Another thing to do is make things so that we would notice that the test hasn't passed before we power up. 

From 07:32 - 07:37 Fri 18oct2024 PDT we had another glitch on the EY CNS-II GPS 1PPS as read by the timing system's EY comparator. This had happened twice recently, 05:50-06:40 Mon 30sep2024 (50mins) and a pair the next day 07:14 (3mins) 07:42 (8mins) Tue 01oct2024.

attached are details of today's glitch, and a month trend showing all three.

On a related note, the EX CNS-II broke on the 8th Oct with a bad power supply (a wall wart). It might be prudent to schedule a replacement of EY's power supply next Tuesday.