Austin, Dave:

At 16:15PDT h1iopseih45 went into DACKILL mode.

There were no dmesg logs corresponding to this time (see below) and no IOP processing time overrun.

Austin put HAM4,5 suspensions into a safe state and I restarted the models, which has cleared the problem.

Mia Culpa, I mistakenly restarted h1seih16 models first before realizing I was on the wrong front end. Austin is recovering HAM6 supsensions as well, apologies.

h1seih45 recent dmesg logs:

Fri May 26 15:16:58 2023] rts_cpu_isolator: entering ligo_play_dead, cpu handler = 00000000b41de7f7
[Fri May 26 15:16:58 2023] rts_cpu_isolator: calling LIGO code
[Fri May 26 15:16:58 2023] h1isiham5: INFO - CPY2=1 CPY2OFFSET=1500 CPY1SZ=1500 CPY2SZ=536 DBLOFFSET=2036
[Fri May 26 15:16:59 2023] h1isiham5: INFO - Controller initialization complete, starting front end control loop
[Fri May 26 15:17:04 2023] smpboot: CPU 5 didn't die...
[Fri May 26 15:18:11 2023] h1isiham4: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Fri May 26 15:18:16 2023] h1isiham5: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Tue Jun 13 12:30:25 2023] h1isiham4: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Sat Jun 24 16:23:32 2023] h1hpiham4: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Sat Jun 24 16:24:21 2023] h1hpiham5: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.

Following the recent lockloss, I have followed Ryan S. instructions to try and get the slow offset servo (H1:PSL-ISS_SECONDLOOP_REFERENCE_SERVO_OUT16) to 0. I tried to do this by adjusting the H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET. I first tried offsetting the servo value, the servo was at -322, so I set the output to +322, which made the servo value worse. I then tried going in the positive direction which seemed to have made it better. After doing some back and forth moving the output, I was finally able to get the servo to ~-0.3 by setting the output to 949 - screenshot attached. I then cleared the history on the servo overview. Just in case we need to revert later, the original value for the output was +1272.

LOCKLOSS @ 9:13 - And so ends the longest lock we have had for O4, 33:34. Something has tripped the ISI and HEPI HAM 4/5 WDs. I have attached the plots for HAM 4 HEPI and ISI. I believe this was caused by a CDS failure, as I see H1IOPSEIH45 is reading red for DAC and DK, will contact Dave.

TITLE: 06/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 138Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 17mph Gusts, 11mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.05 μm/s 
QUICK SUMMARY:

- H1 looks stable and has been locked for 33.5 hours :)

- SEI motion elevated, wind on the rise, will monitor

- SEI/DMs ok

TITLE: 06/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 140Mpc
SHIFT SUMMARY:

Locked the whole shift, 33:30!

In Observing since 06/23 23:08

PSL Anteroom dust alarms Red and Yellow throughout the afternoon, probably from the wind picking up

Handing off to Austin

LOG:

No log for this shift

We have a new record lock time for O4, we've been locked for 29:21! We've been in Observing since 06/23 23:08

It hasnt grown big enough to give any verbals alerts but I've seen guardian damp PI mode 24 effectivly quite a few times on NUC25

Ground motion at the arms is increasing as is the wind speeds, gusts up to the high 30s

Sat Jun 24 10:10:10 2023 INFO: Fill completed in 10min 9secs

TITLE: 06/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 135Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.04 μm/s
QUICK SUMMARY:

• Locked for 25:25
• CDS/VAC/Dust looks good, low seism
• Picket Fence seismometers are still mostly down

TITLE: 06/24 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
SHIFT SUMMARY:

H1 has been locked for over 25hrs! (that's two 24+hr locks in a row).  This also happened during the shift.

HLV Range on nuc27 stopped displaying L1 range about 2hrs ago.  Got it back by bringing up an opened terminal on this machine, scrolling up, and running:  "startup/launch.sh".  This did the trick (I saw that Erik did this previously when Camilla mentioned it in mattermost.)

Since there has not been a lockloss, did not get to perform the ISS 2nd Loop task Ryan S requested (will pass on to Ryan C).
LOG:

n/a

H1 continues to hum along (currently at 21.25+hr lock w/ range hovering just under 140Mpc).  There were a few glitches (one noted in Verbal at 8:20utc and others not noted on verbal but seen on DARM FOM).  Winds are calm, as well as the earth in general for the last 6hrs.

TITLE: 06/24 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 138Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 10mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.04 μm/s
QUICK SUMMARY:

Smooth hand-off, once again, this time from Austin with H1 locked since I last was with it (this is how it should be!!)  Range continues to be a little lower than yesterday's lock, but Austin & Ryan C rode through a flurry of earthquakes and even a quick windstorm with no issues. 

Austin handed over a ISS 2nd Loop task Ryan S wanted operators to do during the next lockloss.

TITLE: 06/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 136Mpc
SHIFT SUMMARY:

- Arrived with H1 locked as of 13:37 UTC

- Wind speeds increased to ~35-40 mph but have since died down, but other than that the IFO looks very stable

- Leaving H1 to Corey going on a 17 hour lock :)
LOG:                                                                                                                                                                                                                                                                                                                                                         

H1 has been locked for 13 hours. All looks stable, ground motion is low, wind however seems to be on the rise (~30-35 mph gusts).

I took a high resolution spectrum of DARM (attachment 1) and looked around 27-29 Hz because this is a region where we have seen large peaks we think come from the HSTS and HLTS mirrors. An assumption I made in the past is that they must come from mirrors in HAM2 and HAM3 due to less isolation in those chambers relative to HAM4 and HAM5. However, I think that assumption may be incorrect.

Attached is the high resolution spectrum. I identified 5 peaks evident. I used the table of modes in 49643 to identify each peak.

27.41 Hz - PR2 bounce mode

27.44 - SRM bounce mode

27.5 Hz - SR2 bounce mode

27.71 Hz - SR3 roll mode

28.2 Hz - PR3 roll mode

Note: according to this document: LIGO-T1700111, the HLTS roll modes are around 28 Hz while the bounce modes are at 40 Hz (how fun and confusing!).

Notably missing here is the PRM bounce mode. I am also not convinced that any of these modes are from MC1-3 due to the fact that none of these peaks are visible in the IMC WFS spectra (see second attachment).

The best way to confirm that these identifications are correct is to try to excite each of these modes in lock, and check the high resolution spectrum. I think there is good motivation to try this measurement for these reasons:

• if we assume we see these peaks because of poorer isolation in HAM2/HAM3, the presence of strong bounce and roll modes from mirrors in HAM4 and HAM5 would disprove that assumption
• if we see these peaks from most of the HSTS/HLTS in the recycling cavities, that suggests that something like BRDs or blade springs would be the best method to reduce or remove them from the spectrum
• why doesn't the PRM mode appear? Maybe there is something to learn there.

These are narrow peaks, but we have seen times in the past when the peaks are rung up enough to cause broadband excess noise. I think it is worth trying to reduce these peaks in some way.

Closes 25587

All looks ok, but does seems like there was a glitch ~3.5 days ago, which shows up mostly on EX_FAN2_570_1/2 (approx. 1:00 AM 6/21 PDT), but the values for all vibrometers seem to have dropped to their nominal state.

Jamie, Jenne, Keith R, Erik, Dave:

The CW psinject injection code on h1hwinj1 would not restart this afternoon due to tconvert issues. When running tconvert in the command line it was showing the "leap seconds data file expired" warning which the control room workstations saw a few weeks ago. I increased the expiration time and verified that the command line warning was gone. We restarted the psject process on h1hwinj1 it started running again.

In the mean time LLO has new psinject code which replaces tconvert with gpstime, which we will do at LHO on an upcoming Tuesday.

TITLE: 06/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 135Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 8mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.05 μm/s 
QUICK SUMMARY:

- H1 has been locked for 9:30 - acquired 13:37

- CDS/SEI ok

- Current commissioning for H1 ongoing

Ran stochastic short and long hardware injections, in coincidence with LLO.  These are of the style described in alog 69723.

hwinj stochastic short --gps 1371592362 --run

hwinj stochastic long --gps 1371593299 --run    (Interrupted by earthquake.  LLO lost lock, so I ctrl-c'd)