Tue Sep 09 10:06:51 2025 INFO: Fill completed in 6min 47secs

FAMIS 27778

I closely inspected the lines starting from the chillers and going to the tables. I saw no evidence of leaks, cracks or stressed pipes along their entire lengths. All notes from the last inspection in March 2025 [alog83162] still stand, but no major issues.

FAMIS 28949

Reboot at 1600UTC(0900PT). Every node came back up without issue, Ryan C helped me bring back nodes to their nominal requests for maintenance.

TITLE: 09/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY: Locked for 4 hours, pem and charge measurements running. Maintenance day will begin at 0800 PT. Rough recovery last night from the earthquake and eletrical storm, but Ibrahim got us back up.

Got the wake up call at 338amPDT.  And literally while I was waking up and trying to remember what to do for the NOTIFY + MANAGER guardian nodes (I just "init-ed" both...is that the right thing?  that's what both of their User Messages said to do), I noticed that there was a Message that said there was an issue with the SQZ MANAGER.  

As I was getting ready to open up the SQZ guardians and assess (I'm pretty sure I spelled that right--I remember Hugh giving me grief for misspelling it in an alog before!), H1 was already back to OBSERVING.  I literally did nothing!  H1 was in OMC Whitening at 932utc, sent wake up call at 1037utc, and automatically went to Observing on its own at1043utc.  I'm going to go back to bed ASAP!  
 

Great job getting H1 thru all the environmental catastrophes at the end of your shift, Ibrahim! :)

TITLE: 09/09 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

IFO is in OMC_WHITENING and LOCKING

The first 5 hours of this extended shift were quiet and well behaved. Truly the calm before the earthquake before the storm. Lockloss alog 86792.

Thanfully the next 3 hours were action packed due to 2 environmental events: a 5.8 EQ (and 4.9 aftershock) off the coast of Oregon and an electrical storm power glitch.

The Earthquake (alog 86793)

5.8 EQ from Oregon downed the IFO before SEI or SUS had a chance to prep.

• A very violent lockloss which tripped: ISI EY ST1 WD, ISI EY ST2 WD, SUS TMSY WD, ISI IX ST1 WD, ISI IX ST2 WD, ISI IY ST1 WD, ISI IY ST2 WD, ISI EX ST2 WD, SUS IX WD, SUS IY WD, SUS EX WD, SUS EY SW WD, SUS IY SW WD, SUS IX SW WD, SEI EY SW WD, SEI IY SW WD, SEI IX SW WD, HEPI EY WD (twice), HEPI IY WD (twice), HEPI IX WD (twice), ISI IY ST2 WD, IY Hardware WD (with warnings from all the others) (pic attached)
• EX overperformed its sibling QUADs by being the only one that didn't completely trip somehow
• Jim and Dave came online to the rescue and we immediately began diagnosing and eventually, untripping.
• IY Hardware WD had tripped so I went into the CER to untrip it successfully
• While I was in the CER, the lights flickered, I heard a loud rumble and when I was back some more greens were red - that was the high voltage.

The Storm (alog 86795)

The storm caused a slight power glitch that caused the high voltage for both HAM6 and the PSL to turn off.

• The PMC was flickering on the camera and I called Ryan S who pointed me to the right place to realize the high voltage had tripped. Jim simultaneously realized this on DIAG_MAIN.
• I called Richard (and TJ but Richard called back and he walked me through how to turn the high voltage back on (pic attached)
• I once again ventured into the CER, this time to the mezzanine and turned on the high voltage
• When I arrived, Jim and Dave had finished untripping the WDs and with that, all the reds were green and that's universal for go - so I started relocking.

The Aftershock:

While relocking in INITIAL_ALIGNMENT, we got an aftershock (4.9 EQ) from the same location, wich prompted EQ mode to turn on.

• I had to INIT some nodes since some of the QUADs were in auto and not managed, which marked the end of my OPS intervention. Just kidding! See below.
• Like most sequels these days, it was underwhelming. It did not even affect initial alignment, which ran fully auto.
• Funnily though, we went into MICROSEISM since the secondary microseism was high enough after EQ mode went to CALM

Initial alignment finished, I began locking, DRMI caught immediately and thanks to the awesome power of nature, this shift was not boring. Except of course, the OMC is not locking (despite turning the high voltage back on) - see below for the 3rd installment nobody asked for.

The OMC:

The OMC high voltage also tripped though I did turn it on (narrator: or so he thought)

• The problem it seems is that the sweep it was doing in looking for lock was going between -40 and 40 instead of -50 and 50, and it's jumping instead of sweeping.
• After ~10 dead ends, and one accidental lockloss trying to fix this, I think I found the solution.
• My guess is that the issue isn't the high voltage but that guardian is not doing a fine sweep to find the right PZT2 steps and is jumping instead. From alog 85876, I think we should auto the OMC guardian and step through the high voltage in 1.0 steps to get the OMC locked.
• Last time the High Voltage tripped, this happened too. Thus, I anticipate we will be stuck at PREP_FOR_DC_READOUT. From context clues, I think this has to be done at DARM_OFFSET properly instead of at PREP_DC_READOUT but frankly, I have no idea.
• Alright guess what? It was the high-voltage AGAIN. The attached picture is from hours ago when I turned it on the first time. It crapped out again, I turned it back on - it's working now.
• We're past PREP_FOR_DC_READOUT. The OMC is still in AUTO so now figuring out how to put it back in MANAGED. Honestly, we should check this power supply out on Tuesday, which is in a few hrs.
• Re-managed OMC_LOCK by Temp-manual -> INIT -> whatever-state-I-manualled-at.

Other:

The DARM FOM is having trouble connecting to NDS so I can't see the Violin Modes but they're extremely high (maybe due to the possbily fake measured 573 Lin Velocity of that Oregon EQ). Violin medm attached.

Anyway, I should log off now before I summon a tornado or something. Honestly, this has been a fantastic learning oppurtunity (and don't worry I have the day off tomorrow).

LOG:

After the watchdogs were reset a lightening strike caused a power glitch. GC UPS reported going into and off of battery power.

h1susauxh2 and h1susauxex went offline, turned out they were rebooting themselves and came back after a few minutes.

PSL is in a bad way, Ibrahim is calling for support.

The lights in my house flickered at the time I saw the sus-aux machines go down. We have been having a storm roll over us for the past hour, moving from Oregon northwards.

SWWDs for ETMY, ITMY and ITMX have tripped.

HWWD for ITMY has just tripped, its RMS is significantly higher than the others.

Strangely ETMX SWWD and HWWD have not come close to tripping.

Lockloss due to an EQ that hit before EQ mode could activate. Seems to be either very local or very large - not on USGS yet.

This afternoon a 6.4 mag eq in Fiji provided another good opportunity to test the high asc gain eq mode. This time peak velocities were about 3.6 micron/s on peakmon, around 2 micron/s on the .03-.1hz blrms on the wall fom. We were out of observe for ~1hr, but we probably would have been down for at least 2hrs without the transition. Definitely one of the largest eqs we've ridden out, ever.

On Sept 4th we had a longer commissioning period to allow us to heat up OM2.  The main goal was to use this to characterize the mode matching of the arm cavities to the OMC, but we also made some other interesting measurements.

Summary:  heating up OM2 now costs us about 1% of optical gain, while it used to cost us 2%.  Heating up OM2 also changed the SRCL offset needed to get flat squeezing significantly, and reduced the amount of squeezing that we could get (without adjusting psams).

The interferometer unlocked (86727) while Jennie Wright was running the DARM offset step script with OM2 cold, to get a measurement of HAM6 throughput and look at OMC refl before heating up OM2.  As that link says, there wasn't an obvious connection between the DARM offset script which was in it's final steps (nearly back to normal) when the lockloss happened. 

We turned on the OM2 heater while relocking after some back and forth, then had an commissioning caused lockloss while trying to recover.  By the time we were relocked, OM2 was heated up according to thermistor 2, which is the thermistor who's timescale matched the timescale of optical gain changes in the past (see screenshot of June 2023 example).  We also had a large earthquake while relocking, so we paused after power up and before going to nominal low noise, so we do not have tracking of the optical gain using pcal in the first 30 minutes of this thermalization. 

The optical gain in this thermalization seemed to be fairly similar to the previous lock where OM2 was cold in the first hour.  The change in optical gain between OM2 hot and cold was much smaller this time around, so we needed to see the full thermalization in order to see what the gain change was.  We set the OM2 heater off 4 hours and 5 minutes after the power up, the attached screenshot shows a trend of optical gain during the previous thermalization while OM2 was cold, with a vertical cursor 4 hours 5 minutes into the lock.  The horizontal cursors show where the optical gain was at 4 hours and once the thermalization was complete, the optical gain continued to increase by 0.2% after the first 4 hours. The next screenshot shows vertical cursors also at the time of power up and 4 hours later, and the horizontal cursors are the same as on the previous screenshot (OM2 cold).   It seems that the optical gain was about 1.1% lower with OM2 hot than cold, although fitting these two thermalizations to an exponential might bring them closer by as much as 0.2%. Our current ring heater settings are 0W on ITMY, 1.5 W/segment on ETMY and 0.44W/segment on ITMX and 1W/segment on ETMX.

Since this is a different result than in the past times of OM2 changes, I've gone back to look at old times when we did this change.  One possible explanation for the difference could be ring heater settings being different. 

• 12/18/2023, cooled off while unlocked, optical gain increased 1.8% ETMX +ETMY RH 1W, ITMX 0.44W
• 9/19/2023, heated while unlocked, optical gain dropped 1.9%   (this time is confusing, it seems like the thermistors which had been disconnected were plugged in and looked to be thermalizing, but the set power was unchanged).
• 7/19/2023, heated in steps, 71484 ifo unlocked and relocked several times, perhaps calibration was reset making optical gain comparison confusing. (it did seem to drop by about 2%, but this seems more like a calibration reset) ETMX + ETMY RH 1W/segment, ITMX RH 0.44W/segment
• 7/12/2023 OM2 cooled off in lock, lockloss happened in the early part of the cooling (2% optical gain increase), ETMX + ETMY RH 1W/segment, ITMX RH 0.44W/segment
• 6/27/2023 OM2 heated while in lock, optical gain dropped by 2%, ETMX + ETMY RH 1W, ITMX RH 0.44W/segment, 0 on ITMY.

Once the OM2 heater was turned back on, the optical gain increased by nearly 1%, but the IFO lost lock before that thermalization finished.  There was a 2% decrease in POP18 during the cool off and a 3 urad shift in SRM top mass. 

While OM2 was hot, we did a few tests.

• We had a large change in the SRCL offset needed to reduce the squeezing rotation: 86739  This is interesting, in the past we have tried to change SRM alignment to see an impact on this number and it was a much smaller change.
• No squeezing time for a senstitivty comparison (not much change): 86788
• quick squeezer data set (before SRCL offset updated: 86736 with SRCL offset updated: 86739 ) , to see a comparison with OM2 cold see Camilla's plot from this morning
• Jennie Wright did DARM offset steps with OM2 hot, and cold.

TITLE: 09/08 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: We stayed locked the whole shift, 14 hours. We dropped obs to ride though a 6.4, we're still waiting to leave EQ mode.
LOG:                                                                                                                                                                     

• 15:20 UTC Superevent S250908y
• 15:43 - 18:32 UTC Commissioning
• 22:12 UTC Earthquake mode from a 6.4 from Vanuatu
• 22:28 UTC We dropped Observing after I pressed the ASC_hi_gain button
  • I transitioned back at 23:30 UTC
  • 23:31 UTC Observing

TITLE: 09/08 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: EARTHQUAKE
    Wind: 14mph Gusts, 8mph 3min avg
    Primary useism: 0.42 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

IFO is in NLN as of 10:30 UTC (14 hr lock)

We are temporarily out of absorbing as we try to ride out a 6.4 EQ using ASC High Gain (new feature), alog 85740

After SEISMIC goes back to CALM, we'll turn off the high gain and go back to OBSERVING

Link to report here.

• Average observing time: 71.2%
• Range was generally between 145-160 Mpc throughout the week, except for a drop at 1100-1200 on Wednesday Sept 3 when it was around 135 Mpc, mostly likely due to SQZ issues
• EQs and anthropogenic ground motion may contribute to increased glitch rates and noise in h(t) several times throughout the week
• Tuesday had quite strong violin modes (esp. 500 Hz) after relocking
• Magnetic corner station plots for Thursday Sept 4 are blank
• Common Hveto channels throughout the week: H1:ASC-Y_TR_A_NSUM_OUT_DQ, H1:ASC-Y_TR_B_NSUM_OUT_DQ, H1:ASC-Y_TR_B_YAW_OUT_DQ, H1:ASC-Y_TR_A_PIT_OUT_DQ, H1:ASC-Y_TR_A_YAW_OUT_DQ, H1:ASC-CSOFT_Y_OUT_DQ, H1:ASC-DSOFT_Y_OUT_DQ
• Fscan-Observing lines stayed around 2700-3000 throughout the week

In the past, when heating up the OM2 heater, we have noticed that the low frequency sensitivity improves, for example alog 78652.

I wanted to make this comparison again, but we have seen that the squeezing performance can be variable, so I first chose to compare a no squeezing time with OM2 hot and cold.

It's also important to note that the calibration has likely changed with the hot OM2. Some of that can be corrected with the TDCFs. However, we saw that the required SRCL offset to minimize SRC detuning with OM2 hot was significantly different, alog 86737. This means that the sensing function is probably significantly different, causing some change in the calibration that is not compensated. We didn't take any calibration measurements with hot OM2, so we have no idea how much or how to compensate the calibration change.

With OM2 hot, the coherence of DARM with SRCL and MICH increased, although SRCL can explain most of the excess noise between the two times. Interestingly, the coherence of DARM with jitter decreased with OM2 hot. There is no significant change in the coherence with PRCL.

Here is a comparison of the cold and hot OM2 times, using GDS CALIB STRAIN CLEAN.

I performed a simple coherence subtraction of SRCL alone, and SRCL and MICH together. I checked that there is no significant coherence between SRCL and MICH.

Given the possible issue with the calibration, I am not certain how comparable the low frequency noise is. There is a small increase in the noise up to 30 Hz with OM2 hot, even with the above subtractions applied.

H1 ISI CPS Sensor Noise Spectra Check - Weekly Famis 26655

BSC high freq noise is elevated for these sensor(s)!!!
    
ITMY_ST1_CPSINF_H3