Attached plot show trends of OSEM movements and subsequent watchdog activity.

Summary: we had three SWWD SEI trips (ETMY, ITMY, ITMX) followed by two SWWD SUS trips (ETMY, ITMY). We had one HWWD trip, ITMY.

Color code ETMY=BLUE, ITMY=RED, ITMX=GREEN, ETMX=GOLD.

Top panel: ETMY top stage 6 OSEMS

Second panel: ITMY top stage 6 OSEMS

Third panel: ITMX top stage 6 OSEMS

Fourth panel: ETMX top stage 6 OSEMS

Fifth panel: SWWD

Bottom panel: HWWD

Time Line (all times PDT Monday 8th September 2025)

(1) [21:11:44]: all four SWWD SEI timers start countdown

(2) [21:15:27]: ETMX goes good again (not shown, but ditto for ETMX HWWD)

(3) [21:16:42]: SWWD SEI trip, SEI IOP countdowns start, they trip in five minutes time

(4) [21:26:41]: ETMY SWWD SUS trip, its second timer is 10 minutes

(5) [21:31:54]: ITMY SWWD SUS trip, it has the original 15 minute timer, closely followed by:

(6) [21:32:29]: ITMY HWWD trip after its 20 minute countdown

(7) [21:38:14]: ITMY HWWD input goes good, ready for its reset button to be physically pushed

(8) [21:38:00]: ITMX SWWD SUS does not trip, it occasionally dips into the good region resetting the timer

(9) [21:42:00]: All SWWDs are reset, their inputs a good after the EQ has cleared

J. Kissel

Continuing my low-level, background, fact building exercises on satellite amplifiers (the only previous installment thus far being LHO:85348) here I compare the three types of satellite amplifiers
    (a) D0900900 / D0901284 :: UK 4CH SatAmp 
    (b) D1002818 / D080276 :: US 8CH SatAmp 
    (c) D1900089 / D1900217 :: US 4CH SatAmp
again, but instead now comparing their transimpedance sign and frequency response. 

Namely, if the systems design intent was
    - the US 8CH SatAmp has been wired up to the OSEM in a POSITIVE reverse bias configuration, with ANODE (A) connected to the NEGATIVE terminal of the transimpedance opamp, and 
    - the US and UK 4CH SatAmps have been wired up to the OSEM in a NEGATIVE reverse bias configuration, with CATHODE (K) connected to the NEGATIVE terminal of the transimpedance opamp,
then these better have differently signed overall transimpendance in order to achieve the same sign at the ADC.

As such, I performed a similar frequency response measurement on an example instantiation of a D1002818 and D1900089 chasses as I've been doing for the UK 4 channel satamps -- see T080062 for the full procedure, and the "only" difference is the physical connection of the SR785 to the device under test. To describe that, I attach here a collection of diagrams of the physical setup for each chassis.

The results are clear: driving a POSITIVE voltage through a series resistor into the NEGATIVE input of the transimpedance op amp (i.e. mimicking the flow of PD current [conventional current, not electron flow] into the NEGATIVE input) results in a
    (a) NEGATIVE differential output voltage for a D0900900 UK 4CH chassis,
    (b) POSITIVE differential output voltage for a D1002818 US 8CH chassis,
    (c) NEGATIVE differential output voltage for a D1900089 US 4CH chassis.

Note, the NEGATIVE input to each chassis transimpedance amp is connected to across pages of the circuit diagram to connector pins labeled as 
    (a) K (CATHODE) for a D0900900 UK 4CH chassis,
    (b) A (ANODE) for a D1002818 US 8CH chassis,
    (c) K (CATHODE) for a D1900089 US 4CH chassis.
implying the correct outer-layer usage.

The second attachment shows the frequency response of the three chassis. Note all of the tested instantiations are pre-ECR E2400330 whitening filter change, so they all have the old z:p = ~0.4:10 Hz frequency response. The legend in the caption of the upper right panel shows the predicted z:p values from the schematic, and the data in the upper and lower right panels show that particular z:p divided out of the data to show how close these instantiations are to the ideal. Yes, the 8CH satamp frequency response is slightly different than the two 4CH responses; this will be rectified with ECR E2400330. The deviation of all three responses from ideal is consistent with the diversity of the instantiations of the UK 4CH sat amps, namely the uncertainty in whitening filter stage capacitance value (which should be a total of 20 [uF], but can vary up to 5%; see LHO:85396).

As such, when using these chassis, it is critical that the chassis-to-flange and in-vacuum wiring is connected in such a way that connects this NEGATIVE input to the right terminal of the OSEM PD.

For detailed breakdown of each stage of the sat amp frequency response and sign, to understand *why* and *how* they're different, check out G2500980.

LVEA Swept and ready for Observing

I was getting lots of overflows when I started the measurement, I tried lower the drive_amp by 20% and retrying. This didn't work, I went from the usual 11000 down to under 100 and still saw lots of overflows. Rahul suggested to halve the gain of the ESD output filters with the original 11000 drive_amp, from 1.0 to 0.5 and this stopped all the overflows during the measurements. I got a few thousand overflows when the script ended and was restoring settings though.

This measurement like previous recent ones (since the ESD bias voltage increase?) has very large errors, there wasn't any work going on at EX and the wind wasn't too bad, gusting up to 20mph.

Despite the huge error bars the charge generally seems lower than the last measument in most quadrants/dofs.

FAMIS Link:  26656

Only CPS channels which look higher at high frequencies (see attached) would be the following:

1. HAM3 H2
2. ITMx St1 V1
3. ITMy St1 H3 (noted below)
4. ITMy St1 V3

In the bash window we get this note:

"BSC high freq noise is elevated for these sensor(s)!!!:        ITMY_ST1_CPSINF_H3 "

Tue Sep 09 10:06:51 2025 INFO: Fill completed in 6min 47secs

FAMIS 27778

I closely inspected the lines starting from the chillers and going to the tables. I saw no evidence of leaks, cracks or stressed pipes along their entire lengths. All notes from the last inspection in March 2025 [alog83162] still stand, but no major issues.

FAMIS 28949

Reboot at 1600UTC(0900PT). Every node came back up without issue, Ryan C helped me bring back nodes to their nominal requests for maintenance.

TITLE: 09/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY: Locked for 4 hours, pem and charge measurements running. Maintenance day will begin at 0800 PT. Rough recovery last night from the earthquake and eletrical storm, but Ibrahim got us back up.

Got the wake up call at 338amPDT.  And literally while I was waking up and trying to remember what to do for the NOTIFY + MANAGER guardian nodes (I just "init-ed" both...is that the right thing?  that's what both of their User Messages said to do), I noticed that there was a Message that said there was an issue with the SQZ MANAGER.  

As I was getting ready to open up the SQZ guardians and assess (I'm pretty sure I spelled that right--I remember Hugh giving me grief for misspelling it in an alog before!), H1 was already back to OBSERVING.  I literally did nothing!  H1 was in OMC Whitening at 932utc, sent wake up call at 1037utc, and automatically went to Observing on its own at1043utc.  I'm going to go back to bed ASAP!  
 

Great job getting H1 thru all the environmental catastrophes at the end of your shift, Ibrahim! :)

TITLE: 09/09 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

IFO is in OMC_WHITENING and LOCKING

The first 5 hours of this extended shift were quiet and well behaved. Truly the calm before the earthquake before the storm. Lockloss alog 86792.

Thanfully the next 3 hours were action packed due to 2 environmental events: a 5.8 EQ (and 4.9 aftershock) off the coast of Oregon and an electrical storm power glitch.

The Earthquake (alog 86793)

5.8 EQ from Oregon downed the IFO before SEI or SUS had a chance to prep.

• A very violent lockloss which tripped: ISI EY ST1 WD, ISI EY ST2 WD, SUS TMSY WD, ISI IX ST1 WD, ISI IX ST2 WD, ISI IY ST1 WD, ISI IY ST2 WD, ISI EX ST2 WD, SUS IX WD, SUS IY WD, SUS EX WD, SUS EY SW WD, SUS IY SW WD, SUS IX SW WD, SEI EY SW WD, SEI IY SW WD, SEI IX SW WD, HEPI EY WD (twice), HEPI IY WD (twice), HEPI IX WD (twice), ISI IY ST2 WD, IY Hardware WD (with warnings from all the others) (pic attached)
• EX overperformed its sibling QUADs by being the only one that didn't completely trip somehow
• Jim and Dave came online to the rescue and we immediately began diagnosing and eventually, untripping.
• IY Hardware WD had tripped so I went into the CER to untrip it successfully
• While I was in the CER, the lights flickered, I heard a loud rumble and when I was back some more greens were red - that was the high voltage.

The Storm (alog 86795)

The storm caused a slight power glitch that caused the high voltage for both HAM6 and the PSL to turn off.

• The PMC was flickering on the camera and I called Ryan S who pointed me to the right place to realize the high voltage had tripped. Jim simultaneously realized this on DIAG_MAIN.
• I called Richard (and TJ but Richard called back and he walked me through how to turn the high voltage back on (pic attached)
• I once again ventured into the CER, this time to the mezzanine and turned on the high voltage
• When I arrived, Jim and Dave had finished untripping the WDs and with that, all the reds were green and that's universal for go - so I started relocking.

The Aftershock:

While relocking in INITIAL_ALIGNMENT, we got an aftershock (4.9 EQ) from the same location, wich prompted EQ mode to turn on.

• I had to INIT some nodes since some of the QUADs were in auto and not managed, which marked the end of my OPS intervention. Just kidding! See below.
• Like most sequels these days, it was underwhelming. It did not even affect initial alignment, which ran fully auto.
• Funnily though, we went into MICROSEISM since the secondary microseism was high enough after EQ mode went to CALM

Initial alignment finished, I began locking, DRMI caught immediately and thanks to the awesome power of nature, this shift was not boring. Except of course, the OMC is not locking (despite turning the high voltage back on) - see below for the 3rd installment nobody asked for.

The OMC:

The OMC high voltage also tripped though I did turn it on (narrator: or so he thought)

• The problem it seems is that the sweep it was doing in looking for lock was going between -40 and 40 instead of -50 and 50, and it's jumping instead of sweeping.
• After ~10 dead ends, and one accidental lockloss trying to fix this, I think I found the solution.
• My guess is that the issue isn't the high voltage but that guardian is not doing a fine sweep to find the right PZT2 steps and is jumping instead. From alog 85876, I think we should auto the OMC guardian and step through the high voltage in 1.0 steps to get the OMC locked.
• Last time the High Voltage tripped, this happened too. Thus, I anticipate we will be stuck at PREP_FOR_DC_READOUT. From context clues, I think this has to be done at DARM_OFFSET properly instead of at PREP_DC_READOUT but frankly, I have no idea.
• Alright guess what? It was the high-voltage AGAIN. The attached picture is from hours ago when I turned it on the first time. It crapped out again, I turned it back on - it's working now.
• We're past PREP_FOR_DC_READOUT. The OMC is still in AUTO so now figuring out how to put it back in MANAGED. Honestly, we should check this power supply out on Tuesday, which is in a few hrs.
• Re-managed OMC_LOCK by Temp-manual -> INIT -> whatever-state-I-manualled-at.

Other:

The DARM FOM is having trouble connecting to NDS so I can't see the Violin Modes but they're extremely high (maybe due to the possbily fake measured 573 Lin Velocity of that Oregon EQ). Violin medm attached.

Anyway, I should log off now before I summon a tornado or something. Honestly, this has been a fantastic learning oppurtunity (and don't worry I have the day off tomorrow).

LOG:

After the watchdogs were reset a lightening strike caused a power glitch. GC UPS reported going into and off of battery power.

h1susauxh2 and h1susauxex went offline, turned out they were rebooting themselves and came back after a few minutes.

PSL is in a bad way, Ibrahim is calling for support.

The lights in my house flickered at the time I saw the sus-aux machines go down. We have been having a storm roll over us for the past hour, moving from Oregon northwards.

SWWDs for ETMY, ITMY and ITMX have tripped.

HWWD for ITMY has just tripped, its RMS is significantly higher than the others.

Strangely ETMX SWWD and HWWD have not come close to tripping.

Lockloss due to an EQ that hit before EQ mode could activate. Seems to be either very local or very large - not on USGS yet.

This afternoon a 6.4 mag eq in Fiji provided another good opportunity to test the high asc gain eq mode. This time peak velocities were about 3.6 micron/s on peakmon, around 2 micron/s on the .03-.1hz blrms on the wall fom. We were out of observe for ~1hr, but we probably would have been down for at least 2hrs without the transition. Definitely one of the largest eqs we've ridden out, ever.

On Sept 4th we had a longer commissioning period to allow us to heat up OM2.  The main goal was to use this to characterize the mode matching of the arm cavities to the OMC, but we also made some other interesting measurements.

Summary:  heating up OM2 now costs us about 1% of optical gain, while it used to cost us 2%.  Heating up OM2 also changed the SRCL offset needed to get flat squeezing significantly, and reduced the amount of squeezing that we could get (without adjusting psams).

The interferometer unlocked (86727) while Jennie Wright was running the DARM offset step script with OM2 cold, to get a measurement of HAM6 throughput and look at OMC refl before heating up OM2.  As that link says, there wasn't an obvious connection between the DARM offset script which was in it's final steps (nearly back to normal) when the lockloss happened. 

We turned on the OM2 heater while relocking after some back and forth, then had an commissioning caused lockloss while trying to recover.  By the time we were relocked, OM2 was heated up according to thermistor 2, which is the thermistor who's timescale matched the timescale of optical gain changes in the past (see screenshot of June 2023 example).  We also had a large earthquake while relocking, so we paused after power up and before going to nominal low noise, so we do not have tracking of the optical gain using pcal in the first 30 minutes of this thermalization. 

The optical gain in this thermalization seemed to be fairly similar to the previous lock where OM2 was cold in the first hour.  The change in optical gain between OM2 hot and cold was much smaller this time around, so we needed to see the full thermalization in order to see what the gain change was.  We set the OM2 heater off 4 hours and 5 minutes after the power up, the attached screenshot shows a trend of optical gain during the previous thermalization while OM2 was cold, with a vertical cursor 4 hours 5 minutes into the lock.  The horizontal cursors show where the optical gain was at 4 hours and once the thermalization was complete, the optical gain continued to increase by 0.2% after the first 4 hours. The next screenshot shows vertical cursors also at the time of power up and 4 hours later, and the horizontal cursors are the same as on the previous screenshot (OM2 cold).   It seems that the optical gain was about 1.1% lower with OM2 hot than cold, although fitting these two thermalizations to an exponential might bring them closer by as much as 0.2%. Our current ring heater settings are 0W on ITMY, 1.5 W/segment on ETMY and 0.44W/segment on ITMX and 1W/segment on ETMX.

Since this is a different result than in the past times of OM2 changes, I've gone back to look at old times when we did this change.  One possible explanation for the difference could be ring heater settings being different. 

• 12/18/2023, cooled off while unlocked, optical gain increased 1.8% ETMX +ETMY RH 1W, ITMX 0.44W
• 9/19/2023, heated while unlocked, optical gain dropped 1.9%   (this time is confusing, it seems like the thermistors which had been disconnected were plugged in and looked to be thermalizing, but the set power was unchanged).
• 7/19/2023, heated in steps, 71484 ifo unlocked and relocked several times, perhaps calibration was reset making optical gain comparison confusing. (it did seem to drop by about 2%, but this seems more like a calibration reset) ETMX + ETMY RH 1W/segment, ITMX RH 0.44W/segment
• 7/12/2023 OM2 cooled off in lock, lockloss happened in the early part of the cooling (2% optical gain increase), ETMX + ETMY RH 1W/segment, ITMX RH 0.44W/segment
• 6/27/2023 OM2 heated while in lock, optical gain dropped by 2%, ETMX + ETMY RH 1W, ITMX RH 0.44W/segment, 0 on ITMY.

Once the OM2 heater was turned back on, the optical gain increased by nearly 1%, but the IFO lost lock before that thermalization finished.  There was a 2% decrease in POP18 during the cool off and a 3 urad shift in SRM top mass. 

While OM2 was hot, we did a few tests.

• We had a large change in the SRCL offset needed to reduce the squeezing rotation: 86739  This is interesting, in the past we have tried to change SRM alignment to see an impact on this number and it was a much smaller change.
• No squeezing time for a senstitivty comparison (not much change): 86788
• quick squeezer data set (before SRCL offset updated: 86736 with SRCL offset updated: 86739 ) , to see a comparison with OM2 cold see Camilla's plot from this morning
• Jennie Wright did DARM offset steps with OM2 hot, and cold.