writting this down for future people recovering from power outages:

We weren't able to lock the IMC, the alignment was not good but not terrible.  With 2 Watts into the IMC, H1:IMC-TRANS_OUT16 was about 10000 counts, while the IMC was well aligned it was 11000 counts.  MC2 trans NSUM however was about 6 counts, well it should be 200-300 counts. 

On the whitening screen for MC2 trans (LSC> Whitening) there were errors Invalid data chn (4,3,2,1).  Elenna found this alog by searching the error message, 12120, Daniel Patrick and Fil reproduced a solution like this. 

TITLE: 09/10 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Locked for about 10 hours when a brief power outage on site took everything down. We have been in recovery since (see list of alogs below). We are currently getting the IMC to lock after realizing that there was an issue with the whitening and Daniel logged into Beckhoff to fix it.

Power outage alogs:

• Ryan C's notes during recovery 86826
• Fil's EE 86839
• Ryan S PSL recovery 86835
• Optic slider recovery 86834
• Camilla's TCS recovery
• Jim's SEI recovery
• Dave's CDS recovery alog

LOG:

• 1907 Power outage, lock loss                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

  Start Time	System	Name	Location	Lazer_Haz	Task	Time End
  15:11	CEBEX	Bubba, contractors	MY	n	CEBEX building prep	23:04
  15:51	FAC	Christina	OSB rec.	n	Rollup door to drip off packages	15:56
  16:41	FAC	Randy, Eric	MY	n	Check on something	19:59
  17:18	SYS	Betsy	Opt Lab	n	Parts	17:22
  19:19	VAC	Eric	Ends	n	Getting HVAC at ends checks	20:04
  19:20	VAC	Gerardo	LVEA	n	Reset Ion pumps for RGAs	19:43
  19:21	VAC	Jordan	Ends	n	Reset ion pumps for RGAs	19:51
  19:25	-	Fil	Ends	n	Turning on lasers and checking HV	20:25
  19:36	VAC	Travis	LVEA	n	Grab a part	19:42
  19:36	VAC	Travis	MY	n	Pump work	19:58
  19:47	PSL	Richard	LVEA	n	PSL diode controller power cycle at PSL racks	19:58
  19:48	SEI	Jim	Mech room	n	HEPI pump startup	20:04
  19:59	VAC	Travis	LVEA	n	Parts	20:16
  20:04	SEI	Jim	Ends	n	HEPI pump stations	21:24
  20:07	CDS	Erik	EX	n	Power dolphin extender	20:34
  20:35	CDS	Fil	EX	n	HEPI trouble shooting	21:10
  21:25	SYS	Betsy	LVEA	n	Taking pictures of HAM2/3	21:41
  21:38	VAC	Travis, Janos	MY	n	Pump work	22:22
  21:40	EE	Marc	LVEA	n	Checking out fast shutter electronics	21:58

TITLE: 09/10 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 6mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

IFO is DOWN due to a POWER OUTAGE

We're still recovering H1, with the IMC being a big hiccup in recovery.

1. Both corner station HV interlocks tripped. Reset. The Fast Shutter, HAM6 PZT, and ITM ESD high voltage enabled.
2. PSL and SQZ high voltage enabled
3. Safety Laser Interlock System tripped, Reset. Lasers enabled.
4. ETMX ESD HV interlock tripped. Reset, HV enabled
5. ETMY ESD HV was found powered on
6. Found 2 ETMY SEI ISI Coil Drivers with overtemp errors. Reset button on front panel cleared all faults. Need to check all ECRs have been applied to chassis.
7. Jim reported issues bringing back ETMX HEPI. Normal power OFF/ON did not bring main pump controller panel online. Asked Patrick to check the Beckhoff software, no issues found. Unit eventually powered on when the ON button was activated during troubleshooting.
8. EY Pulizzi was found unresponsive and power cycled. Erik restarted code. This enabled power to WIFI and ITMY camera

The PSL is fully recovered with all subsystems and watchdogs enabled following the site power outage earlier today. I'll add more details in a comment to this entry later.

I restored all the sliders using my restore script on IFO_ALIGN_COMPACTEST to before the glitch, 19:00 UTC.

Remotely turned on the TCS X and Y chillers.

I restarted the ETMX and ETMY HWS code, following TCS wiki. ITMs were already running. ITM lasers turned back on by themselves.

I had some difficulty connecting to the corner hepi pump controller after the power outage. The workstations have gotten too far ahead of the old Athena controller, so when I attempted to ssh to the pump station, I got:

Unable to negotiate with XX.XXX.X.XX port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

I had to add some options to ssh:

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -oHostKeyAlgorithms=+ssh-dss controls@h1hpipumpctrll0

After deleting the old host key again I was able to connect.

We had a site wide power outage around 12:11 local time. Recovery of CDS has started.

J. Kissel

Ops corps -- please run the following DARM FOM template on the top, front-and-center, wall display from 08:00 to 12:00 PDT on Saturday (9/13).
    /opt/rtcds/userapps/release/cds/h1/scripts/fom_startup/nuc30/
        H1_DARM_FOM_wO1.xml


Details:
For this upcoming Saturday's tours only, we'd like to celebrate how much the detector sensitivity has improved in the past 10 years by displaying the early O1 sensitivity (from G1501223), rather than show the L1 trace (whose live sensitivity will still be captured by the display of BNS range). 

As such, I've augmented the standard template from
    /opt/rtcds/userapps/release/isc/h1/scripts/H1_DARM_FOM.xml
with the following changes.

    Functional:
        ** Changed the pwlech FFT chunk "% overlap" parameter to be 50%, rather than 75%, which is appropriate for the Hanning window specified to be used.
        ** Changed the number of rolling exponential averages parameter to be 10 rather than 3.
        - Updated the H1 reference from April 11 2024 (representing O4B sensitivity) to yesterday, September 09 2025, so as to better represent the O4C sensitivity 
        - Imported G1501223 H1 "start of O1" representative sensitivity, replacing the L1 reference.

        ** these will have the effect of "slowing down" the live traces, and/or making "glitches contaminate the sensitivity for longer," but this comes with the benefit of a better (less noisy) estimate of the current stationary noise. That makes it a more apples-to-apples comparison with the O1 sensitivity.

    Aesthetic:
        - Added "Displacement" to the "aLIGO DARM" title
        - Synchronized the format of the legend entries for the references
        - Updated the O1 reference trace color so it can be seen a bit better
        - Re-ordered the last three traces from "PCALY, GWINC, PCALX" to "PCALX, PCALY, GWINC" (while preserving the colors and symbols)
        - Moved the legend to be centered in the window, but still showing violin modes.


The attached screenshot is of this template running live yesterday afternoon.

The hope is that this version of the central display will still fully function as a monitor of how the detector is doing for observational readiness, but also work as a good display piece for the tours.
This can reverted to the standard template as soon as the tours are done.

Satellite amplifiers for MC1 and MC3 were swapped during maintenance (15:00-19:00 UTC) on July 22, 2025 (85922), and MC2 satamp had been swapped before that (85770). I wanted to see if we could determine any improvement in noise in the LSC and ASC input mode cleaner channels.

LSC Channel: 
    - H1:IMC-L_OUT_DQ
ASC Channels:
    - H1:IMC-MC2_TRANS_PIT_OUT_DQ
    - H1:IMC-MC2_TRANS_YAW_OUT_DQ
    - H1:IMC-WFS_A_DC_PIT_OUT_DQ
    - H1:IMC-WFS_A_DC_YAW_OUT_DQ
    - H1:IMC-WFS_B_DC_PIT_OUT_DQ
    - H1:IMC-WFS_B_DC_YAW_OUT_DQ
    - H1:IMC-WFS_A_I_PIT_OUT_DQ
    - H1:IMC-WFS_A_Q_PIT_OUT_DQ
    - H1:IMC-WFS_A_I_YAW_OUT_DQ
    - H1:IMC-WFS_A_Q_YAW_OUT_DQ
    - H1:IMC-WFS_B_I_PIT_OUT_DQ
    - H1:IMC-WFS_B_Q_PIT_OUT_DQ
    - H1:IMC-WFS_B_I_YAW_OUT_DQ
    - H1:IMC-WFS_B_Q_YAW_OUT_DQ

I looked at many times before and after these swaps looking for the lowest noise for each to be the best representative of the noise level we can achieve, and have settled on a set of before and after times that differ depending on the channel.
BEFORE:
2025-06-18 09:10 UTC (DARK RED)
2025-07-07 09:37 UTC (DARK BLUE)
AFTER:
2025-08-01 07:05 UTC (GREEN)
2025-08-01 10:15 UTC (PINK)
2025-08-02 08:38 UTC (SEA GREEN)
2025-09-05 05:23 UTC (ORANGE)
These measurements were taken for 47 averages with a 0.01 BW.

Results:

Our range hasn't been the most stable during this lock. There's one drop in particular that goes down about 8Mpc to 145Mpc for 20min or so (attachment 1). Running our range comparison script, and changing the span appropriately (thanks Oli!), it looks to be a very broadband from ~40-200Hz. The SQZ BLRMS see something around that time (attachment 2).

Sheila had me run another template for that time. The template, which was copied from one of Camilla's, I've now saved in the sqz userapps - attachment 3. Green being the trace from this lock.

Lockloss due to an EQ that hit before EQ mode could activate. Seems to be either very local or very large - not on USGS yet.