Over roughly 7hrs, we had 2-consecutive ETMx Glitch locklosses taking H1 down from OBSERVING.

After that last lockloss, have had 2-consecutive LOWNOISE_ESD_ETMX locklossess while trying to get H1 back.  An alog search on LOWNOISE_ESD_ETMX lockloss issues were noted in alog82912.  Since we are having a similar situation (high microseism and frequent locklosses at this state), I made changes to LOWNOISE_ESD_ETMX in ISC_LOCK.py by increasing the (1) ETMx ramp time & (2) a timer after this step (see attached):

• etmx_ramp = 60  (this was 20)
•  self.timer['ETMswap'] = 60 (this was 30)

Change Save & Loaded for ISC_LOCK and we are currently on the way to see if we can get through LOWNOISE_ESD_ETMX (or have a 3rd Consecutive lockloss here).

TITLE: 03/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 10mph Gusts, 6mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.45 μm/s
QUICK SUMMARY:

Happy Spring Forward Morning (& 1-hr less sleep)!

H1 just went down (after a 3h51m lock, and a 1435utc lockloss w/ an ETMx Glitch tag).  Taking H1 immediately into Initial Alignment to skip Locking ALS locklosses. 

The one Owl 932utc lockloss (also an ETMx Glitch tag) was fully automatic (after about 22min of 7 Locking ALS attempts) & no Alignment was needed.

Looks like secondary microseism hit the 95th percentile about 10hrs ago and has stayed there.  Winds are under 10mph.

TITLE: 03/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 01:36 UTC (~4hr 30 min lock).

Extremeley calm IFO that stayed locked for majority of shift.

Lock acquisition after the EQ lockloss (alog bla bla) went fully automatically.

Since then, we've been locked with the microseism steadily increasing. However, the wind has died down.

PSL Dust has been fluctuating a lot (swapping between red, yellow and no alerts) this shift, despite wind calming down.

LOG:

None

TITLE: 03/08 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:

Two locklosses today which were straightforward to recover from.  One happened at the waning minutes of the calibration suite of measurements which is sad (EQ inbound, higher winds, and microseism increasing); this was tagged as a WIND + Calibration lockloss.

Also had 5-tours for our 2nd Saturday Tour day.
LOG:

• H1 was recovering from an ETMx glitch lockloss at beginning of shift.
• 1638 Back to Observing
• 1700-1715 Picked up around the Control Room before Saturday Tour visitors arrive (mainly putting lots of used scratch paper in the "data archive" (aka recycling bin), one coffee cup in dishwasher, one battery in battery recycling container, and organized chairs.
• 1815-1832 930am Saturday tour (cassidy)
  • 1955-2010 1030am Saturday tour (janos)
  • 2040- 1130am Saturday tour (mike)
  • 2125-2156 1230pm Saturday tour (amber)
  • 2245-2314 1:30 Saturday Tour (fred)
• 1911 Incoming EQ alert for M5.7 from southern japan
• 1930 Out of Observing for Saturday Calibration
  • Lockloss (presumably due to incoming EQ) during the last few minutes of the simulines measurement at 1958utc  :(
  • Had WINDY + Calibration tags for this lockloss.
• 1952 Noticed nuc25 PI Mode ndscope was not running for last 4-days---restarted it.
• 1958 LOCKLOSS (right before the Saturday Calibration measurement)
  • 2004-2024 Initial Alignment
  • 2122 OBSERVING....could have been a few minutes sooner, but for some reason a GRD_IFO in "AUTOMATIC OPERATION" failed to take us in automatically which is odd.
• 0002 EQ alert for another one south of Japan
• 0017 Lockloss....guessing it is EQ related since we had an alert for one, but we also have high microseism and winds
  • Proactively went for an Initial Alignment and now handing off to Ibrahim!
  • Late NOTE:  This lockloss has a WINDY tag with winds passing the 20mph threshhold, but we also had the earth moving with that incoming EQ from South Japan and rising usiesm...so, still think it's a toss-up.  Both locklosses with the South Japan EQ never had the picket fence go yellow, did not see any movement on ndscopes and really didn't look bad on seismometer bands, so chalking it up to a comedy of seismic noise teaming up to be an H1 foe.

Lockloss due to EQ, exacerbated by a 5.9 EQ (which interestingly didn't trigger EQ mode).

IFO is relocking now.

TITLE: 03/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 26mph Gusts, 21mph 3min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.41 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 21:22 UTC (3 hr lock).

Unfortunately, we have a 5.9 EQ arriving in a few minutes. If microseism was lower, we could probably ride it out but seeing as we lost lock due to a lower mag EQ with lower microseism just 4ish hrs ago, I would say chances are low.

Calibration sweep did not run today due to an EQ so I can run one pending thermalization (and green light from comissioners/run coord).

Sat Mar 08 10:08:10 2025 INFO: Fill completed in 8min 7secs

FAMIS Link:  26033

Only CPS channels which look higher at high frequencies (see attached) would be the following:

1. ITMx St1 V1
2. ETMy St2 V2

TITLE: 03/08 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 3mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.31 μm/s
QUICK SUMMARY:

H1 had a lockloss about 70min ago due to an ETMx Glitch, due to trouble locking ALS, automation finished up an alignment 10min ago and continues locking (H1 had a pretty quick automated acquisitioon overnight in about 90min for comparison).  Secondary microseism has a slow increase over the last 12-ish hours and it has been a little breezy the last 24hrs (w/ gusts going over 20mph at various points). Today is Saturday Tour day & Calibration is scheduled for 1830utc/1130amPST.

TITLE: 03/08 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
INCOMING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 9mph Gusts, 6mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.26 μm/s
SHIFT SUMMARY:
Had a lockloss just before my shift.
Did some SQZr changes while  relocking : Corey's alog: 83225 , My alog about this: 83236
Once Relocked H1 has stayed locked and Observing for 3 hours.

LOG:                                                                                                                                                                                                                                                                                             

TITLE: 03/08 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 11mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY:

00:14 UTC H1 Aquired a Lockloss  of unknown cause.
There was a request to load the SQZ_SHG Guardian, but there was a syntax in the code that caused the Guardian node to go into error.  Simple fix took me a few minute to find and fix.
SQZ_SHG Guardian node is running smoothly now.

Corey then took H1 into Initial_Alignment, while I edited the  sqzparams.py.
Line 18  was take from 75 to 80:
opo_grTrans_setpoint_uW = 80  #reload OPO guardian for this to take effect
I then took SQZ_OPO_LR  Guardian to LOCKED CLF_DUAL_NO_ISS,  Then reloaded the Guardian code.
Once reloaded I took SQZ_OPO_LR back to LOCKED_CLF_DUAL. 

Then tried to adjust the OPO_TEC_TEMP, this was a mistake. I could tell that something wasnt correct because I couldn't get the SQZ-CLF_RFEL_RF6_ABS_OUTPUT back to the same hieght it was before while adjusting the OPO_TEC_TEMP. Contacted Sheila.
She informed me that the SQZ_CLF_LR should be locked when doing this.

Once locked the SQZ-CLF_RFEL_RF6_ABS_OUTPUT went up and Looked much better.

Relocking notes.
Initial_Alignment is finished and I'm trying to relock and I have not had success past DRMI. But the night is stil young.

Update:
NOMINAL_LOW_NOISE reached at 2:50 UTC
Observing reached at 3:02 UTC

TITLE: 03/07 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

edit:  h1 had a lockloss minutes before the end of the shift, so Tony and I started addressing SQZ To-Dos (see below in the log)

Ops To Do At Next Opportunity (next H1 lockloss, or L1 goes down.  See sheila alog 83227):

1. Load SQZ_SHG (this can be done with H1 in any state)
2. Restore opo_grTrans_setpoint_uW back to 80 (was set to 75 this morning), and re-tune the OPO TEC setpoint .  This can be done with H1 in any state.

LOG:

• At the beginning of the shift, H1 was at NLN, but not Observing due to issues with the SQZ_SHG (alog83224)
• 1758 Dave TeamSpeak-ed in saying we have a red FMCS Stat for the EX VEA temperature---he mentioned this is normal when we get warmer days and the sun on the building.  No changes were made, so will leave this FMCS Stat red as a heads up for a temperature increase in the EX VEA. 
• 1831-2035 ISS Array work in optics lab (myank, siva)
  • 2209 hardware work (myank)
  • 2319 siva joining
• 1855-1914 Optics Lab clean (kim)
• 1921-2005 y-arm walk (ibrahim)
• 2155-2213 Packing sensor in PCal lab (tony)
  • 2222-2226 changing shutter status (tony)
• 0014 Lockloss toward end of shift.  Took this opportunity to do the (2) SQZ items listed above.
• (1)  SQZ_SHG node was LOADed, but came up with this error:
  • 2025-03-08_00:15:52.522971Z File "/opt/rtcds/userapps/release/sqz/h1/guardian/SQZ_SHG.py", line 39
  • 2025-03-08_00:15:52.522971Z elif ezca['SQZ-SHG_TRANS_RF24_PHASE_ERROR_FLAG'] != 0:
  • 2025-03-08_00:15:52.522971Z ^ 2025-03-08_00:15:52.522971Z SyntaxError: invalid syntax
  • In SQZ_SHG.py, here is the fix Tony made to fix the error above:

def in_hardfault():
    #if ezca['SQZ-LASER_IR_DC_ERROR_FLAG'] != 0:
    #    notify('Squeezer laser PD error')
    #    return True
    #elif ezca['SQZ-PMC_TRANS_DC_ERROR_FLAG'] != 0:
    #    notify('Squeezer PMC not locked')
    #    return True
    if ezca['SQZ-SHG_TRANS_RF24_PHASE_ERROR_FLAG'] != 0:
        notify('SHG Trans phase shifter error')
        return True
    #elif ezca['SQZ-SHG_TRANS_RF24_DEMOD_ERROR_FLAG'] != 0:
    #    notify('SHG Trans Demod error')
    # see comment to alog 83224 for why this was commented out
    #    return True

• (2)  Restored opo_grTrans_setpoint_uW back to 80 (from 75).
  • Then Tony tried adjusting OPO TEC TEMP to maximize CLF_REFL_RF6_ABS_OUTPUT, but it was not totally clear it was working.  Granted H1 is not at NLN (does H1 need to be at NLN to do this?)
  • At any rate, Tony got a hold of Sheila and they are chatting on the phone now.

J. Kissel

I've been looking through the data captured about the PMC in the context of the two years of observatory use between July 2022 and July 2024 where we spanned a few "construction, commission, observe" cycles -- see LHO:83020. Remember the end goal is to answer the following question as quantitatively as possible: "does the PMC have a high enough duty cycle in the construction and commissioning phases that the SPI does *not* need to buy an independent laser?"

Conclusions: 
 - Since the marked change in duty cycle after the PMC lock-loss event on 2023-05-16, the duty-cycle of the PMC has been exceptionally high, either 91% during install/commissioning times or 99% during observing times. 
 - Most of the down time is from infrequent planned maintenance. 
 - Recovery time is *very* quick, unless the site loses power or hardware fails. 
 - The PMC does NOT lose lock when the IFO loses lock. 
 - The PMC does NOT lose lock just because we're vented and/or the IMC is unlocked. 
 - To-date, there are no plans to make any major changes to the PSL during the first one or two O4 to O5 breaks.
So, we shouldn't expect to lose the SPI seed light frequently, or even really at all, during the SPI install or during commissioning. And especially not during observing. 

This argues that we should NOT need an independent laser from an "will there even be light?" "won't IFO construction / commissioning mean that we'll be losing light all the time?"  duty-cycle stand point.
Only the pathfinder itself, when fully functional with the IFO, will tell us whether we need the independent laser from a "consistent noise performance" stand-point.

Data and Historical Review
To refresh your memory, the major milestones that happened between 2022 and 2024 (derived from a two year look through all aLOGs with the H1 PSL task):
- By Mar 2022, the PSL team had completed the complete table revamp to install the 2x NeoLase high-power amps, and addressed all the down-stream adaptations.

- 2022-07-01 (Fri): The data set study starts.
- 2022-09-06 (Tue): IO/ISC EOM mount updated, LHO:64882
- 2022-11-08 (Tue): Full IFO Commissioning Resumes after Sep 2022 to Dec 2022 vent to make FDS Filter Cavity Functional (see E2000005, "A+ FC By Week" tab)
- 2023-03-02 (Thu): NPRO fails, LHO:67721
- 2023-03-06 (Mon): NPRO and PSL function recovered LHO:67790
- 2023-04-11 (Tue): PSL Beckhoff Updates LHO:68586
- 2023-05-02 (Tue): ISS AOM realignment LHO:69259
- 2023-05-04 (Thu): ISS Second Loop Guardian fix LHO:69334
- 2023-05-09 (Tue): "Something weird happened to the PMC, then it fixed itself" LHO:69447
- 2023-05-16 (Tue): Marked change in PSL PMC duty-cycle, nothing specific PSL team did with the PMC, but the DC power supplies for the RF & ISC racks we replaced, 69631, while Jason tuned up the FSS path LHO:69637 
- 2023-05-24 : O4, and what we'll later call O4A, starts, we run with 75W requested power from the PSL.
- 2023-06-02 (Fri): PSL ISS AA chassis it was replaced, but PMC stays locked through it LHO:70089
- 2023-06-12 (Sun): PMC PDH Locking PD needs threshold adjustment, LHO:70352, for "never found out why" reason FRS:28260
- 2023-06-19 (Mon): PMC PDH Locking PD needs another threshold adjustment, LHO:70586, added to FRS:28260, but again reasons never found.
- 2023-06-21 (Wed): Decision made to reduce requested power into the IFO to 60W LHO:70648
- 2023-07-12 (Wed): Laser Interlock System maintenance kills PSL LHO:71273
- 2023-07-18 (Tue): Routine PMC / FSS tuneup, with quick PMC recovery LHO:71474
- 2023-08-06 (Sun): Site-wide power glitch takes down PSL LHO:72000
- 2023-09-22 (Fri): Site-wide power gltich takes down PSL LHO:73045
- 2023-10-17 (Tue): Routine PMC / FSS tuneup, with quick PMC recovery LHO:73513
- 2023-10-31 (Tue): Jeff does a mode scan sweeping the PMC FSR LHO:73905
- 2023-11-21 (Tue): Routine PMC / FSS tuneup, with quick PMC recovery LHO:74346
- 2024-01-16 : O4A stops, 3 months, focused on HAM567, no PSL work (see E2000005, "Mid-O4 Break 1" tab)
O4A to O4B break lock losses: 7
       2024-01-17 (Wed): Mid-vent, no IFO, no reported cause.
       2024-01-20 (Sat): Mid-vent, no IFO, no reported cause.
       2024-02-02 (Fri): Mid-vent, no IFO, no reported cause.
       2024-02-08 (Thu): Mid-vent, no IFO, no reported cause. During HAM6 close out, may be related to alarm system
       2024-02-27 (Tue): PSL FSS and PMC On-table Alignment LHO:76002.
       2024-02-29 (Thu): PSL Rotation Stage Calibration LHO:76046.
       2024-04-02 (Tue): PSL Beckhoff Upgrade LHO:76879.
- 2024-04-10 : O4 resumes as O4B start
O4B to 2024-07-01 lock losses: 1
       2024-05-28 (Tue): PSL PMC REFL tune-up LHO:78093.
- 2024-07-01 (Mon): The data set study ends.

- 2024-07-02 (Tue): The PMC was swapped just *after* this data set, LHO:78813, LHO:78814

By the numbers

Duty Cycle (uptime in days / total time in days)
     start_to_O4Astart: 0.8053
    O4Astart_to_O4Aend: 0.9450
    O4Aend_to_O4Bstart: 0.9181
       O4Bstart_to_end: 0.9954
(Uptime in days is the sum on the values of H1:PSL-PMC_RELOCK_DAY just before lock losses [boxed in red] in the attached trend for the given time period)

Lock Losses (number of times "days" goes to zero)
     start_to_O4Astart: 80
    O4Astart_to_O4Aend: 22
    O4Aend_to_O4Bstart: 7
       O4Bstart_to_end: 1
(Number of lock losses is mere the count of red boxes for the given time period)

Lock Losses per calendar days
     start_to_O4Astart: 0.2442
    O4Astart_to_O4Aend: 0.0928
    O4Aend_to_O4Bstart: 0.0824
       O4Bstart_to_end: 0.0123
(In an effort to normalize the locklosses over the duration of the time period to give a more fair assessment.)

I also attach a histogram of lock durations for each duration, as another way to look at how the duty cycle dramatically changed around the start of O4A.

Closes FAMIS 28458. Last checked in alog 82659.

• CO2 IY and IX PD_OUT signals are constant rather than offset at week intervals compared to last month. I assume this is due to changes between Tuesdays.
• HWS SLEDPOWERMON signals for both IX and IY are continuing to go down very gradually as last month's trend showed.

This is for FAMIS #26370.
Laser Status:
    NPRO output power is 1.853W
    AMP1 output power is 70.51W
    AMP2 output power is 140.0W
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 30 days, 23 hr 28 minutes
    Reflected power = 22.36W
    Transmitted power = 106.5W
    PowerSum = 128.8W

FSS:
    It has been locked for 0 days 5 hr and 18 min
    TPD[V] = 0.7969V

ISS:
    The diffracted power is around 3.5%
    Last saturation event was 0 days 5 hours and 18 minutes ago

Possible Issues: None reported

I assembled the 45MHz WFS unit in the optics lab. Assembly drawing: D1102002.

BOM:

• 1x Pre-assembled WFS body (D1102004 s/n016 Main Body, sealed & scribed as "S1300637" according to Corey's email)
• 1x Alumina spacer (D1102007)
• 1x Baseplate (D1102003)
• 1x Q3000 QPD
• 1x QPD hood (D1102006)
• 2x SHCS 1/4-20 X .75" 18-8 VENTED for the baseplate
• 2x 1/4" PEEK washer (D1102010) for the baseplate
• 4x SHCS 4-40 X 3/16" 18-8 Vented for the QPD hood (spec is BSHCS but SHCS works fine)

I confirmed that the baseplate and the WFS  body are electrically isolated from each other.

There were many black spots on the WFS body (2nd pic) as well as the aluminum foil used for wrapping (3rd pic). It seems that this is a result of rubbing of aluminum against aluminum. I cannot wipe it off but this should be aluminum powder and not some organic material.

QPD orientation is such that the tab on the can is at 1:30 o'clock position seen from the  front (4th pic). You cannot tell it from the picture but there's a hole punched to the side of the can.

Clean SMP - dirty SMA cables are in a bag inside the other clean room in the optics lab. DB25 interface cable is being made (or was made?) by Fil.

I found evidence of possible scattered light while looking at some data from a lock yesterday. Attached is a whitened spectrogram of 30 minutes of data starting at GPS 1425273749. It looks like the peaks are around 28, 38, and 48 Hz, but they are broad and it's hard to tell the exact frequency and spacing. Sheila thinks this may have appeared after Tuesday maintenance. Tagging detchar request so some tests can be run to help us track down the source!