6.6 Mag Iceland EQ Caused Lockloss that happened 4 minutes after we reached NLN again. Ground motion is too high to attempt locking so staying in IDLE until it passes.

Lockloss from what looks to be seismic motion caused by a 4.9 EQ combo'd with high microseism. The timing doesn't exactly math up though so I'll investigate further.

TITLE: 03/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:

Today (and last night) was ETMx-related with (2) ETMx locklosses (one overnight & one at very start of shift).  Then had almost 5hrs of downtime with 3 locklosses in a row at LOWNOISE_ESD_ETMX (resulting in making some Calls for help, but H1 came back on its own...after pausing for about an hr before this troublesome state---see alogs).  Keita also suggests some measurements we should run next time we are locking.  Secondary microseism has mostly been flat (but maybe if you squint at it right, you can see it possibly drifting down).  Winds have  been around 20mph for much of the last 6hrs.

H1 is finally thermalized, so a calibration could be run (but microseism is high still, so I'd be wary...esp with the grief of getting through LOWNOISE ESD ETMx this morning).
LOG:

• 1435 ETMx Glitch LOCKLOSS
  • 1438-1502  Initial Alignment
  • 1541 lockloss @LOWNOISE_ESD_etmX
    • 1548-1554 DRMI bad, so ran CHECK MICH FRINGES + PRMI
  • 1629 lockloss @LOWNOISE_ESD_etmX----> (perhaps look at alog82912)
  • Increased ramptime & timer for LOWNOISE_ESD_ETMX for 3rd attempt thru this state.
  • 1735 lockloss @LOWNOISE_ESD_etmX  (time changes above did not work, so backed them out).  Started Call List protocol.
• 651-1621 SEI_ENV @USEISM (now at CALM)
  • 1721 back to USEISM
  • 2255 back to CALM
• 1924 Back to OBSERVING (w/ nothing new done...other than pause at MAX POWER for about 52min)
• 2003 alert for an EQ from Mexico (I think Seismon meant to say this was a M4.1 from Malibu, CA at 2003utc)---either way this did not affect H1 or show on our seismometers.

TITLE: 03/09 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 22mph Gusts, 18mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.39 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 19:24 UTC (3.5 hr lock)

IFO is stable but was having issues regarding LOWNOISE_ESD_ETMX state, assumed due to high microseism, which is coming down. There were also 2 locklosses. Both the locklosses and locking issues are detailed in Corey's alog 83249.

Sun Mar 09 10:08:50 2025 INFO: Fill completed in 8min 47secs

Over roughly 7hrs, we had 2-consecutive ETMx Glitch locklosses taking H1 down from OBSERVING.

After that last lockloss, have had 2-consecutive LOWNOISE_ESD_ETMX locklossess while trying to get H1 back.  An alog search on LOWNOISE_ESD_ETMX lockloss issues were noted in alog82912.  Since we are having a similar situation (high microseism and frequent locklosses at this state), I made changes to LOWNOISE_ESD_ETMX in ISC_LOCK.py by increasing the (1) ETMx ramp time & (2) a timer after this step (see attached):

• etmx_ramp = 60  (this was 20)
•  self.timer['ETMswap'] = 60 (this was 30)

Change Save & Loaded for ISC_LOCK and we are currently on the way to see if we can get through LOWNOISE_ESD_ETMX (or have a 3rd Consecutive lockloss here).

TITLE: 03/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 10mph Gusts, 6mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.45 μm/s
QUICK SUMMARY:

Happy Spring Forward Morning (& 1-hr less sleep)!

H1 just went down (after a 3h51m lock, and a 1435utc lockloss w/ an ETMx Glitch tag).  Taking H1 immediately into Initial Alignment to skip Locking ALS locklosses. 

The one Owl 932utc lockloss (also an ETMx Glitch tag) was fully automatic (after about 22min of 7 Locking ALS attempts) & no Alignment was needed.

Looks like secondary microseism hit the 95th percentile about 10hrs ago and has stayed there.  Winds are under 10mph.

TITLE: 03/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 01:36 UTC (~4hr 30 min lock).

Extremeley calm IFO that stayed locked for majority of shift.

Lock acquisition after the EQ lockloss (alog bla bla) went fully automatically.

Since then, we've been locked with the microseism steadily increasing. However, the wind has died down.

PSL Dust has been fluctuating a lot (swapping between red, yellow and no alerts) this shift, despite wind calming down.

LOG:

None

TITLE: 03/08 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:

Two locklosses today which were straightforward to recover from.  One happened at the waning minutes of the calibration suite of measurements which is sad (EQ inbound, higher winds, and microseism increasing); this was tagged as a WIND + Calibration lockloss.

Also had 5-tours for our 2nd Saturday Tour day.
LOG:

• H1 was recovering from an ETMx glitch lockloss at beginning of shift.
• 1638 Back to Observing
• 1700-1715 Picked up around the Control Room before Saturday Tour visitors arrive (mainly putting lots of used scratch paper in the "data archive" (aka recycling bin), one coffee cup in dishwasher, one battery in battery recycling container, and organized chairs.
• 1815-1832 930am Saturday tour (cassidy)
  • 1955-2010 1030am Saturday tour (janos)
  • 2040- 1130am Saturday tour (mike)
  • 2125-2156 1230pm Saturday tour (amber)
  • 2245-2314 1:30 Saturday Tour (fred)
• 1911 Incoming EQ alert for M5.7 from southern japan
• 1930 Out of Observing for Saturday Calibration
  • Lockloss (presumably due to incoming EQ) during the last few minutes of the simulines measurement at 1958utc  :(
  • Had WINDY + Calibration tags for this lockloss.
• 1952 Noticed nuc25 PI Mode ndscope was not running for last 4-days---restarted it.
• 1958 LOCKLOSS (right before the Saturday Calibration measurement)
  • 2004-2024 Initial Alignment
  • 2122 OBSERVING....could have been a few minutes sooner, but for some reason a GRD_IFO in "AUTOMATIC OPERATION" failed to take us in automatically which is odd.
• 0002 EQ alert for another one south of Japan
• 0017 Lockloss....guessing it is EQ related since we had an alert for one, but we also have high microseism and winds
  • Proactively went for an Initial Alignment and now handing off to Ibrahim!
  • Late NOTE:  This lockloss has a WINDY tag with winds passing the 20mph threshhold, but we also had the earth moving with that incoming EQ from South Japan and rising usiesm...so, still think it's a toss-up.  Both locklosses with the South Japan EQ never had the picket fence go yellow, did not see any movement on ndscopes and really didn't look bad on seismometer bands, so chalking it up to a comedy of seismic noise teaming up to be an H1 foe.

Lockloss due to EQ, exacerbated by a 5.9 EQ (which interestingly didn't trigger EQ mode).

IFO is relocking now.

TITLE: 03/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 26mph Gusts, 21mph 3min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.41 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 21:22 UTC (3 hr lock).

Unfortunately, we have a 5.9 EQ arriving in a few minutes. If microseism was lower, we could probably ride it out but seeing as we lost lock due to a lower mag EQ with lower microseism just 4ish hrs ago, I would say chances are low.

Calibration sweep did not run today due to an EQ so I can run one pending thermalization (and green light from comissioners/run coord).

Sat Mar 08 10:08:10 2025 INFO: Fill completed in 8min 7secs

FAMIS Link:  26033

Only CPS channels which look higher at high frequencies (see attached) would be the following:

1. ITMx St1 V1
2. ETMy St2 V2

J. Kissel

I've been looking through the data captured about the PMC in the context of the two years of observatory use between July 2022 and July 2024 where we spanned a few "construction, commission, observe" cycles -- see LHO:83020. Remember the end goal is to answer the following question as quantitatively as possible: "does the PMC have a high enough duty cycle in the construction and commissioning phases that the SPI does *not* need to buy an independent laser?"

Conclusions: 
 - Since the marked change in duty cycle after the PMC lock-loss event on 2023-05-16, the duty-cycle of the PMC has been exceptionally high, either 91% during install/commissioning times or 99% during observing times. 
 - Most of the down time is from infrequent planned maintenance. 
 - Recovery time is *very* quick, unless the site loses power or hardware fails. 
 - The PMC does NOT lose lock when the IFO loses lock. 
 - The PMC does NOT lose lock just because we're vented and/or the IMC is unlocked. 
 - To-date, there are no plans to make any major changes to the PSL during the first one or two O4 to O5 breaks.
So, we shouldn't expect to lose the SPI seed light frequently, or even really at all, during the SPI install or during commissioning. And especially not during observing. 

This argues that we should NOT need an independent laser from an "will there even be light?" "won't IFO construction / commissioning mean that we'll be losing light all the time?"  duty-cycle stand point.
Only the pathfinder itself, when fully functional with the IFO, will tell us whether we need the independent laser from a "consistent noise performance" stand-point.

Data and Historical Review
To refresh your memory, the major milestones that happened between 2022 and 2024 (derived from a two year look through all aLOGs with the H1 PSL task):
- By Mar 2022, the PSL team had completed the complete table revamp to install the 2x NeoLase high-power amps, and addressed all the down-stream adaptations.

- 2022-07-01 (Fri): The data set study starts.
- 2022-09-06 (Tue): IO/ISC EOM mount updated, LHO:64882
- 2022-11-08 (Tue): Full IFO Commissioning Resumes after Sep 2022 to Dec 2022 vent to make FDS Filter Cavity Functional (see E2000005, "A+ FC By Week" tab)
- 2023-03-02 (Thu): NPRO fails, LHO:67721
- 2023-03-06 (Mon): NPRO and PSL function recovered LHO:67790
- 2023-04-11 (Tue): PSL Beckhoff Updates LHO:68586
- 2023-05-02 (Tue): ISS AOM realignment LHO:69259
- 2023-05-04 (Thu): ISS Second Loop Guardian fix LHO:69334
- 2023-05-09 (Tue): "Something weird happened to the PMC, then it fixed itself" LHO:69447
- 2023-05-16 (Tue): Marked change in PSL PMC duty-cycle, nothing specific PSL team did with the PMC, but the DC power supplies for the RF & ISC racks we replaced, 69631, while Jason tuned up the FSS path LHO:69637 
- 2023-05-24 : O4, and what we'll later call O4A, starts, we run with 75W requested power from the PSL.
- 2023-06-02 (Fri): PSL ISS AA chassis it was replaced, but PMC stays locked through it LHO:70089
- 2023-06-12 (Sun): PMC PDH Locking PD needs threshold adjustment, LHO:70352, for "never found out why" reason FRS:28260
- 2023-06-19 (Mon): PMC PDH Locking PD needs another threshold adjustment, LHO:70586, added to FRS:28260, but again reasons never found.
- 2023-06-21 (Wed): Decision made to reduce requested power into the IFO to 60W LHO:70648
- 2023-07-12 (Wed): Laser Interlock System maintenance kills PSL LHO:71273
- 2023-07-18 (Tue): Routine PMC / FSS tuneup, with quick PMC recovery LHO:71474
- 2023-08-06 (Sun): Site-wide power glitch takes down PSL LHO:72000
- 2023-09-22 (Fri): Site-wide power gltich takes down PSL LHO:73045
- 2023-10-17 (Tue): Routine PMC / FSS tuneup, with quick PMC recovery LHO:73513
- 2023-10-31 (Tue): Jeff does a mode scan sweeping the PMC FSR LHO:73905
- 2023-11-21 (Tue): Routine PMC / FSS tuneup, with quick PMC recovery LHO:74346
- 2024-01-16 : O4A stops, 3 months, focused on HAM567, no PSL work (see E2000005, "Mid-O4 Break 1" tab)
O4A to O4B break lock losses: 7
       2024-01-17 (Wed): Mid-vent, no IFO, no reported cause.
       2024-01-20 (Sat): Mid-vent, no IFO, no reported cause.
       2024-02-02 (Fri): Mid-vent, no IFO, no reported cause.
       2024-02-08 (Thu): Mid-vent, no IFO, no reported cause. During HAM6 close out, may be related to alarm system
       2024-02-27 (Tue): PSL FSS and PMC On-table Alignment LHO:76002.
       2024-02-29 (Thu): PSL Rotation Stage Calibration LHO:76046.
       2024-04-02 (Tue): PSL Beckhoff Upgrade LHO:76879.
- 2024-04-10 : O4 resumes as O4B start
O4B to 2024-07-01 lock losses: 1
       2024-05-28 (Tue): PSL PMC REFL tune-up LHO:78093.
- 2024-07-01 (Mon): The data set study ends.

- 2024-07-02 (Tue): The PMC was swapped just *after* this data set, LHO:78813, LHO:78814

By the numbers

Duty Cycle (uptime in days / total time in days)
     start_to_O4Astart: 0.8053
    O4Astart_to_O4Aend: 0.9450
    O4Aend_to_O4Bstart: 0.9181
       O4Bstart_to_end: 0.9954
(Uptime in days is the sum on the values of H1:PSL-PMC_RELOCK_DAY just before lock losses [boxed in red] in the attached trend for the given time period)

Lock Losses (number of times "days" goes to zero)
     start_to_O4Astart: 80
    O4Astart_to_O4Aend: 22
    O4Aend_to_O4Bstart: 7
       O4Bstart_to_end: 1
(Number of lock losses is mere the count of red boxes for the given time period)

Lock Losses per calendar days
     start_to_O4Astart: 0.2442
    O4Astart_to_O4Aend: 0.0928
    O4Aend_to_O4Bstart: 0.0824
       O4Bstart_to_end: 0.0123
(In an effort to normalize the locklosses over the duration of the time period to give a more fair assessment.)

I also attach a histogram of lock durations for each duration, as another way to look at how the duty cycle dramatically changed around the start of O4A.