We had a lockloss at 17:03 UTC that has the same "pulse" behavior noted by Jennie and Oli yesterday. This pulses can be seen in the DARM loop, on the L3 master out, and in the EX and EY L2 master outs. These pulses are about 5.8 Hz in frequency.
I'm going to use this post to keep running notes of the 1 Hz investigations:
DHARD P at 1 Hz attached.
We edited the thermalization guardian so that it should set the csoft p gain to 20 only once, after 30 minutes (as before), then after that a user can change it as they wish.
We have been getting saturation warnings on EY after going through the lownoise coil drivers state. We finally tracked it down to the R0 F2 and F3 drivers. The reaction chain length drive (aka reaction chain tracking) is causing these saturations. This servo is run using the L2 OSEMs as a witness to then drive the R0 chain to follow the main chain. I turned off the length drive and the saturations stopped. When I turn it back on it continues to saturate.
The only recent change I can think of is the L2 satamp swap on all the test masses.
Adding a comment: We've only had these saturations intermittently the last few days, in this lock it is now contiuous if the R0 tracking is on. The sat amp swap happened Oct 14th, so perhaps this is some delayed consequence of the swap.
I was also reminded that the EY tiltmeter is possibly not performing as well as it can either as of Tuesday this week.
After the ETMY L2 satamp was swapped back, we no longer get saturations on EY with R0 tracking on!
Because we continue to have issues with the lownoise state of DHARD P (1 Hz ring up), I have commented out both DHARD P lownoise steps in LOWNOISE_ASC. This includes the engagement of FM4 (lownois reshape with low pass) and FM8 (boost). These steps are done separately. This means that we can safely run the LOWNOISE_ASC guardian state, there will just be a little more noise until we can resolve the DHARD P issues.
We had a small mains power glitch 17:25:25 Thu 23 Oct 2025 PDT. It was only reported by the OSB GC UPS.
TITLE: 10/24 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 8mph Gusts, 6mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.62 μm/s
QUICK SUMMARY: H1 was down overnight due to ongoing locking issues. I'll start by running through an initial alignment while I get caught up on events of the past couple days.
After confirming it was damping well over mulitple locks I added the new settings of ITMX13 to lscparams and reloaded the GRD, -30 phase with a gain of -1.0.
TITLE: 10/24 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Microseism is still slowly trending up, I got up past MAX_POWER twice, I confirmed the Roll mode damping works at 62Ws.
LOG: No log.
Lock#1
Lock#2
I modified Oli's script make_EST_model.m that gets the OSEM estimator filters into the front-end models. This script is now the preferred way to get any estimator model filters into the real-time computers. The change was made to add failsafes to avoid further errors on the translation of fitted filters into foton ( see LHO: 87689 ).
For the script to work, any .mat files with estimator filters must have the following syntax:
The format for the filters can be seen in any of the .mat files listed below that live in (svnroot)/sus/trunk/HLTS/Common/FilterDesign/Estimator/. The files come with a README that indicates at how the fits inside were created
est_model_H1PR3_L_2025-10-07.mat -- contains slightly tweaked versions of the PR3 L (and P) models reported in [LHO: 87593]
est_model_H1PR3_P_2025-10-07.mat -- contains slightly tweaked versions of the PR3 P (and L) models reported in [LHO: 87593]
est_model_H1PR3_Y_2025-08-19.mat -- contains the PR3 Y fits from [LHO: 86563]
est_model_H1SR3_L_2025-10-07.mat -- contains slightly tweaked versions of the SR3 L (and P) models reported in [LHO: 87612]
est_model_H1SR3_P_2025-10-07.mat -- contains slightly tweaked versions of the SR3 P (and L) models reported in [LHO: 87612]
est_model_H1SR3_Y_2025-08-05.mat -- contains the cleaned SR3 Y fits from [LHO: 86366]
These are the new .mat files that have the current OSEM estimator models. Until further notice, all previously existing .mat and .m files for OSEM estimator models/fits have been deprecated. We will later talk with Oli to standardize the addition of the OSEM estimator blends.
The updated make_EST_model.m, as well as the rest of the .mat files with the new syntax have been updated to the SUS svn under revision 12754.
TITLE: 10/23 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Multiple locklosses today while continuing to troubleshoot the ifo. Multiple initial alignments were ran (it seems like we're needing them more often the past few days). Locking attempts and locklosses are documented in 87679 and 87701. There are multiple issues so it's been hard trying to distinguish them from each other and investigate one at a time. Secondary microseism has been heading back up too unfortunately.
LOG:
14:30UTC Started trying to relock
- Ran initial alignment
- Various locklosses
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 14:52 | FAC | Randy | YARM | n | Sealing the tube | 21:57 |
| 15:32 | Richard, Ken | MX, MY | n | Checking out emergency pump | 16:03 | |
| 16:28 | FAC | Kim | EX, MX | n | Putting down rain mats | 16:56 |
| 16:44 | FAC | Tyler + contractor | MY | n | Looking at scaffolding | 18:44 |
| 19:37 | Keita | OptLab | y(local) | ISS Array | 22:26 | |
| 20:09 | PCAL | Tony | PCAL Lab | y(local) | Take equipment in and put on apetures | 20:14 |
TITLE: 10/23 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 9mph Gusts, 6mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.51 μm/s
QUICK SUMMARY:
Looking at the last few locklosses from higher states from today and looking for any similarities:
2025-10-23_15:50:38 UTC (CLOSE_BEAM_DIVERTERS) quads, asc
MAX_POWER to LL: 00:08:30
- DHARD increase right before, DARM oscillation @ 10 Hz
2025-10-23_18:23:42 UTC(LOWNOISE_LENGTH_CONTROL) quads, asc
MAX_POWER to LL: 01:14:00
- DARM oscillation @ 6 Hz
2025-10-23_20:59:21 UTC (LOWNOISE_ESD_ETMX) quads, asc
MAX_POWER to LL: 01:13:00
- DARM oscillation @ 10 Hz
2025-10-23_22:12:25 UTC (TRANSITION_FROM_ETMX) quads, asc -> maybe secondary useism?
MAX_POWER to LL: 00:02:30
- IMC-REFL_SERVO_SPLITMON saturated right before the lockloss (possible cause - can high microseism do this?)
Maybe the roll mode we're fighting today isn't ETMY after all?
In the attached plot, the traces are 20 min apart, and you can see that the mode we were watching at 13.75 Hz gets bigger. Throughout this time, Rahul was changing setttings on SUS-ETMY_M0_DARM_DAMP_R, trying to get it to stop increasing. However, when the output of the damping filter was a very small number of counts, we weren't really seeing big changes in the slope of the increase in the monitor filter H1:SUS-ROLL_ETMY_RMS_LOG10_OUTMON. When Rahul increased the gain to 1200 (from the original of 40), we started ringing up a different mode quite quickly. That other mode is 13.94 Hz.
13.94 Hz doesn't match any of the roll mode monitor filters we have in place (second attachment) according to the names of the filters. But, maybe this means that ETMY is actually at 13.94 Hz, and some other optic is at 13.75 Hz? Back in July (alog 84982 that Elenna pointed me to), we originally thought that 13.7 Hz was ETMX, but had found at that time that actuating ETMY seemed to improve things.
Anyhow, we may also consider trying to actuate on other optics, to see if we can do anything to that 13.75 Hz mode? While the monitor filters are all quite narrow (as they should be), the actuator filters appear to all be the same between the optics.
Rahul is working to identify which optic this problematic 13.75 Hz mode is, and so has been actuating on ITMY. We're now seeing a mode at 13.87 Hz with the ITMY actuation, so I think ITMY is absolved from being the problematic optic.
Attached plot is during ITMY actuation.
I am able to ring up 13.75 Hz by actuating on ETMX roll - see attached plot.
I was able to ring up this mode and then damp it down using the following settings - FM2+FM3+FM4, Gain -200 (-60 deg phase filter) for ETMX @53W input power. This needs to be checked at higher power and at NLN.
As per Elenna's suggestion - we will start with damping ITMY roll mode first.
ETMY roll mode gain will be set to zero for the next lock.
To help with troubleshooting H1 locking, I have compiled a list of Guardian code changes which have been made since the last robust locking on Monday 20th October 2025.
30 nodes have been reloaded with modifed code since Monday. Note that some of these reloads were to just resync the nodes to the latest lscparams.py and clear the GRD-CFC.
A summary of the nodes is given in the attached txt file.
Full details of the code changes can be found in /ligo/home/david.barker/tuesdaymaintenance/23oct2025/grd_diffs/
Filter Module changes since Monday.
the following models have had filter module changes since Monday:
h1sus[mc1, mc2, mc3, pr3, sr3]
h1isietm[x, y]
h1asc
The full list of filter differences can be found in /ligo/home/david.barker/tuesdaymaintenance/23oct2025/filter_diffs
Following up on [LHO: 87659], we found an error on the file that transfers the SR3/PR3 OSEM estimator fits posted in [LHO: 87612 LHO: 87593] into foton. Maybe this error explains the 0.67 Hz ringing that we had with the estimator ON.
The make_PR3_estimator_LP.m file was jamming the ISI fits into the M1 drive model. This was a copy-paste type error that I have resolved by now. The corrected code was committed to the SVN under revision 12752.
Next update I will make these pieces of code that commission the estimator less hardcoded to hopefully avoid repeating this mistake in the future.
These filters have been updated in both PR3 and SR3. The estimator is still off though!
There seem to maybe be lines in DARM that I don't recall from the past, so we're looking in to a few of those.
The line at 843 Hz seems to be created in the calibration process, since it is present in GDS-CALIB_STRAIN, but not present in CAL-DELTAL_EXTERNAL. Probably not a problem (unless it's there in NLN), but worth noting that it is *not* related to locking problems, since it's not in the raw DARM data.
Jenne asked me to check a line around 27 Hz, this does show up in max power (today see brown trace) but also was there in max power before maintenance Tuesday this week (pink trace).
It doesn't seem to really show up in the NLN state (red trace is before maintenance, blue is one of our locks from early this morning).
My conclusion from Jennie's plot is that that 27 Hz mode isn't a major cause for concern; it's not something new that's giving us trouble.
