TITLE: 10/23 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Multiple locklosses today while continuing to troubleshoot the ifo. Multiple initial alignments were ran (it seems like we're needing them more often the past few days). Locking attempts and locklosses are documented in 87679 and 87701. There are multiple issues so it's been hard trying to distinguish them from each other and investigate one at a time. Secondary microseism has been heading back up too unfortunately.
LOG:
14:30UTC Started trying to relock
- Ran initial alignment
- Various locklosses
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 14:52 | FAC | Randy | YARM | n | Sealing the tube | 21:57 |
| 15:32 | Richard, Ken | MX, MY | n | Checking out emergency pump | 16:03 | |
| 16:28 | FAC | Kim | EX, MX | n | Putting down rain mats | 16:56 |
| 16:44 | FAC | Tyler + contractor | MY | n | Looking at scaffolding | 18:44 |
| 19:37 | Keita | OptLab | y(local) | ISS Array | 22:26 | |
| 20:09 | PCAL | Tony | PCAL Lab | y(local) | Take equipment in and put on apetures | 20:14 |
TITLE: 10/23 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 9mph Gusts, 6mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.51 μm/s
QUICK SUMMARY:
Looking at the last few locklosses from higher states from today and looking for any similarities:
2025-10-23_15:50:38 UTC (CLOSE_BEAM_DIVERTERS) quads, asc
MAX_POWER to LL: 00:08:30
- DHARD increase right before, DARM oscillation @ 10 Hz
2025-10-23_18:23:42 UTC(LOWNOISE_LENGTH_CONTROL) quads, asc
MAX_POWER to LL: 01:14:00
- DARM oscillation @ 6 Hz
2025-10-23_20:59:21 UTC (LOWNOISE_ESD_ETMX) quads, asc
MAX_POWER to LL: 01:13:00
- DARM oscillation @ 10 Hz
2025-10-23_22:12:25 UTC (TRANSITION_FROM_ETMX) quads, asc -> maybe secondary useism?
MAX_POWER to LL: 00:02:30
- IMC-REFL_SERVO_SPLITMON saturated right before the lockloss (possible cause - can high microseism do this?)
Maybe the roll mode we're fighting today isn't ETMY after all?
In the attached plot, the traces are 20 min apart, and you can see that the mode we were watching at 13.75 Hz gets bigger. Throughout this time, Rahul was changing setttings on SUS-ETMY_M0_DARM_DAMP_R, trying to get it to stop increasing. However, when the output of the damping filter was a very small number of counts, we weren't really seeing big changes in the slope of the increase in the monitor filter H1:SUS-ROLL_ETMY_RMS_LOG10_OUTMON. When Rahul increased the gain to 1200 (from the original of 40), we started ringing up a different mode quite quickly. That other mode is 13.94 Hz.
13.94 Hz doesn't match any of the roll mode monitor filters we have in place (second attachment) according to the names of the filters. But, maybe this means that ETMY is actually at 13.94 Hz, and some other optic is at 13.75 Hz? Back in July (alog 84982 that Elenna pointed me to), we originally thought that 13.7 Hz was ETMX, but had found at that time that actuating ETMY seemed to improve things.
Anyhow, we may also consider trying to actuate on other optics, to see if we can do anything to that 13.75 Hz mode? While the monitor filters are all quite narrow (as they should be), the actuator filters appear to all be the same between the optics.
Rahul is working to identify which optic this problematic 13.75 Hz mode is, and so has been actuating on ITMY. We're now seeing a mode at 13.87 Hz with the ITMY actuation, so I think ITMY is absolved from being the problematic optic.
Attached plot is during ITMY actuation.
I am able to ring up 13.75 Hz by actuating on ETMX roll - see attached plot.
I was able to ring up this mode and then damp it down using the following settings - FM2+FM3+FM4, Gain -200 (-60 deg phase filter) for ETMX @53W input power. This needs to be checked at higher power and at NLN.
As per Elenna's suggestion - we will start with damping ITMY roll mode first.
ETMY roll mode gain will be set to zero for the next lock.
To help with troubleshooting H1 locking, I have compiled a list of Guardian code changes which have been made since the last robust locking on Monday 20th October 2025.
30 nodes have been reloaded with modifed code since Monday. Note that some of these reloads were to just resync the nodes to the latest lscparams.py and clear the GRD-CFC.
A summary of the nodes is given in the attached txt file.
Full details of the code changes can be found in /ligo/home/david.barker/tuesdaymaintenance/23oct2025/grd_diffs/
Filter Module changes since Monday.
the following models have had filter module changes since Monday:
h1sus[mc1, mc2, mc3, pr3, sr3]
h1isietm[x, y]
h1asc
The full list of filter differences can be found in /ligo/home/david.barker/tuesdaymaintenance/23oct2025/filter_diffs
Looking back at this alog #87678 from Ibrahim, I looked at the lockloss tool for this time and could see the 1Hz growing oscillation on DHARD_P but also three impulse/glitchy things on ETMY drive and showing up in DARM1 loop.
Thu Oct 23 10:08:16 2025 INFO: Fill completed in 8min 13secs
Gerardo confirmed a good fill curbside.
Attached is comparison of the LSC FF drives before the HAM1 vent (reference traces) and today (live traces). The first time we noticed the roll mode ringing was around June 11, one of the first days we were back into observing after the HAM1 vent (84982).
It doesn't look like there is significantly any difference in the drive between 10-20 Hz for any of the feedforwards. I have already previously checked the filters for high Q features around 13 Hz that might drive the roll mode and there are none. Sheila previously noticed that the roll mode started to ring up after the SRCL and PRCL feedforwards were engaged (87690).
We did a comparison of the FF drives before and after Tuesday, posting here for reference:
The first three are from two days ago with overall MICH FF gain =2, the last three are from earlier today with MICH FF gain =4.
The forst three are from NLN, the second three from low noise length control this morning when we had the MICH FF on.
Following up on [LHO: 87659], we found an error on the file that transfers the SR3/PR3 OSEM estimator fits posted in [LHO: 87612 LHO: 87593] into foton. Maybe this error explains the 0.67 Hz ringing that we had with the estimator ON.
The make_PR3_estimator_LP.m file was jamming the ISI fits into the M1 drive model. This was a copy-paste type error that I have resolved by now. The corrected code was committed to the SVN under revision 12752.
Next update I will make these pieces of code that commission the estimator less hardcoded to hopefully avoid repeating this mistake in the future.
There seem to maybe be lines in DARM that I don't recall from the past, so we're looking in to a few of those.
The line at 843 Hz seems to be created in the calibration process, since it is present in GDS-CALIB_STRAIN, but not present in CAL-DELTAL_EXTERNAL. Probably not a problem (unless it's there in NLN), but worth noting that it is *not* related to locking problems, since it's not in the raw DARM data.
Jenne asked me to check a line around 27 Hz, this does show up in max power (today see brown trace) but also was there in max power before maintenance Tuesday this week (pink trace).
It doesn't seem to really show up in the NLN state (red trace is before maintenance, blue is one of our locks from early this morning).
My conclusion from Jennie's plot is that that 27 Hz mode isn't a major cause for concern; it's not something new that's giving us trouble.
TITLE: 10/23 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 5mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.39 μm/s
QUICK SUMMARY:
IFO was in IDLE when I got in due to issues last night with a high EY Roll mode. I just started an initial alignment so we can be further along in locking once commissioners arrive onsite.
Locking attempt 1 this morning: Locked to full power, Oli ran the asc high bandwidth script (userapps/isi/h1/scripts/revert_asc.py) premptively to avoid having problems with the 1 Hz ring up.
The last thing this script does is sets the LSC FF gains to 0, 45 seconds after this finished we had a sudden large increase in noise first at 300 Hz in DARM (ETMX ESD) and DHARD then 0.1 second later in CHARD, which unlocked the IFO. This happened in the close beam diverters state, but is probably not related to the beam diverters. This is a different problem than what we've been seeing overnight.
No gains or anything else in the CHARD or DHARD filter banks were changed in those 45 seconds either, which rules out some other rogue change
Elenna and I commented out the DHARD boost from low noise ASC, because it takes 45 degrees of phase at 1Hz and so leaving it off may help our improve our 1Hz problems. This boost was added to improve the stationarity of SRCL to DARM coupling by reducing the DAHRD RMS 77857. We are leaving it off for now because we can probably go to observing without it if we can stay locked.
Next locking attempt we sat for over an hour at full power, without doing the LOWNOISE ASC step.
We had ETMY saturations when we ran through the ESD transitions, and they continued after DARM was switched, until Elenna turned off LSC FF. When we turned the feedforward back on ETMY didn't start saturating. However, at the time that the PRCL and SRCL feedforward came on, the roll mode started ringing up. We lost lock a few minutes into this ring up, but the amplitude was much smaller than the locklosses from this mode overnight (about a factor of 100 lower).
We aren't sure if the roll mode was the cause of the lockloss or not.
18:45 UTC While Jim was running a measurement on ISI HAM5, SUS ITMY started to flash its' red Rocker Switch Death on L2 and report that its' L2 OSEMs were in fault, we also noticed there's a big new line at ~92Hz. Jeff turned off feedback from L2. It looks like it could be a sat amp issue, the L2 sat amp was upgraded this past Tuesday alog87469. Jeff is starting to work on characterizing a sat amp box in the EE shop to swap in.
We would need a lockloss to swap this component.
We are going to try to go back into Observing for a bit until we can fix it or lose lock.
19:31 UTC Observing
FYI -- we went into observing with the ITMY PUM sat amp OSEM PD's still dead. We turned OFF the ITMY L2 (PUM) to R0 "damping" feedback loop, so the failed sensors *should* not have any impact on the IFO. However, an obvious, loud ~92-93 Hz feature remained new and glaringly obvious in the DARM sensitivity.
@DetChar / @CW Group Probably best to just veto this observation stretch, unless we -- in the future -- find some amazingly exceptional event.
2025-10-16 19:31 UTC to 19:54 UTC.
ITMY L2 (PUM) Sat Amp S1100148 installed on 10/14/2025. Replaced on 10/16/2024 with S1100080.
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=87469
WP 12836
ECR E2400330
Modified List T2500232
The following SUS SAT Amps were upgraded per ECR E2400330. Modification improves the whitening stage to reduce ADC noise from 0.05 to 10 Hz.
| Suspension | Old | New | OSEM |
| ETMX L2 (PUM) | S1100146 | S1100119 | ULLLURLR |
| ETMY L2 (PUM) | S1100137 | S1100127 | ULLLURLR |
| ITMX L2 (PUM) | S1100135 | S1100118 | ULLLURLR |
| ITMY L2 (PUM) | S1000277 | S1100148 | ULLLURLR |
F. Clara, J. Kissel, O. Patane
Here's the characterization data and fit results for S1100148, assigned to ITMY L2's ULLLURLR OSEMs.
This sat amp is a UK 4CH sat amp, D0900900 / D0901284. The data was taken per methods described in T080062-v3, using the diagrammatic setup shown on PAGE 1 of the Measurement Diagrams from LHO:86807.
The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/
plotresponse_S1100148_ITMY_L2_ULLLURLR_20250917.m
Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are:
| Optic | Stage | Serial_Number | Channel_Number | OSEM_Name | Zero_Pole_Hz | R_TIA_kOhm | Foton_Design |
|---|---|---|---|---|---|---|---|
| ITMY | L2 | S1100148 | CH1 | UL | 0.095:5.19 | 120.5 | zpk([5.19],[0.095],1,"n") |
| CH2 | LL | 0.0957:5.24 | 120.0 | zpk([5.24],[0.0957],1,"n") | |||
| CH3 | UR | 0.0958:5.24 | 120.125 | zpk([5.24],[0.0958],1,"n") | |||
| CH4 | LR | 0.0967:5.28 | 120.375 | zpk([5.28],[0.0967],1,"n") |
The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Results/
2025-09-17_UKSatAmp_S1100148_D0901284-v5_fitresults.txt
Per usual, R_TIA_kOhm is not used in the compensation filter -- but after ruling out an adjustment in the zero frequency (by zeroing the phase residual at the lowest few frequency points), Jeff nudged the transimpedance a bit to get the magnitude scale within the ~0.25%, shown in the attached results. Any scaling like this will be accounted for instead with the absolute calibration step, i.e. Side Quest 4 from G2501621, a la what was done for PR3 and SR3 top masses in LHO:86222 and LHO:84531 respectively.
Here's the characterization data and fit results for S1100119, assigned to ETMX L2's ULLLURLR OSEMs.
This sat amp is a UK 4CH sat amp, D0900900 / D0901284. The data was taken per methods described in T080062-v3, using the diagrammatic setup shown on PAGE 1 of the Measurement Diagrams from LHO:86807.
The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/
plotresponse_S1100119_ETMX_L2_ULLLURLR_20250916.m
Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are:
| Optic | Stage | Serial_Number | Channel_Number | OSEM_Name | Zero_Pole_Hz | R_TIA_kOhm | Foton_Design |
|---|---|---|---|---|---|---|---|
| ETMX | L2 | S1100119 | CH1 | UL | 0.0949:5.19 | 120 | zpk([5.19],[0.0949],1,"n") |
| CH2 | LL | 0.0971:5.32 | 120 | zpk([5.32],[0.0971],1,"n") | |||
| CH3 | UR | 0.0964:5.27 | 120 | zpk([5.27],[0.0964],1,"n") | |||
| CH4 | LR | 0.0957:5.23 | 120 | zpk([5.23],[0.0957],1,"n") |
The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Results/
2025-09-16_UKSatAmp_S1100119_D0901284-v5_fitresults.txt
Per usual, R_TIA_kOhm is not used in the compensation filter -- but after ruling out an adjustment in the zero frequency (by zeroing the phase residual at the lowest few frequency points), Jeff nudged the transimpedance a bit to get the magnitude scale within the ~0.25%, shown in the attached results. Any scaling like this will be accounted for instead with the absolute calibration step, i.e. Side Quest 4 from G2501621, a la what was done for PR3 and SR3 top masses in LHO:86222 and LHO:84531 respectively.
Here's the characterization data and fit results for S1100118, assigned to ITMX L2's ULLLURLR OSEMs.
This sat amp is a UK 4CH sat amp, D0900900 / D0901284. The data was taken per methods described in T080062-v3, using the diagrammatic setup shown on PAGE 1 of the Measurement Diagrams from LHO:86807.
The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/
plotresponse_S1100118_ITMX_L2_ULLLURLR_20250916.m
Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are:
| Optic | Stage | Serial_Number | Channel_Number | OSEM_Name | Zero_Pole_Hz | R_TIA_kOhm | Foton_Design |
|---|---|---|---|---|---|---|---|
| ITMX | L2 | S1100118 | CH1 | UL | 0.0966:5.27 | 120 | zpk([5.27],[0.0966],1,"n") |
| CH2 | LL | 0.0961:5.25 | 120 | zpk([5.25],[0.0961],1,"n") | |||
| CH3 | UR | 0.0963:5.26 | 120 | zpk([5.26],[0.0963],1,"n") | |||
| CH4 | LR | 0.097:5.3 | 120 | zpk([5.3],[0.097],1,"n") |
The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Results/
2025-10-14_UKSatAmp_S1100118_D0901284-v5_fitresults.txt
Per usual, R_TIA_kOhm is not used in the compensation filter -- but after ruling out an adjustment in the zero frequency (by zeroing the phase residual at the lowest few frequency points), Jeff nudged the transimpedance a bit to get the magnitude scale within the ~0.25%, shown in the attached results. Any scaling like this will be accounted for instead with the absolute calibration step, i.e. Side Quest 4 from G2501621, a la what was done for PR3 and SR3 top masses in LHO:86222 and LHO:84531 respectively.
Here's the characterization data and fit results for S1100127, assigned to ETMY L2's ULLLURLR OSEMs.
This sat amp is a UK 4CH sat amp, D0900900 / D0901284. The data was taken per methods described in T080062-v3, using the diagrammatic setup shown on PAGE 1 of the Measurement Diagrams from LHO:86807.
The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/
plotresponse_S1100127_ETMY_L2_ULLLURLR_20250916.m
Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are:
| Optic | Stage | Serial_Number | Channel_Number | OSEM_Name | Zero_Pole_Hz | R_TIA_kOhm | Foton_Design |
|---|---|---|---|---|---|---|---|
| ETMY | L2 | S1100127 | CH1 | UL | 0.0963:5.26 | 121.25 | zpk([5.26],[0.0963],1,"n") |
| CH2 | LL | 0.0958:5.24 | 121.25 | zpk([5.24],[0.0958],1,"n") | |||
| CH3 | UR | 0.0952:5.2 | 121.25 | zpk([5.2],[0.0952],1,"n") | |||
| CH4 | LR | 0.0954:5.21 | 121.25 | zpk([5.21],[0.0954],1,"n") |
The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Results/
2025-09-16_UKSatAmp_S1100127_D0901284-v5_fitresults.txt
Per usual, R_TIA_kOhm is not used in the compensation filter -- but after ruling out an adjustment in the zero frequency (by zeroing the phase residual at the lowest few frequency points), Jeff nudged the transimpedance a bit to get the magnitude scale within the ~0.25%, shown in the attached results. Any scaling like this will be accounted for instead with the absolute calibration step, i.e. Side Quest 4 from G2501621, a la what was done for PR3 and SR3 top masses in LHO:86222 and LHO:84531 respectively.
ITMY L2 (PUM) Sat Amp S1100148 installed on 10/14/2025. Replaced on 10/16/2024 with S1100080.
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=87515