I wrote a script that looks at sudden range drops for both H1 and L1 and searches those times for ETMX glitches. With this script I have been able to confirm that LLO gets ETMX glitches that they're able to ride out. However, we don't know if the glitches cause locklosses for them too.

I used /ligo/home/oli.patane/Documents/WIP/etmglitch/range_drop_investigation.ipynb to look for ETMX glitches that would cause the range to drop below 100 Mpc. I have only looked over a few days at each ifo, but it's already clear that they definitely have ETMX glitches, or some glitch that presents itself very similarly. The plots for LHO can be found in /ligo/home/oli.patane/Documents/WIP/etmglitch/range_drop_investigation/H1/, and the LLO plots in /ligo/home/oli.patane/Documents/WIP/etmglitch/range_drop_investigation/L1/. I've attached a couple here of each as examples.

I wanted to make the plots between the two ifos as similar as possible to help better judge glitch size and channels it appears in. Both ifos have matching ylims for L1, L2, and L3, and although I couldn't use the same ylims for the DCPDs, I have scaled them so the delta between the ymin and ymax is 0.3 mA for both ifos. Unfortunately, I was not able to do any scaling for DARM or CALIB STRAIN due to the amount they vary between both locks as well as between ifos.

LHO example1, LHO example2

LLO example1, LLO example2

Both LHO and LLO seem to have ETMX glitches that both appear alone and in groups. As you can see, LLO generally has much noisier ETMX L3, DCPD, and DARM channels. This hides the true morphology of the glitches in ETMX L3, and may be preventing us from seeing the glitch appear in the DCPDs and DARM as often as they appear in LHO's DCPDs and DARM channels. In LLO's examples, you can see very small glitches in the DCPDs and DARM at the same time, but proportional to the entire trace, they aren't affecting those channels as much as they can do at LHO. Feel free to take a look through the rest of the glitch examples in the directories to get a better idea of the range of how these glitches can present and affect the different parts of the ifo.

Through messing with this script I've also been able to find good thresholds to use for searching for these glitches at LLO, since their DARM and ETMX L3 channels are much noisier than ours, so it would be very easy to implement an ETMX glitch lockloss search/tag for them.

21:42 UTC lockloss, we were starting to shake from a 5.6 from El Salvador but it looks to be another ASC_Y / TMSX_Y oscillation lockloss.

Sheila, Camilla WP# 12716

We followed the setup in 80451 and additionally I plugged BNC's inside SQZT7 to directly connect t-ed off OPO_IR_PD and OPO_TRANS PDs to the table feed-through, as OPO REFL "test in" and "test out", see photo. This let us do all the work in laser safe and have been left plugged in. 

We started in the second leftmost spot from the right (85589, 85488, 85297), had been there since late June but the spot degraded quickly and the green power used to achieve 80uW from OPO was now ~30mW which is close to the max available.

We first checked where we were by going to the left edge of the crystal and then returned to the leftmost spot.

Once there we measured NLG, moved the crystal in steps on 100 counts, re-optimized OPO temperature and measured NLG, similar to 85589. Once we got to a NLG above 15 with 80uW of OPO TRANS, we stopped. The required green power used to achieve 80uW was now halved from 30mW to 15mW, much better. We started witht the OPO temps et to 31.2deg and ended at 32.3deg, this is closer to where we were operating 3 years ago, photo. 

As the next step in fine-tuning the SR3 Y estimator, we needed to retake the SUSPOINT to M1 measurements as well as OLTFs so they could be used to calculate the filter modules for the suspoint and drive estimators. I took those measurements today.

General setup:
- in HEALTH_CHECK (but damping back on)
    - damping for Y changed from -0.5 to -0.1
- OSEMINF gains and DAMP FM7 turned on (and left on afterwards 86070)

SUSPOINT to M1:
Data for those measurements can be found in /ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/SR3/Common/Data/2025-07-29_1730_H1ISIHAM5_ST1_WhiteNoise_SR3SusPoint_{L,T,V,R,P,Y}_0p02to50Hz.xml  r12492

M1 to M1 (OLTFs):
After this, the next steps are to take regular transfer functions with the above setup of Y having -0.1 damping
That data is in /ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/SR3/SAGM1/Data/2025-07-29_1830_H1SUSSR3_M1_WhiteNoise_{L,T,V,R,P,Y}_0p02to50Hz_OpenLoopGainTF.xml  r12493
Reminder that these open loop transfer functions were taken with the damping Y gain of -0.1, so they should not be taken as 'nominal' OLTFs.

Shivaraj Kandhasam posted in MatterMost that the EY BRS signal looked off, and had been since last Tuesday (July 22). I talked to Jim about this and indeed the velocity of the BRS was consistently very high. Jim said that it looked like it needed to grab new frames for the C code calculations. Immediately after Jim reset this, by logging into the BRS computer and clicking the "Grab New Frames" button on the GUI, the velocity slowed down and it was quickly brought back to a normal realm.

I trended this back and it's happened a few times, most recently from May 21-June 5. To catch this next time it happens again, I've done two things:

1. Lowered the threshold that the BRS{X,Y}_STAT nodes use before moving into their DAMPER_ON_HIGH_VEL state. When these nodes go to this state, then the ST1 ETM sensor correction nodes will remove the BRS corrected signal from sensor correction, since it assumes that the sensor is not in a reliable state. The velocity during the corrupted frame times tends to be +/- 3600 or so, so I changed the threshold from 10000 to 6800. We rarely get above this value, and when we do it's from maintenance activities, very large earthquakes, or extreme wind to the point where locking isn't happening anyway. This is commonly shared code, so I had to check that this value would work for LLO as well. I've only been able to go back a week or two though before I was running into data issues. I'll double check this with LLO then coordinate with them to update the value if desired.
2. Added a test in DIAG_MAIN to watch the BRS{X,Y}_STAT nodes and if they are in the DAMPER_ON_HIGH_VEL state for more than 6 hours consistently, then it will notify. This should be enough time to let large earthquakes ring down and allow for a dip in extreme winds to reset the timer. This test is not common, but LLO can copy if they so choose.

We previoiusly swapped the satamp boxes for TMSX M1 F1/F2/F3/LF (85980), and I had just put in the generic 5.31:0.0969 zp compensation filters in at the time since the 'best possible' filters for that satamp were under the optic name of the OMC in the txt files. Now that we've decided that we're going to be keeping this satamp box in, we have fixed the naming in the txt files and so I was able to update the compensation filters to be the 'best possible' for each satamp channel (output). These were loaded in.

As part of the estimator work, we measured and calculated OSEMINF gains (85907) and compensating gain filter modules for the DAMP bank (86026). We believe these new values to be correct, so we have gone ahead and permanently updated the OSEMINF gains as well as turned on the compensating gains in FM7 in the DAMP filter bank for SR3.
This update means that there will be a difference in the apparent location of the DAMP INs, but this is just because of the change in the OSEMINF gains. These gains were changed (along with the compensating gains filter modules turned on) at 2025-07-29 16:58 UTC. They were put into SDF safe and will be saved in OBSERVE once we get there.

The compensation gains put in in the DAMP filter bank are in FM7, and they are the following:

L: 0.740
T: 0.732
V: 0.548
R: 0.550
P: 0.628
Y: 0.757

Randy, Jim, and Mitchell

One section of the work platform was installed on the -X side of BSC2 this morning.

LVEA has been swept.

Closes WP 12719.

Patrick T., Gerardo M., Dave B.

"Update the h0vacly Beckhoff TwinCAT 3 system manager and PLC code to add the IPFCC6 and IPFCC8 filter cavity tube ion pumps and the PTCC7 filter cavity tube gauge. Will require a restart of the PLC and IOC. Will require a DAQ restart."

This has been completed. The code on h0vacly is now at commit 6a6e7bc55cb87bc9188312eee1c9a8129bdf7946. No issues. Dave did a burt restore. A DAQ restart is pending.

Trying to narrow down why TMS x is involved in lock losses we have replaced the TMS coil driver that works on F1,2,3 and LF.  Chassis S1102670 was replaced with S1102666.  The operator returned the system to damping.  This is a wait and see test.

Tue Jul 29 10:08:48 2025 INFO: Fill completed in 8min 44secs

After Richard swapped the TMSX coil driver chassis I took a look at the raw OSEM counts for the TMSX (F1, F2, F3, LF) comparing to a time earlier when TMSX and ISC_LOCK were both in the same state (ALIGNED, DOWN). I see that LFs counts are ~2500 counts lower post swap. I'm not sure if there are any other checks to run.

The dolphin IX switch at EX had broken management interface which prevented fencing of front end from the dolphin network for safe reboots.

I installed a new switch.  As part of the process, h1cdsrfm was turned off to avoid crashing it and in turn crashing CS and EY dolphin networks. 

When I turned h1cdsrfm back on, the cdsrfm port on the switch was already enabled, which led to timing glitches on h1susex and h1iscex.  This could have been prevented by re-fencing h1cdsrfm after the new switch was turned on.

I restarted the models on h1susex and h1iscex.  A few minutes later, h1susex crashed in the dolphin driver and had to be rebooted.  The crash was reminiscent of several susex (plus all of ey) crashes related to restarts of dis_networkmgr on the bootserver.  See 67389 for one example.

TITLE: 07/29 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 0mph Gusts, 0mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

• We've been locked for almost 9 hours
• Magnetic injections are running, to be followed by SUS charge
• Lots of maintenance tasks on the docket today

Fil, Elenna, Oli

In an effort to find and solve the ASC excursion locklosses that seem to be linked to TMSX (85973), Fil swapped out the satellite amplifier that he had installed last Tuesday (85770) that had serial number S1100150 with another modified satamp that he had on hand, S1100122 (originally meant for OMC T1 T2 T3 LF). We are hoping this fixes the problem. Since this is partially for testing and since we originally were planning to use this satellite amplifier for the OMC, for now I've replaced the OSEMINF compensation filters for TMSX (which had the specific tuned filters) with the generic 5.31:0.0969 zp filters. We can update these later when we figure out which satamp will be staying there. I've loaded these filters in and brought TMSX back.

taken out: S1100150

put in: S1100122 (originally meant for OMC T1 T2 T3 LF)

Ivey used the ISO calibration measurements that I took earlier (85906) to calculate what the OSEMINF gains should be on SR3 (85907), and this script also calculates what it thinks the compensation gain in the DAMP filter bank should be.
The next step is to use OLG TFs to measure what values we would use in the DAMP filter bank to compensate for the change in OSEMINF gains, and we can compare them to the calculated values to see how close they are.

I took two sets of OLG measurements for SR3:
- a set with the nominal OSEMINF gains
    T1: 1.478
    T2: 0.942
    T3: 0.952
    LF: 1.302
    RT: 1.087
    SD: 1.290
- a set with the OSEMINF gains changed to the values in 85907
    T1: 3.213
    T2: 1.517
    T3: 1.494
    LF: 1.733
    RT: 1.494
    SD: 1.793

Measurement settings:
- SR3 in HEALTH_CHECK but with damping loops on
- SR3 damping nominal (all -0.5)
- HAM5 in ISOLATED

Nominal gain set:
/ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/SR3/SAGM1/Data/2025-07-22_1700_H1SUSSR3_M1_WhiteNoise_{L,T,V,R,P,Y}_0p02to50Hz_OpenLoopGainTF.xml r12478

New gain set:
/ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/SR3/SAGM1/Data/2025-07-22_1800_H1SUSSR3_M1_WhiteNoise_{L,T,V,R,P,Y}_0p02to50Hz_OpenLoopGainTF.xml r12478

Once I had taken these measurements, I exported txt files for each dof's OLG and used one of my scripts, /ligo/svncommon/SusSVN/sus/trunk/HLTS/Common/MatlabTools/divide_traces_tfs.m to plot the OLG for each dof to compare the traces between OSEMINF gain differences and then divide the traces and grab an average of that, which will be the compensation gain put in as a filter in the DAMP filter bank (plots). The values I got for the compensation gains are below:
    L: 0.740
    T: 0.732
    V: 0.548
    R: 0.550
    P: 0.628
    Y: 0.757

 These are pretty similar to what my script had found them to be last time before the satamp swap (85288), as well as being very similar to the values that Ivey's script had calculated.
Maybe the accuracy from Ivey's script means that in the future we don't need to run the double sets of OLG transfer functions and can jsut use the values that the script gives.