I think this is causing us locklosses, twice we've lost lock in this state as it turns on when I slowly stepped through the states, and twice we've lost it a few seconds into POWER_10Ws when GRD was moving automatically. I reduced the gain to 30 from 40 (SVN commited and reloaded ISC_LOCK, I had to first commit the DAMP_BOUNCE_ROLL state edits) and doubled the tramp to 10 (SDFed in SAFE).

The reduced gain and increased tramp didn't stop it from killing the lock, as soon as it engaged we lost lock. I've commented it out from ISC_LOCK - line 3937.

I think the BOUNCE_ROLL channel was mistyped in ISC_LOCK, the line is ezca['SUS-ETMY_M0_DAMP_R_GAIN'] = 40 where it should be ezca['SUS-ETMY_M0_DARM_DAMP_R_GAIN'] = 40 ? I should have noticed this earlier.

I edited the channel in ISC_LOCK to add "DARM_" but I did not get a chance to reload before we went into Observing.

The cause of the range drop and eventual lockloss this morning appears to be from the problematic roll mode we've been seeing recently (see alog84982).

If I see that it's still rung up once H1 relocks, I'll apply the damping gain of 30 that seemed to work yesterday evening.

These MCL trigger thresholds come from the IMC_LOCK Guardian and are set in the 'DOWN' and 'MOVE_TO_OFFLINE' states.

In 'DOWN', the trigger ON and trigger OFF thresholds are set at 1300 and 200, respectively, for the IMC to prepare to lock as seen in the setpoints from Corey's screenshot.

In 'MOVE_TO_OFFLINE', the trigger ON and trigger OFF thresholds are set at 100 and 90, respectively (for <4W input), as seen in the EPICS values from Corey's screenshot.

So, it would seem that after the lower thresholds were set when taking the IMC offline sometime recently, they were incorrectly accepted in SDF. I'll accept them as the correct values in the OBSERVE.snap table once H1 is back up to low noise, as I expect they'll show up as a difference again.

2025-06-12 23:39 UTC Back to Observing

Accepted sdfs for SUSPROC (filter for Roll modes) and for LSC (IMC-MCL_FM_TRIG thresholds) to get into Observing. Control room at the time didn't know who changed the MCL thresholds, so I just accepted them.

Since we were able to actuate on ETMY, that indicates this is likely the ETMY roll mode. That indicates that either the ASC or the LSC feedforward is driving this mode.

I just took a look at the SRCL feedforward, and I see a pair of high Q zeros/poles (Q ~ 1600!) at about 13.5 Hz that I missed (facepalm). That seems a bit low, since this roll mode appears to be around 13.7 Hz, but we should still remove that anyway. We can take care of this tomorrow during commissioning and that might prevent the driving of this mode.

I can't think of any ASC control that would have a significant drive at 13.7 Hz.

I just removed two high Q features at 6.5 and 13.5 Hz that were in the SRCL feedforward. I kept the same filter but removed the features, so there should be no SDF or guardian changes. My hope is that this will prevent the roll mode from being rung up, so I have turned the gain off and SDFed it. Attached is a screenshot of the SDF change for ETMY roll.

Unfortunately, it appears that the ETMY roll mode is still ringing up, so the SRCL feedforward is not the cause. Another possibility is the ASC. The CHARD P control signal is larger now around 13.7 Hz than it was in April. The attached plot shows a reference trace from April and live trace from last night's lock. I don't know if this is enough to drive this mode. The bounce roll notches are engaged on ETMY L2, and have 40 dB attenuation for the roll mode between 13.4 and 13.7 Hz.

These large BS PIT changes began 5th to 6th July 2024 (plot). This is the day shift from the time that the first lock like this happened 5th July 2024 19:26UTC (12:26PT): 78877 at the time we were doing PR2 spot moves. There also was a SUS computer restart 78892 but that appeared to be a day after this started happening.

Sheila, Camilla

This reminded Sheila of when we were heating a SUS in the past and causing the bottom mass to pitch and the ASC to move the top mass to counteract this. Then after lockloss, the bottom mass would slowly go back to it's nominal position.

We do see this on the BS since the PR2 move, see attached (top 2 left plots). See in the green bottom mass oplev trace, when the ASC is turned off on lockloss, the BS moves quickly and then slowly moves again over the next ~30 minutes, do not see simular things on PR3. Attached is the same plot before the PR2 move. And below is a list of other PR2 positions we tried, all the other positions have also made this BS drift. The total PR2 move since the good place is ~3500urad in Yaw.

• Different time May 21st to 24th 2024:
  • BS Oplev Drift
  • Plot shows 30urad M1 drift
  • PR2 Alignment Sliders P: 1435, Y: 1130
• Pre July 5th 2024:
  • No BS Oplev Drift
  • Plot shows 5urad M1 drift
  • PR2 Alignment Sliders P: 1565, Y: 3210
• July 5th 2024 to 6th Feb 2025:
  • BS Oplev Drift
  • Plot shows 50urad M1 drift
  • PR2 Alignment Sliders P: 1535, Y: 2785
• 6th Feb 2025 to 10th Feb 2025:
  • BS Oplev Drift
  • Plot shows 30urad M1 drift
  • PR2 Alignment Sliders P: 1480, Y: 1195
• 10th Feb 2025 to now:
  • BS Oplev Drift
  • Plot shows 30-40urad M1 drift
  • PR2 Alignment Sliders P: 1430, Y: -245

To avoid this heating and BS drift, we should move back towards a PR2 YAW of closer to 3200. But, we moved PR2 to avoid the spot clipping on the scrapper baffle, e.g.  77631, 80319, 82722, 82641.

I did a bit of alog archaeology to re-remember what we'd done in the past.

• In August of 2015, we found that we were struggling with PR3 pitch alignment jumping, then cooling down upon lockloss.  Alog 20268 talks about the implementation of the lock loss compensation, which first appeared in ISC_DRMI guardian in rev 11228.
• At some point (I didn't dig to find out when precisely), we also implemented the same filters for BS pitch.
• By Jan 2020, both BS and PRM had the soft ASC turn-off.
• In Jan 2020, ISC_DRMI rev 20905 we removed this soft ASC turn-off for both PR3 and BS.  The referenced alog 54709 notes that we shouldn't need those anymore, since we had installed wire heating baffles, to prevent the wires from being illuminated and heating up.
• We haven't had the soft turn-off filters in use since 2020, about 3 months before the end of O3b.  This may be why Camilla saw that we were seeing BS drift at the end of O3b.
• Perhaps our alignment during O4, until we moved the PR spots in May 2024, was such that we weren't susceptible to this wire heating.
• I don't think PR3 is seeing the same kind of trouble that it did back in 2015 upon lockloss, so I think its wire heating baffles are working as designed, so no need to make any changes to the PR3 controls.
• Sheila made the point that because we unclipped some of the +Y side of the beam (without moving the spot on the BS), maybe there is a bit more light that is illuminating the barrel of the BS or getting to the wires.  Or, something?  Without having looked at the actual drawings, I could imagine that the wire heating baffles are working better on PR3 than they are on the BS, because we hit PR3 much closer to normal incidence, whereas with the BS the light could be sneaking around the baffles.  Robert thinks that light could get inside the cage baffle and reflect around and be hitting and heating the wires.
• All of this seems to say that we should re-implement the soft ASC turn-off for the BS. I had a quick look at the 1/e time for the BS to move after lockloss (it's about 241 seconds), and the 1/e time for the filters (about 240 seconds, despite my quoting in alog 54706 that they were 25 min filters (2*pis are hard!)

To put back the soft turn-off of the BS ASC, I think we need to:

• Disable the BS M1 ASC lockloss trigger.  Jeff reminded me that this would foil my plans, since it turns off the ASC signals to the EUL2OSEM matrix.  This will mean that neither the Pit nor the Yaw BS M1 signals will be shut off by the lockloss trigger.  To disable, we'll need to set H1:SUS-BS_M1_TRIG_ASC_ENABLE to zero (which means that the ASC signals will always be passed to the EUL2OSEM matrix).  I don't think this is in guardian anywhere, so we should only need to change it and then accept in safe and observe snap files.
• Change ISC_DRMI around line 66 such that BS pit gain is not set to zero.  Also, have it turn off FM1 in addition to turning off the input.
• Change ISC_DRMI around line 141 to not hit the BS pit RSET button.

Camilla made the good point that we probably don't want to implement this and then have the first trial of it be overnight.  Maybe I'll put it in sometime Monday (when we again have commissioning time), and if we lose lock we can check that it did all the right things.

I've now implemented this soft let-go of BS pit in the ISC_DRMI guardian, and loaded.  We'll be able to watch it throughout the day today, including while we're commissioning, so hopefully we'll be able to see it work properly at least once (eg, from a DRMI lockloss).

This 'slow let-go' mode for BS pitch certainly makes the behavior of the BS pit oplev qualitatively different. 

In the attached plots, the sharp spike up and decay down behavior around -8 hours is how it had been looking for a long time (as Camilla notes in previous logs in this thread).  Around -2 hours we lost lock from NomLowNoise, and while we do get a glitch upon lockloss, the BS doesn't seem to move quite as much, and is mostly flattened out after a shorter amount of time.  I also note that this time (-2 hours ago) we didn't need to do an initial alignment (which was done at the -8 hours ago time).  However, as Jeff pointed out, we held at DOWN for a while to reconcile SDFs, it's not quite a fair comparison. 

We'll see how things go, but there's at least a chance that this will help reduce the need for initial alignments.  If needed, we can try to tweak the time constant of the 'soft let-go' to further make the optical lever signal stay more overall flat.

The SUSBS SDF safe.snap file is saved with FM1 off, so that it won't get turned back on in SDF revert.  The PREP_PRMI_ASC and PREP_DRMI_ASC states both re-enable FM1 - I may need to go through and ensure it's on for MICH initial alignment.

RyanS, Jenne

We've looked at a couple of times that the BS has been let go of slowly, and it seems like the cooldown time is usually about 17 minutes until it's basically done and at where it wants to be for the next acquisition of DRMI.  Attached is one such example.

Alternatively, a day or so ago Tony had to do an initial alignment.  On that day, it seemed like the BS took much longer to get to its quiescent spot.  I'm not yet sure why the behavior is different sometimes.

Tony is working on taking a look at our average reacquisition time, which will help tell us whether we should make another change to further improve the time it takes to get the BS to where it wants to be for acquisition.