I previously noted a glitch about 30 seconds before lockloss in LOWNOISE ASC, 85685. However, we had two more locklosses from this state last night and I do not see such a glitch so that is a random coincidence. One of those locklosses appears to be caused by an earthquake. However, since 6/11, we have had 9 locklosses in this state that occurred exactly 47 seconds into the state, which seems suspicious, one of those occurred last night, and the lockloss with the glitch was the same.
This seems to be coincident with the engagement of a few DHARD P filters:
2025-07-14_14:26:54.641531Z ISC_LOCK executing state: LOWNOISE_ASC (522)
2025-07-14_14:26:54.642230Z ISC_LOCK [LOWNOISE_ASC.enter]
2025-07-14_14:26:54.655894Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_PIT3_OSC_CLKGAIN => 300
2025-07-14_14:26:54.656325Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_PIT4_OSC_CLKGAIN => 300
2025-07-14_14:26:54.656732Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_PIT5_OSC_CLKGAIN => 300
2025-07-14_14:26:54.657043Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_YAW3_OSC_CLKGAIN => 300
2025-07-14_14:26:54.657438Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_YAW4_OSC_CLKGAIN => 300
2025-07-14_14:26:54.657892Z ISC_LOCK [LOWNOISE_ASC.main] ezca: H1:ASC-ADS_YAW5_OSC_CLKGAIN => 300
2025-07-14_14:26:54.658134Z ISC_LOCK [LOWNOISE_ASC.main] timer['LoopShapeRamp'] = 5
2025-07-14_14:26:54.658367Z ISC_LOCK [LOWNOISE_ASC.main] timer['pwr'] = 0.125
2025-07-14_14:26:54.783581Z ISC_LOCK [LOWNOISE_ASC.run] timer['pwr'] done
2025-07-14_14:26:59.658298Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:26:59.719537Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y_GAIN => 200
2025-07-14_14:26:59.720456Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y_SW1 => 256
2025-07-14_14:26:59.846249Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y_SW2 => 20
2025-07-14_14:26:59.971686Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_Y => ON: FM3, FM8, FM9
2025-07-14_14:26:59.972384Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_Y_SW1 => 5392
2025-07-14_14:27:00.098073Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_Y_SW2 => 4
2025-07-14_14:27:00.223528Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_Y => ON: FM1, FM3, FM4, FM5, FM8
2025-07-14_14:27:00.224135Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CSOFT_P_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.224497Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CSOFT_Y_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.224868Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_P_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.225188Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_Y_SMOOTH_ENABLE => 0
2025-07-14_14:27:00.225433Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:10.225728Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:10.281803Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_TRAMP => 5
2025-07-14_14:27:10.408120Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_SW2 => 16
2025-07-14_14:27:10.533563Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P => OFF: FM9
2025-07-14_14:27:10.534285Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_SW1 => 256
2025-07-14_14:27:10.660088Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_SW2 => 4
2025-07-14_14:27:10.785453Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P => ON: FM3, FM8
2025-07-14_14:27:10.786315Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-CHARD_P_GAIN => 208
2025-07-14_14:27:10.786535Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 5
2025-07-14_14:27:15.786858Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:15.847152Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_Y_TRAMP => 5
2025-07-14_14:27:15.847580Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_Y_GAIN => 5
2025-07-14_14:27:15.848666Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DSOFT_P_GAIN => 5
2025-07-14_14:27:15.848917Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 5
2025-07-14_14:27:20.849050Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:20.906577Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMX_M0_DAMP_Y_TRAMP => 10
2025-07-14_14:27:20.907206Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ETMX_M0_DAMP_Y_TRAMP => 10
2025-07-14_14:27:20.907700Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMY_M0_DAMP_Y_TRAMP => 10
2025-07-14_14:27:20.908422Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMX_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.908830Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ETMX_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.909148Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ITMY_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.909562Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-ETMY_M0_DAMP_Y_GAIN => -0.5
2025-07-14_14:27:20.909789Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:30.910055Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:30.968166Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_P_GAIN => -0.2
2025-07-14_14:27:30.968527Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_Y_GAIN => -0.2
2025-07-14_14:27:30.968806Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_L_GAIN => -0.2
2025-07-14_14:27:30.969073Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_R_GAIN => -0.2
2025-07-14_14:27:30.969343Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_T_GAIN => -0.2
2025-07-14_14:27:30.969606Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:SUS-SR2_M1_DAMP_V_GAIN => -0.2
2025-07-14_14:27:30.969838Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:40.970003Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] done
2025-07-14_14:27:40.972085Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_P_SW1 => 1024
2025-07-14_14:27:41.097962Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_P_SW2 => 4
2025-07-14_14:27:41.223313Z ISC_LOCK [LOWNOISE_ASC.run] ezca: H1:ASC-DHARD_P => ON: FM4, FM8
2025-07-14_14:27:41.223637Z ISC_LOCK [LOWNOISE_ASC.run] timer['LoopShapeRamp'] = 10
2025-07-14_14:27:41.593743Z ISC_LOCK [LOWNOISE_ASC.run] Unstalling IMC_LOCK
2025-07-14_14:27:41.765955Z ISC_LOCK JUMP target: LOCKLOSS
I will take a look and see if there is anything unstable about these filters. Whatever is occurring seems to be too fast to be seen in the ASC signals themselves, and at first glance I don't see anything strange in the suspension channels either.
DHARD FM4 is engaged with a 10 second ramp- this is a change I made on 6/11: 84973 because we had lost lock on that day twice in the same spot. Two of the locklosses at 47 seconds occurred before that change. Then, later that day on 6/11 I reengaged a boost in DHARD P, which only has a 5 second ramp, 84980. Engaging that boost shouldn't be unstable, but maybe something bas occurrs when they ramp at different times. I'm lengthing the ramp to 10 seconds.
We had another lockloss from this state at the 00:47 mark last night, 1436615757 so I'm not sure this fixed the problem.
However, the lockloss was proceeded by a glitch about 30 seconds before, like another lockloss I noticed in this state. This could be coincidence again, but it's looking a little suspicious!
The glitch appears to be occuring due to the CHARD P change. We ramp a boost off with 2 seconds, and a new shaping and low pass on with 2 seconds, and then change the gain with 5 seconds. Looking at the step response of the shaping and lowpass filter, this ramp should probably be 10 seconds, and the gain also 10 seconds to match. I will keep the boost at 2 seconds to ramp off though. I increased the wait timer to 10 seconds to match this ramping. Model and guardian changes saved and loaded.
I am still not sure what is going on with DHARD P, but as a test I've now separated the low pass and loop shape from the engagement of the boost, since we know those individually are stable to engage. We now engage FM4 with a 10 second ramp and wait time, then engage FM8 with a 5 second ramp and wait time. I edited the ramps and gaurdian code to do so, svaed and loaded. This is kind of annoying, but it might help me debug what's going wrong here.
I watched the signals during lownoise ASC, and this time I saw no glitch in CHARD during its lownoise transition. However, I saw a glitch when the DHARD P FM4 filter was engaged, and no glitch when FM8 was engaged. Maybe the ramp of FM4 should be even longer than 10 seconds. I increased the filter ramp to 15 seconds and increased the guardian wait timer to match. Both changes saved and loaded.
We haven't had a lockloss in this state since this fix (but we've had plenty of locks), so I am going to declare this problem fixed!