Despite some great efforts to track down the source from Jenne and Jeff, we are still seeing a 102 Hz line rung up right at the end of the lownoise_length_control state. Since we had a random lockloss, I asked Tony to take us to lownoise_esd_etmx and I tried walking through lownoise length control by hand (copying the guardian code line by line into the shell).
The lines 5427-5468 ramp various gains to zero, set up the filters and drive matrix for the LSC feedforward, and prepare for the SRCL offset. These lines run fine and do not ring up the 102 Hz line.
I am able to run the first action line of the run state, which sets the MICH FF gain to 1 (line 5480). This runs fine, no 102 Hz line. Then, I ran the next line to turn on the SRCL FF gain (line 5481). This caused an immediate lockloss (huh?), despite the fact that this code has run many times just fine.
On the next lock attempt, I tried running the MICH and SRCL gain lines at the exact same time. Also immediate lockloss.
I have no idea why this is such an issue. All it does it ramp the gains to 1 (the tramps are set on a previous line to 3 seconds).
Both of these locklosses seem to ring up a test mass bounce mode, suggesting that the SRCL FF (I assume) is kicking a test mass pretty hard.
This might be a red herring, or maybe it's a clue. I don't see any 102 Hz line during these locklosses though.
The offending lines:
ezca['LSC-SRCLFF1_GAIN'] = lscparams.gain['SRCLFF1'] * lscparams.dc_readout['sign']
I think it's pretty clear that this is an LSC feedforward problem. I attached two ndscopes of the ETMX L3 master outs, one zoomed in and one zoomed out. The massive oscillation in the signal is the 102 Hz line, which I first begin to see in the time series starting at UTC 5:24:32 and some milliseconds. This corresponds exactly to the time in the guardian log when the LSC feedforward gain is ramped on (see copied guardian log below).
2023-08-16_05:24:22.551849Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.main] ezca: H1:LSC-OUTPUT_MTRX_2_10 => -1
2023-08-16_05:24:22.552715Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.main] ezca: H1:LSC-OUTPUT_MTRX_1_11 => 1
2023-08-16_05:24:22.553143Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.main] ezca: H1:LSC-OUTPUT_MTRX_2_11 => -1
2023-08-16_05:24:22.554026Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.main] ezca: H1:LSC-OUTPUT_MTRX_1_13 => 1
2023-08-16_05:24:22.554332Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.main] ezca: H1:LSC-OUTPUT_MTRX_2_13 => -1
2023-08-16_05:24:22.554746Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.main] ezca: H1:LSC-SRCL1_OFFSET => 0
2023-08-16_05:24:22.555266Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.main] timer['wait'] = 10
2023-08-16_05:24:32.555489Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] timer['wait'] done
2023-08-16_05:24:32.593316Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] ezca: H1:LSC-MICHFF_GAIN => 1
2023-08-16_05:24:32.594884Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] ezca: H1:LSC-SRCLFF1_GAIN => 1
2023-08-16_05:24:32.595219Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] timer['wait'] = 1
2023-08-16_05:24:33.595397Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] timer['wait'] done
2023-08-16_05:24:33.660429Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] ezca: H1:LSC-SRCL1_GAIN => -7.5
2023-08-16_05:24:33.660711Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] ezca: H1:LSC-PRCL1_GAIN => 10.0
2023-08-16_05:24:33.661306Z ISC_LOCK [LOWNOISE_LENGTH_CONTROL.run] ezca: H1:LSC-MICH1_GAIN => 3.2
I have added two new lines to lownoise_length_control that increases the LSC FF ramp time from 3 to 10 seconds. These lines are at the end of the main state, right before the run state. I also increased the timer in the first step of the run state to wait 10 seconds after the FF gains are set, before moving to the next part of the run state which changes LSC gains.
This will result in two SDF diffs for the feedforward ramp times in the LSC model. It can be accepted. Tagging Ops
SDF changes accepted picture attached.
Reverted these changes to make it past LOWNOISE_LENGTH_CONTROL
5471 ezca['LSC-MICHFF_TRAMP'] = 3 Changed back to 3 from 10
5472 ezca['LSC-SRCLFF1_TRAMP'] = 3 Changed back to 3 from 10
And
This may not be a problem with a filter kick, but with the filter making some loop unstable and driving up the 102Hz line. I suspect that changing the aux dofs gain immediately afterwards makes it stable again. If so, slowing down the transition only makes it worse. We may need to reorder the steps.