I spent the morning mining the seemingly random locklosses from the weekend. Then Sheila pointed me towards the ndscope useful channels list that Jenne made - indeed the dither system (ADS) wanders off in many cases.
I don't know what they mean, but the lockloss tool shows the following:
39: 2019-09-01_12:13:59Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-2 sec before lockloss, SRC channels wander at -2 sec.
H1:OAF-RANGE_RPL chans all WANDER at -7 sec.
38: 2019-09-01_16:15:40Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-10 sec before lockloss, SRC channels wander at -1- sec.
No DARM glitch in the OAF RANGE RPL chans.
35: 2019-09-01_20:19:04Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
~WINDY - ends at ~16-20MPH
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-3 sec before lockloss, SRC channels wander at -2 sec.
ADS PIT and YAW DOFs and DEMOD chans WANDER 10 sec before LL.
H1:OAF-RANGE_RPL chans all WANDER at -5 sec.
30: 2019-09-02_03:19:04Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CSOFT_Y_OUT_DQ starts to really wander off t-6 sec before lockloss, SRC channels wander at -5 sec.
ADS PIT and YAW DOFs WANDER 8 sec before LL.
H1:OAF-RANGE_RPL chans all WANDER at -7 sec.
24: 2019-09-02_13:10:16Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-5 sec before lockloss, SRC channels wander at -4 sec.
ADS PIT and YAW DOFs WANDER 8 sec before LL.
H1:OAF-RANGE_RPL chans all WANDER at -9 sec.
19: 2019-09-03_01:52:57Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CSOFT_Y_OUT_DQ starts to really wander off t-8 sec before lockloss, SRC channels wander at -10 sec.
ADS PIT and YAW DOFs WANDER 12 sec before LL. H1:OAF-RANGE_RPL chans all WANDER at -12 sec.
13: 2019-09-03_10:17:41Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:SUS-ETMX_L3_MASTER_OUT - glitch 1 sec before LL, but other glitches of same size/shape earlier not an issue... SRC/SOFT/HARD loops loop nominal in seconds prior to LL.
ADS NOMINAL before LL.
No DARM glitch in the OAF RANGE RPL chans.
________________________
on command line:
userapps
lockloss -c ADS_3_4_5.yml select (to show ADS)
lockloss -c darm_blrms.yaml select (to show glitch info on DARM, not sure what math is happening but large wander from -10sec on H1:OAF-RANGE_RPL_1_OUTPUT and all other similar chans, seems there was a glitch in DARM which maybe saturated the Dither line in DARM so that ADS couldn't work correctly... or something like that in commissioning person words...)
So, is it the old story that a glitch causes havoc in with the ADS - and does it seem worse this weekend? Dunno...
Here are some plots of the locklosses caused by a DARM glitch sending a large signal to ADS, and the rest of the ASC loops not holding their error signals as the ADS moves the IFO around. This is the lockloss labeled 19 in Betsy's log, 1251510796
This seems like something that we could add a tag for in the lockloss tool, for example if we are in low noise and the ADS error signals go above some threshold we could tag it as an ADS glitch lockloss.
Today Keita and I increased the low frequency gains in PR2, SRC1+SRC2 P+Y loops, hopefully this will help the ASC signals stay closer to zero in future events like this 51715. It seems as though we may also need to increase the INP1 gain.
We could also avoid having these glitches impact the ADS by moving the dither lines to lower frequencies where they wouldn't be drowned out by the glitches. That caused excess noise in DARM when Georgia tried it, but Matthew Ball has been doing some modeling that suggests that a frequency dependent A2L decoupling might help allow us to move them down in frequency.
The last and probably best solution would be to get rid of the glitches, of course.