The short version: There have been some Resonance locklosses recently with simultaneous BS ISI Stage 2 trips, I think the ISI trips are a symptom, not a cause. We could avoid the trips by possibly moving the BS Stage 2 loops later in locking, or it might be possible to adjust SRCL to make it more stable. It would help diagnosing this if we could collect a SRCL open loop measurement while pausing at Resonance the next time we lock.

The blah blah blah: Over the last week or so there have been roughly 4 locklosses at Resonance that have had simultaneous trips of the BS ISI Stage 2. I have looked at a couple of these locklosses and it looks like SRCL might be going unstable, causing the locklosses. Because SRCL and MICH are coupled, this is causing trouble for the BS ISI. According to the lockloss page there have been about a dozen total locklosses at Resonance since Nov 1 of last year. Many of them are associated with BS M2 and SRM M3 drives ringing up.

First attached plot shows the LSC lockloss ndscope for one of the locklosses recently where the BS ISI ST2 tripped. MICH and SRCL see something ring up at ~82 hz before the lockloss.

The second attached plots are the drives for the BS ST2 for the most recent lockloss/trip, but every case I've looked at so far looks pretty similar, due to the SUS BS drive pushing on the ISI BS Stage 2. I've also seen locklosses where this same ring up happens in MICH and SRCL, but the BS Stage 2 doesn't trip. At this point I don't think the ISI is causing the locklosses, the trips are just a symptom of a particular kind of lockloss.

Sheila has suggested that optical gains are changing rapidly at this point, and was ready to believe that maybe something in SRCL in particular might be the culprit. When we get a chance, during the next re-lock she suggested we try measuring the open loop gain with the DTT template here: /opt/rtcds/userapps/release/lsc/h1/templates/SRCL/SRCL/SRCL_OLG_NOISE_FULL_LOCK.xml

Another solution would be to push turning the BS Stage 2 loops on later in the process. The third attached plot shows the BS ISI Stage2 V2 drives for 2 ISI trip/locklosses and 1 successful lock acquisition. For each subplot, the blue trace is the V2 drive, the red trace is the ISC_LOCK state *10. For each lock, after DRMI asc comes on the SUS BS is pushing pretty hard on the ISI Stage 2, increasing the V2 drive in response. At Resonance (where the red trace on the upper 2 plots show a lock loss), each lock shows the ISI drive increasing a bit more, ringing up and causing a trip for the top two plots. The bottom plot was a successful lock, and after Resonance the BS Stage2 drives calm down a lot. If we could wait until after DRMI on POP to turn on the Stage 2 loops, we would run much less risk of tripping the BS ISI. There were some foggy recollections of trying this in the past and it was too disruptive to turn on the loops later, but no one was very sure if that was true.