Jeff K, Sheila, TJ, Jim
Jim had two more locklosses at engage ASC while trying to relock after this afternoon's lockloss. In one case, there aren't many clues except for the usual error signals deviating from 0: 1252450014
In the earlier one 1252449154, the lockloss happened after all the ASC was on and the first convergence checker had passed. At that time we increase the gain of some loops and turn off the offsets in AS36Q which are set in DRMI. I noticed that we have a 4 second wait for a 10 second ramp on the offsets, so I set the wait time to be the TRAMP for the offsets.
The attached ndscope for this lockloss shows both CHARD and INP1Y hitting the soft limiters in the 5 seconds before the lockloss. We use the soft limiters to allow us to turn on all the ASC loops at once, in principle we would like to not have soft limiters on our slowest loops, and only limit those which move fast so that the slow ones can keep up. We haven't taken the time to figure out which loops actually need the limiters, so we have them on all the loops that get engaged here. This means we are probably unnecessarily limiting the ability of our slower loops to keep up in some cases, so eventually we probably want to be more selective about which soft limiters we are using.
The soft limiter is only meant to help us when we are first engaging the loops. Once we have all the loops on and they have converged once, they shouldn't be on anymore since all they will do is make slow down the ASC system's ability to respond to disturbances. In this lockloss, INP1 + CHARD Y hit their limiters after everything was converged (I am not sure what caused a disturbance). I changed the guardian code so that the soft limiters will be turned off after the first convergence check.
Jeff is looking through some of our recent locklosses from ENGAGE_ASC to see if this could potentially have helped us avoid other similar locklosses.
?
