Stefan, Sheila

We had a lockloss last night during the INCREASE_POWER state when the ISS third loop was coming on.  This is done by a function that had a ten second sleep in a while loop, so the guardian didn't recognize the lockloss for about 90 seconds, in which time ETMY and ITMX suspesnions tripped.  Stage 2 of both ISIs tripped after the suspensions had been tripped for about an hour.  Screenshot shows the sequence of events, first a 90 minute time span then in a 2 minute time span around the lockloss. 

To make sure that in the future the guardian will recognize the guardian more promptly, we moved the ISS third loop engagement to the IMC_LOCK guardian which currently handles the ISS second loop engagement.  The ISC_LOCK guardian just requests this now in the increase power state when it reaches 10 Watts.  

We also looked at why the lock broke.  One potential explanation is that there are large glitches in both DHARD P and Y when the boosts get engaged about 3 seoncds before lockloss (screenshot attached).  These are simple single pole single zero boosts, which were Always on with a 3 second ramp for engaging.  We changed the ramps to 0.2 seconds to see if by ramping faster than any features in the filter we can avoid glitches.  The real cause of the lockloss might have been changes to the offloading that weren't compatible with the guardian as it was last night, but in any case we would like to avoid these large glitches when engaging the boosts.