Summary: After being down 11hrs, H1 is back to OBSERVING after locking on 1st attempt post-backing out the change to BS/PR3 from Fri.
In hindsight, I probably should have focused on backing out the BS/PR3 changes much sooner, but wanted to make a few attempts to see if I could make it back as is. But with every attempt at locking taking 30-45min each, making a few attempts to give H1 a chance, running an alignment, and then finally putting time into reading alogs/looking at python scripts/trending channels/etc., it just took me alot of time to finally turn things around flying solo in the middle of the night. Overall, I had (3) locklosses at LOWNOISE_ASC (and we ended up having a total of 6 of these LOWNOISE_ASC locklosses over the ~26hrs).
With that said, and as posted, I finally ended up backing out/loading BS/PR3 changes at around 13:55 (5:55amlocal) & then started relocking.
Sure enough, the path looked brighter when DRMI locked on its own on this first lock. About 35min later, H1 was back to NOMINAL LOW NOISE. There were a handful of SDF diffs (BS & PR3 gains which was expected and also some ETMy pit/yaw offsets we've been seeing lately; see attached).
With this being only the first locking attempt after the changes I made to ISC_DRMI.py, I would like a commissioner to just go over the changes I made to see that they look square and good.
Shoot. This was definitely due to the guardian change on Friday. :(
I had assumed that the BS LOCK P gain was set to -1 somewhere, just as the Y gain must be. But, since P and Y for the BS and PR3 have always been treated differently, there must have been another place that I didn't catch and fix.
We ran all weekend without any BS P offloading to the top stage. BS still had normal angular control on, but for pitch it was entirely on the lower stage, and the low frequency pointing wasn't being offloaded up the chain. As long as there were no BS saturations, this shouldn't affect data quality, I believe.
I will continue to look into this, including finding where the gain for P didn't get set.
I'm not seeing an obvious place where we're saturating the BS M2 stage at lownoise_ASC that would be causing these locklosses. However, we certainly should not be running with the M1 pitch gain off. Maybe I'll re-implement the change tomorrow, and can watch the relock after maintenance to ensure things go smoothly.
This is a good reminder of using and posting the SDF diffs. After reverting my guardian code change, Corey had SDF diffs that he accepted that correctly put the gains back to their nominal values, so they were accepted with the incorrect values over the weekend (although the reason they were incorrect was my guardian error). We are continuing to remove unnecessary diffs to reduce 'alarm noise', so that any diffs that do show up should be a cause for a small amount of investigation.
I have made one more modification to the ISC_DRMI guardian, so that it will clear the history of the pitch filter. Since the line 71 was still commented, FM1 was not going to be turned off (which is what used to do the 25 min clearing). The gain was getting set to 0 (which makes the pitch the same as yaw), but the clear history (near line 143, setting the _RSET channel equal to 2) was only being done to yaw. This meant that when the gains were put back to their nominal values near lines 158, the BS would still have all the integrated output. So, I fixed line 143 so that it will clear history for both pitch and yaw. With Corey's addition of the writing the pitch gains at line 158 (this is the piece that I had forgotten on Friday), now the guardian should work as I had intended on Friday, not taking 25 min to clear the BS pitch.
I have saved and svn-ed the guardian, so next time we're out of observing we should load ISC_DRMI. (The reason to not wait until Tuesday is that I think the code as-loaded will cause relocking problems).