Over the past couple of acquisitions of NLN, we noticed that the observe SDF diffs took ~25 minutes to read 0 and allow us to flip to the OBSERVING bit. The node in question that is the culprit of this, falls under ASC, screenshot attached below. After further investigation, we noticed that ~7/8 minutes into what seems to be every aquisition of NLN, something causes the ADS signals to drift (when it was originally converging), and therefore causing the ADS to take longer to converge and preventing us from going into OBSERVING. Also noticed that this drift in question, seems to be most apparent on ADS_YAW_3, but this mystery drift does seem to cause a divergence of all degrees of freedom (including the ADS P signals). However, the magnitude of this change does seem to vary depending on the lock. I have attached screenshots of some examples for visual representation.

So far I have checked a couple things and still have yet to find a definitive reason. I have looked at both ISC_LOCK and the CAMERA_SERVO guardian logs during first acquisitions of NLN to see if they are doing any actuation, but both nodes seem to be dormant during this period. Jenne suggested that I also look into the OMC_LOCK guardian to see if the OMC is doing any whitening, as that time period falls around the same time as the drifts take place. I have attached a screenshot shows the guardian state of OMC_LOCK alongside an ADS drift - no state change. After checking some of the OMC_LOCK logs during these drifts, the node seemed to also be dormant during this time (but it does seem to be adding whitening ~10 minutes before the drift in question, not sure if this time frame is suspicious or not - screenshot attached - the drift time occured ~14:45 UTC for that particular occurance).  After doing some alog digging, I saw that this issue has happened before fairly recently - alog here. The issue then was that the REFL signals were way off alignment. After looking into some of ASC signals that feed into the ADS, some ASC signals - particularly PRC 2 Y (and maybe INP1?) seem to also behave in a similar fashion - but I'm not entirely convinced that this is the culprit, since the ASC signals don't seem to move a whole lot. It could also be the case where the ADS drifts are causing PRC 2 to act up as well. But, it might be worth looking into the sensors that feed into these signals (REFL A/B 9I for PRC2 and REFL A/B 45I for INP1). I have also started looking into some LSC signals - particularly the PRCL signals, but have not found anything really suspicious in that initial search.

I do think that this is something we should look at. This issue is preventing us from going into OBSERVING mode for ~15 minutes for every NLN aquisition, which in the long run, is quite substantial when considering total uptime. I will continue to look into this issue and post any updates/findings accordingly.