TLDR - Looks llike it was a race condition between during a lock loss and a timer never getting reset. Put in some code to hopefully get around this.
Recently we've seen IFO_NOTIFY alerting as soon as we get to the state NOMINAL_LOW_NOISE on a few locks, rather than waiting for the camera servo timer or the nln timer. Ryan S and I took a look at it last week and didn't find the issue, but left some more log messages to help us debug. These messages showed that the timer for nln was getting reset at a lockloss. The only way for this to happen is if during a lock loss the IFO node went out of its Observing state before ISC_LOCK went out of NOMINAL_LOW_NOISE. This happened twice last night because of the ITMY ST2 trips (alog72666), but looking at slow channels for other times make it difficult to confirm.
I added a reset for the timer to some absurdly large number when ISC_LOCK is in DOWN. This should stop this past situation and let us use IFO_NOTIFY during commissioning times if we wanted. I loaded it into the node, hopefully this is the fix.