Once we got to observing and ADS was converged, we couldn't go to Observing as the IFO guardian was reporting ISC_LOCK "NOT OK". ISC_LOCK looked okay, it was arrived in NLN and reporting DONE, see attached medm screens and guardian states.
IFO went to Observing on it's own at 05:13UTC. The logs (attached) show that nothing happened in ISC_LOCK at the time IFO got requested to go from WAITING_FOR_NODES to OBSERVE.
The IFO node and ISC_LOCK node were functioning correctly, but we'll need to make a change.
ISC_LOCK was not OK=True (1) because one of its subbordinates was not OK, VIOLIN_DAMPING. The violin guardian finished its 'monitor pause' timer and then transitioned to DAMP_VIOLINS_FULL_POWER at 5:12:58UTC where it then went into its OK=True (1). ISC_LOCK and The IFO node quickly followed and 6 seconds later we were in Observing.
I time machined the Guardian Overview screen to 05:12:57UTC and you could see the violin guardian not OK. This node didn't show up in the IFO list of not ok nodes because it's currently in our exclude list. This clearly creates an issue since it is managed by ISC_LOCK and all of ISC_LOCK's subbordinates need to be OK in order for ISC_LOCK to be OK. I confirmed with Rahul that we don't need to go to the DAMPING_ON_SIMPLE state to adjust modes in full power anymore, so I've taken the VIOLIN_DAMPING node out of the exclude list.
