Over the last day and a half the gain of the POP_X Q3 quadrant was not scaled correctly when we went through the reduce 45 modulation depth state. 47759 We don't know how much this was contributing to locking difficulties over the last day and a half.
The guardian stopped setting this gain yesterday, but didn't make any obvious error messages. I was alarmed that we didn't get any errors, and filed a bug report 1137
Jamie called the control room and asked us to check if this was using the fast_ezca script that some of our guardians use, it was and this was allowing the guardian continue to move through the state even though some values which could be important were not written.
I've expunged fast_ezca from ISC_LOCK, and hope we can delete it completely soon.
Once we removed the fast ezca, the guardian didn't keep moving along as though nothing was wrong, but instead stopped and turned maroon. We stopped the node, and then went back to exec, and it was able to connect to the channel.
Dave and Jonathan are investigating why the guardian was unable to connect: 47773