In consultation with Sheila, I've added a new flag in lscparams.py called "ignore_sqz" (currently set to False) which when True, removes SQZ_MANAGER from ISC_LOCK's managed nodes list and sets up ISC_LOCK such that no requests are made of SQZ_MANAGER. I've additionally removed all instances of ISC_LOCK requesting SQZ_MANAGER to 'NO_SQUEEZING.'

The motivation for this stems from an issue encountered earlier this week (alluded to in Monday's shift summary) where after H1 dropped observing from the SQZ PMC having to relock, SQZ_MANAGER eventually stalled (which is not unexpected) and ISC_LOCK then requested it to 'NO_SQUEEZING' for a then-not-well-understood reason. After looking into the frequently used "unstall nodes" decorator in ISC_LOCK, I learned that the revive function it calls simply requests the stalled subordinate node to whatever its last requested state was. The catch here is that the last requested state refers only to what the manager's last request was to the subordinate, not whatever request a user, other node, or standalone script may have made, as the manager node has no way of knowing about requests outside of its own. A discrepancy between a manager node's last request to a subordinate and a different request to that subordinate can be seen with a notification on the manager saying the subordinate's request changed.

My understanding of the sequence of events that led to the confusion on Monday is that commissioners had been working with the squeezer and wanted its Guardian manager node to remain in 'NO_SQUEEZING' while H1 was relocking following their work. In its design at the time, in a few different states such as 'INJECT_SQUEEZING', ISC_LOCK would check the status of SQZ_MANAGER, and if it was in 'NO_SQUEEZING', move on with the state and re-request SQZ_MANAGER to 'NO_SQUEEZING'. This makes SQZ_MANAGER's manager's last request 'NO_SQUEEZING', so when SQZ_MANAGER later stalled, it was requested back to 'NO_SQUEEZING' even though a user had set SQZ_MANAGER to its nominal state sometime later. This is why I've removed requests to 'NO_SQUEEZING' in ISC_LOCK; I believe it's a sound assumption that if SQZ_MANAGER is in 'NO_SQUEEZING', someone wants it there and Guardian should just move on. Further, this makes it so that the only request ISC_LOCK ever makes to SQZ_MANAGER is its nominal state (except in 'DOWN', of course, which would then later get updated to the nominal), meaning there should be no confusion as to what its last request was.

Changes have been saved and committed to svn, but ISC_LOCK has not yet been loaded. This should be done at the next drop from observing.