J. Kissel, J. Batch, H. Radkins, H. Paris

We discovered this morning that HAM2&HAM3 ISIs and HPIs (run on the h1seih23 [computer /  I/O chassis]) were not sending drive signals out to the chambers even though 
- Drive was requested
- The ISI & HPI overview screens indicated the ability to drive (Watchdogs all green, master switch on, "you can drive" border all green)
- The user model FEC state words were all green
- The user model GDS_TP DAC bits were all green

It turns out the only indication of badness is on the FEC STATE WORD of the IOP (in this case H1:FEC-53_STATE_WORD) -- the 8th bit (or 7th if you count from 0), marked "WD" on the CDS_OVERVIEW screen, which is a mislabeled bit representing the "DAC ENABLE" function, which can be triggered by several things other than watchdogs. A minute trend of this bit shows that it has been triggered since Dec 15th 2013 12:51 PST, 2013-12-15 20:51 UTC, 1071175876

Hugo informs me that this symptom has been seen before (I'm sure there are aLOGs, but they'd be written by four different people, each, like me, unsure of how to explain the problem exactly, and probably tagging it with different tasks or use different buzzwords), but the cause is unknown.

Jim had initially suspected the the IOP's FIFO buffer had somehow been emptied, which would cause this bit to go bad, but after looking at the front-end's log, by doing
controls@opsws3:~ 0$ ssh h1seih23
controls@h1seih23:~ 0$ dmesg
where he expected to see some error from the h1iopseih23 process indicating this fact, but no obvious error was reported.

As with all problems like this, we rebooted all front end processes and it cleared the problem and actuation ability has been restored, but the root cause remains a mystery.

@DetChar: can you take a look? If anything just to narrow down exactly when the IOP bit failed *this time*, but maybe get a long history of he channel to see how often this happens, if it only happens on this front end or if it happens on other front ends, etc. etc.

@SEI: Y'all should include the STATE WORD for the user model and the IOP on your overview screen, and include this DAC ENABLE bit in your logic for computing the "you can drive" border.