After the sus model restarts last night around 21:50 local time, the AI Chassis for h1sush2a and h1sush23 went into a locked state whereby they were not outputting the DAC drives and could not be reset.

We first worked on h1sush2a. We stopped the user models, restarted h1iopsush2a, checked the loopback duotone and saw it was now working. We started the user models and everything was driving correctly so we handed this over to the IMC group.

The we looked around and saw that h1sush34 was also in a bad state. There was less pressure to restore this system so we could do some testing.

First we went into the CER and verified that all 6 AI chassis were disabled by the IOP model.

We stopped all the user models.

First test, start h1iopsush34, reset its DACKILL, start h1sussr2, reset its DACKILL. All looks good, cannot reproduce error.

Second test, start h1iopsush34, DO NIT reset its DACKILL, start h1sussr2, reset its DACKILL, then reset IOP DACKILL. All looks good, cannot reproduce error.

For further information, we started all the models on h1sush34 and all DAC statuses are GREEN. Power down one AI Chassis, DAC0 and DAC1 5th bit went RED, it is a readback of AI status (16 channels per AI, two 18bit DACs per AI).

In Summary, code restarts caused IOP/AI lockout. Restarting IOP models cleared the problem. We cannot reproduce the problem easily. Last night the sus models were restarted at least four times, but only the last restart caused the problem.