Ed, Jeff, Fil, Dave:
I'm not sure if this is a coincidence, but the h1iopsush2a IRIG-B excursion had almost completed its recovery (after almost 3 hours) when the IOP glitched again. The IRIG-B had gone up to about 1500 and was down at 75 at the time of the glitch (24 is the start of the good range).
This time all user models showed an FE error (see attachment).
I stopped all the models and ran h1iopsush2a by itself, verifying there was not an IRIG-B error.
As an unconnected problem, I noticed that the second 18bit DAC AI chassis was reporting an ON status even though the IOP was commanding all AI's to be OFF (see attachment). Jeff verified the AI rack locations, and it was found that the second 8-channel block of the AI was permanently ON, even with the DAC cable disconnected from the rear. Fil replaced the AI chassis with a spare, the switching function has been restored.
Around this time h1sush2b glitched. Its models were restarted with no problems.
I restarted the remaining models on h1sush2a (h1susmc[1,3], h1suspr[m,3]).
Attached plot shows a 6 hour minute trend of h1iopsush2a's IRIG-B value (red) and its STATE_WORD (black). It can been seen that the STATE_WORD is in its good state (value=0) for most of the IRIG_B excursion, and fails close to its end.
Failure of h1sush2a is associated with FRS Ticket 11222. Identification and fix of AI chassis is associated with FRS Ticket 11223.
AI chassis S1108081 replaced with S1104370.