Here is the timeline of what happened this morning at EX (see attached minute trend plot, showing time span 03:00 - 10:00 PDT).

At 03:28 h1iopsusex lost track of its DAC channels, and went into the safe state of not driving any DAC outputs. At this point TMSX and ETMX are not being damped, but the ISI is still operational (red line jumps to 132 in plot). Over the next hour, TMSX slowly rang up and eventually hit the software watchdog limit of 110mV RMS. Ten minutes later the SWWD tripped h1iopseiex DACs (puple line drop to zero) stopping the ISI isolation, which then quietened the suspensions down over the next three hours. When the control room untripped the SWWD at 08:28 the sequence repeated itself. The problem was resolved by restarting the h1iopsusex model at 09:43.

The initial problem of the IOP model losing track of its DAC channels has been seen before, and is more likely on the end station SUS machines due to their increased ADC glitch rate (faster computers).