The seismic computer at EX froze (see attached picture). It was rebooted
along with the AI chassis and IO chassis.
Dave Barker was notified.
Richard / Peter
Richard power cycled h1seiex in the order: CPU power down, IOChassis power down, AI chassis power down, IOChassis power up, CPU power up, AI chassis power up (The AI chassis power cycle is to prevent problems when the 16bit DAC sends voltage when the unmanaged IOChassis is powered up).
I attempted to remotely take h1seiex out of the Dolphin fabric using h1iscex, but this failed. When h1seiex was powered back up, this glitched the other dolphin'ed models at EX. This required a restart of all the models on h1susex and h1iscex as well as h1seiex.
The order of model restarts was : kill all models on h1susex, h1seiex, h1iscex, then turn on all models in the same order (using /etc/model_kill.sh and /etc/model_start.sh to ensure correct local order). We had no timing excursions from the IOP models.
All IPC errors were then cleared (global DIAG_RESET). Corey reset the SWWD-EX. The accumulated DAQ CRC counts were cleared. User models' watchdogs were reset which cleared the DK status in the state words.
The CW excitation bit became good, however someone in the Hardware Injection team should verify CW excitations are running correctly.
Restarts were completed at 07:25 PDT (14:25 UTC)