Patrick, Richard, Dave:

To prevent Dolphin errors from filling up the /var/log disk partitions, we rebooted (actually power cycled) h1susey and h1seiey. These two were chosen because their /var/log/messages log files had many IXH Adapter error lines, while that of  h1iscey (the third EY Dolphin machine) was just reporting errors from SUS and SEI.

The reboots of SUS and SEI did not fix their errors, so we rebooted h1iscey and that resolved the errors on all three systems. Details:

08:30 PST all models on h1susey, h1seiey and h1iscey stopped.

SUS and SEI Dolphin ports disabled.

IPMI remote power cycle of h1susey, but IPMI not working for h1seiey so it was just warm-booted. Problem persists. (Note: h1susey came back with the wrong system time, off by over 7 hours)

08:40 PST. Richard drove to EY to manually power cycle SUS and SEI, they were prepared for reboot as described above. Problem persists.

While Richard was at EY, I IPMI remote power cycled h1iscey. The last IXH errors on SUS and SEI corresponded with this time. Problem resolved.

h1iscey has a slight negative IRIG-B excursion, after a few minutes this cleared up and DAQ/IPC data was validated.

P.S. h1susey system clock problem only appeared once. I checked all front ends currently have the correct time.