At 13:03 PDT all h1psl0 models stopped running. I found that one of the 16bit DAC cards was not seen on the pci bus.

We power cycled both CPU and IO Chassis, the missing card reappeared. The IOP model did not run due to the IX Dolphin card in the CPU, but not configured on the fabric *.

Powered down CPU and removed IX611 card. IOP still wont run, its configured to use the card.

Removed pciRfm=1 directive from h1ioppsl0 and installed new IOP model. It started but we had a negative IRIG-B excursion which took about 15 minutes to clear.

Started user models, all is running and Peter verified correct DAC drives on all models.

* a description of how we got here. During the 18 Sept 2018 upgrade of Dolphin from DX to IX, h1psl0 was upgraded along with all the other IPC'ed systems. We then entered the series of Dolphin crashes, which took the PSL down each time, we quickly took the PSL out of the Dolphin fabric(%)by simply disconnecting its IX cable and taking h1psl0 out of the Dolphin manager's configuration. This kept h1psl0 stable and no reboot was necessary at that time. Fast forward to today, and h1psl0 had not been rebooted for 332 days, which is why we are seeing these startup problems today.

(%) In consultation with Keita, Daniel and Peter, the Dolphin IPC was used for ISS third loop PD readout from the end stations, and DBB Jitter transmission to LSC. Neither were needed for O3.