Here is an overview of how the h1oaf0 problem is presenting and what has been tried so far:

Problems started when we added a 7th ADC for PEM expansion.
Initial ADC card was a PMC card on PMC-2-PCIe adapter.
An old style adapter was used, computer would not attempt to
boot (no bios screen) if the chassis was powered.
Eventually a pci-e ADC was installed as the 7th card.
h1iopoaf0 was recording ADC/DAC errors at random times.

Investigating the proc files, each event is an ADC/DAC timing error.
ADC records a very high adcHoldTimeEverMax (>90) which is recoverable.
16bit DAC records fifo_status of 0 (not first quarter) and is not recoverable.

----------------------------------------------------------------------------------------------
Thu 11/10:

Removed 7th ADC from chassis, reverted h1iopoaf0.mdl model back to earlier version.
Reseated and screwed down all cards.

----------------------------------------------------------------------------------------------
Fri 11/11:

09:55PST
replaced DC power supply in chassis.
Reseated ADC, DAC, BIO and interface cards.

Replaced IO Chassis with x1psl0. onestop/BIO/ADC/DAC/interface/ribbon-cables transfered over.

Timing, OneStop came with new chassis.

Replaced 1st ADC set (ADC+ribbon+i/f) from x1psl0.

----------------------------------------------------------------------------------------------
Mon 11/14:

went back to original chassis.

installed new one stop card in chassis (original card had problems with heat-sink)

swapped first and second ADC card sets (ADC+ribbon+i/f)

replaced 18bit DAC with modified card

replaced SFP on fanout, and returned to second slot. Replaced SFP on timing slave.

pull power cords out of h1oaf0 to get it to boot.

Replaced long run one-stop fibre between computer and chassis

We are currently testing if the fiber change has helped, has been running for 4 hours so far.

Attached plot shows last 14days trend of h1iopoaf0 STATE_WORD, showing the ADC+DAC+DK+OVF events.