Richard, Jonathan, EJ, Erik, Fil, Dave:

First thing this morning we did a round of h1seih16 power cycling, with Richard power cycling the IO Chassis. The models ran for about 5 minutes this time and then ADC1-timed-out like before.

Then we power cycled the CPU only, and this time the IOP would not run because of a card count issue. And indeed, both showcards and lscpi report that the A2-slot4 is empty, which contains the second 16bit-DAC card.

So we are operating under the theory that all the recent problems have been caused by a flakey DAC1, manifesting as ADC1 timing errors. We are preparing to replace DAC1.

Card 2
    Extender 1
        slot 1: (10ee d8c6) Xilinx Corporation,LIGO Timing Board
        slot 2: empty
        slot 3: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
        slot 4: (10b5 3120) PLX Technology, Inc.,General Standards 16AO16
Card 3
    Extender 2
        slot 1: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
        slot 2: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
        slot 3: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
        slot 4: empty <<< Should be 16AO16 DAC1 <<<<
Card 4
    Extender 3
        slot 1: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
        slot 2: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
        slot 3: empty
        slot 4: empty
Card 5
    Extender 4
        slot 1: (1221 8682) Contec Co., Ltd,Digital I/O DIO-64
        slot 2: (1221 8682) Contec Co., Ltd,Digital I/O DIO-64
        slot 3: empty
        slot 4: empty