The h1susex models stopped running at 13:45 UTC (06:45 PDT) this morning (Sunday 27 Oct). This is a repeat of the last week's problem, in which h1susex appears to have lost an 18bit DAC (1-7 in that case, which was replaced with a spare).
I've stored all the system and Dolphin logs.
Looking at the PCI bus scan, there is a strange double entry for the 1-7 slot. Perhaps a board fault on this slot since this there is a new 18bit DAC card located there now?
Remember that even after replacing the suspect 18bit DAC, we got an 18bitDAC error on the first time the IOP model was started. I tested the DAC which was removed from h1susex on the DTS and it was not seen on the bus, suggesting it really was damaged.
Here is the alog from Wednesday afternoon when we replace the 1-7 18bitDAC on h1susex alog52664
FRS13748 had been opened Wednesday to cover the h1susex 18bitDAC issue, I'm adding today's details to it.
Richard and I have discuss this problem on the phone, we are going to avoid using slot 1-7 and we hope the DAC card currently in that slot is still good, if not we will use a second spare. I'm planning how to reshuffled the cards to avoid this slot with the minimum disruption in the ribbon cabling.
Richard is on site. We have a card shuffle plan, h1susex is powered down, Richard is making the change to the chassis initially relocating the current 18bitDAC card which was in slot 1-7.
Referencing the IO Chassis layout shown above (it is slightly out-of-date, there is a second ADC in slot 1-8 and the 18bit DAC in slot 2-5 is now a 20bit DAC)
We are:
- moving 18bit DAC from slot 1-7 to 1-2 (was the 2nd DAC, becomes the 1st DAC)
- move ribbon cable from 1-6 to 1-2
- move ribbon cable from 1-7 to 1-6
This leaves the suspect slot 1-7 empty, using 1-2 instead. DACs in 2-x slots are unaffected.
Bad news, the 18bit DAC which was in slot 1-7, now relocated to 1-2 is not seen on the bus, presumably it has been damaged (will test on DTS tomorrow). Richard is now installing a spare 18bit DAC into slot 1-2.
suggests slot 1-7 damages cards installed in it.
Spare 18bit DAC in slot 1-2 is seen, we have a full compliment of cards.
As a sanity check that we have the correct DACs wired, I'm going to run the Duotone DAC-ADC loopback.
BTW, h1iopsusex came back with a negative IRIG-B error, should clear in a few minutes.
h1susauxex had a timing glitch while we were working on h1susex's IO Chassis. I restarted the models and it is good now.
IRIG-B timing is good again. I did a quick DAC duotone loopback test (last channel of first DAC outputs the duotone signal, which instead of going out to the field is looped back to the second to last channel of the first ADC). Timing value on IOP-TP MEDM showed a steady measurement of 70uS, which went back to a noise signal when I turned the DT test off.
Richard is damping SUS-EX and recovering SEI-EX from its watchdog trips.
Thank you Dave and Richard!
I have logged the latest h1susex dmesg. The IOP model is not reporting any issues, the AUTOCAL of the 18bit DACs and the 20bit DAC looks good.
It certainly looks like h1susex IO Chassis slot 1-7 is broken to the point that it damages cards. This will ultimately be resolved when we upgrade h1susex to the new IO Chassis, which presumably we could think about doing early if we have further problems with this IO Chassis (given the ease of running MTP fiber at the end station).