david.barker@LIGO.ORG - posted 07:39, Friday 14 December 2018 - last comment - 13:59, Friday 14 December 2018(45929)
h1sush2b has lost connection with its IOC Chassis
Richard and I are power cycling the system now.
The IO Chassis is visible and the models are running, but the IOP has a temporary negative IRIG-B excursion. This should clear in a few minutes.
Turns out this was a messy recovery. After the IRIG-B came good again, I noticed the user models had not in fact started correctly and were reporting that they had no IOP model. I restarted the user models (h1sus[im, htts]) and now they are running correctly.
h1sush2b was recently upgraded to a V4 computer (Sep 2018), and in the process it got a new One Stop fiber (2011 and older fibres do not work on V4s). I seem to remember we had IO Chassis issues with this unit before, so with a new fibre the problem may be in the One Stop PCI/PCIe cards.
DIAGS and CRCs have been cleared, h1sush2b is ready to drive its DACs.
More problems:
IOP continued to have a DAC error. Further investigation shows that lcpci is only seeing one 18 bit DAC card, there should be two. We are doing one more power cycle of CPU and IO Chassis.
After power cycle we now see two 18bit DAC cards. Richard confirms he saw the IOP report two cards on the first power cycle. We are carefully monitoring the DAC count now.
Again we lost one of the 18bit DACs once everything got going and the IRIG-B errors cleared. We are going to replace both DACs.
system is stable now with its two new 18bit DACs. I'll test the ones removed on the DTS.
FRS: ticket 12003
18bit DAC Card Details:
Note that it is not possible to know which specific card was replace with which.