Reports until 22:43, Monday 14 October 2019
H1 CDS
david.barker@LIGO.ORG - posted 22:43, Monday 14 October 2019 - last comment - 08:09, Tuesday 15 October 2019(52472)
summary of hardware swaps we tried to get EY RFM Dolphin working again

Jonathan, Erik, Dave:

Following the replacement of the EY Dolphin switch with bad ports with a new switch, and after finding problems with the new switch re-installing the original switch only to find the RFM EY Dolphin card was not being activated, here are the hardware investigations we tried to restore the Y arm RFM (note that when swapping Adnaco chassis we reused the single-mode SFP and fiber).

(Starting with the identical hardware the system had before today's work, but now after rebooting h1cdsrfm its EY card is not being activated but is seen on the PCI bus)

Instead of just rebooting h1cdsrfm we power cycled the computer and the EY Adnaco chassis. Subsequently this was also done several times as we tried different chassis. 

We did a 'big boot' and powered h1cdsrfm and both EX and EY adnaco chassis.

We noted EY was running hot, and the chassis fan was not working. After discovering that the fan power cord was loose, and was fixed by reseating it. After this EY was still running hotter than EX.

We replaced the Adnaco with a spare from the DTS (equiped with the orginal IXH611 card from sep2018 through to jul2018).  The fiber link did not work.

We tried the second spare Adnaco from the DTS equiped with a brand-new out-of-the-box IXH611 card (we remembered to set the dip switches), again no fiber link.

We put the original Adnaco back (the one whose link works) now with a fixed fan and equiped with the brand-new IXH611.

We used h1iscey's switch port and X8 cable.

Final configuration at the end of the day is original Adnaco with new IXS611 card. We found a mismatch in the card serial number and the dishosts.conf and ixnodetab files, but fixing this did not change EY.

Some items to note:

Most of the times after we took h1cdsrfm out of the Dolphin fabric to reboot/power itr there were no problems with other system, except in two cases where all the EY models crashed. I would estimate 6 good restarts to each crash restart.

On one of these crashes, h1iscey could no longer see its IO Chassis. After a full system power cycle this was restored.

Comments related to this report
david.barker@LIGO.ORG - 23:28, Monday 14 October 2019 (52473)
Link

The CDS system as we have left it overnight:

local Dolphin working at all three locations (CS, EX and EY). Looks like h1sustmsy still has its watchdogs engaged.

long range dolphin not working, giving various IPC errors.

It appears that h1susauxh56 crashed after we left this evening? I'll restart it tomorrow.

Various CFC flags are showing following Jenne and Jim's make-install for new isi-ham, seiproc and oaf model changes.

keith.thorne@LIGO.ORG - 08:09, Tuesday 15 October 2019 (52474)
Link 

I used /opt/DIS/sbin/dis_diag remotely to figure out what it going on.
h1iscey - OK - Switch port 3 - sees ISC,SEI,SUS not RFM card
h1seiey - OK - Switch port 2 - sees ISC,SEI,SUS not RFM card
h1susey - OK - Switch port 1 - sees ISC,SEI,SUS not RFM card
h1cdsrfm - Sees 3 cards - CTR, X OK, Y-end NOT OK - sees only RFM card

-- Behavior acts as if either cable from Y-end adapter to switch is not OK, or port on Y-end Dolphin [table says #6] is disabled. I tried a manual port enable (by running /etc/dolphin_ix_port_enable.sh on h1cdsrfm), but no change.

* I would suggest moving h1cdsrfm at Y-end to another port.  Ports are enabled by default.  If that works, then later update 'ixnodetab'.