Reports until 12:33, Wednesday 24 July 2019
H1 CDS
david.barker@LIGO.ORG - posted 12:33, Wednesday 24 July 2019 - last comment - 13:27, Wednesday 24 July 2019(50786)
Long range dolphin operational

Richard, Fil, Rolf, Dave:

Summary: long-range Dolphin IPC data is flowing again from EY to CS  after we moved its Dolphin switch port.

Details:

Here is a summary of what we tried to get EY->CS data flowing over the past 14 hours:

rebooted h1cdsrfm

power cycled h1cdsrfm and both EX,EY Adnaco expansion chassis

power cycled EY IXS600 Dolphin switch

Checked fiber optic power to/from EY Adnaco SFP

Replaced IXH611 Dolphin card at EY with spare

Moved cdsrfm switch port from port 6 to port 4 (Fixed problem)

I'm in the process of changing the dolphin configuration to follow the port move.

Comments related to this report
jeffrey.kissel@LIGO.ORG - 12:43, Wednesday 24 July 2019 (50787)
Link 

The dolphin network was fixed (by switching the port into which the problematic card was plugged to another) by 2019-07-24 19:20 UTC.
keith.thorne@LIGO.ORG - 13:27, Wednesday 24 July 2019 (50788)
Link 

I suspect the root cause is that the end-station Dolphin switch got 'wedged' due to power glitch.  In this case it can be in a state where you can't re-enable ports.  I had this happen on the LLO DAQ test stand See TST log 12353, TST log 12338.  The cable swap work because it was to a port that had not been disabled previously (default state is enabled)