WP8354 Dolphin EY Switch Replacement
Jonathan, Erik, Dave:
Summary: We have fixed the EY Dolphin problem. The short explanation is that with the later version of the Dolphin code running at LHO the 3rd adapter in h1cdsrfm (the one located at EY) is reinitialized, at which point its Dolphin switch port is unusable (either data only goes in one direction or the port is unavailable). By disabling and re-enabling the switch port normal operation is restored.
Some History: Following the IX Dolphin and long-range-dolphin upgrade of H1 in Sep 2018 we started seeing various types of glitching at a rate of about one a week. As part of the glitch investigation we upgraded H1 to the latest version of the Dolphin software (5.7.0) to facilitate support from Dolphin. After running for 10 months h1cdsrfm was rebooted (following a site power glitch) at which time its EY switch port (port 6) went into one-way-only mode. We moved it to port 4 and fixed the problem. One month later h1cdsrfm was again rebooted (following a h1boot1 crash) at which time its port (port 4) also went into one-way-only mode. We moved it to port 5 and fixed the problem.
This week: Thinking that perhaps the EY switch needs replacing, on Monday we installed a new IXS600 switch at EY and rebooted h1cdsrfm. Its port on the new switch was found to be not working. We put the original switch back in, connected h1cdsrfm to port 5 and found it was now not working. We replaced many hardware items at EY, the h1cdsrfm Dolphin card, its switch port, the cable. We discovered a problem when trying to install both of the spare Adnaco chassis (no link light) and we found that no matter which IXH611 card we put in the original Adnaco chassis it ran significantly hotter than EX (97C with chassis fan not running, 94C after we fixed the chassis fan, compare with 82C for EX).
Tuesday we looked at the logs and discovered that our version of Dolphin was reinitializing h1cdsrfm Adapter2, which was not happening at LLO, presumably due to the code version difference. We also found out that the spare Adnaco chassis most probably did not work because its DIP switch setting was not correct.
Adnaco DIP settings: For an Adnaco chassis to work at an endstation, it needs to be a Long Range version of the board (signified with a 'LR' label on the board itself). It also needs the DIP switch 3 to be set to the non-default setting of OFF, which then matches the DIP settings on the h1cdsrfm adapter PCIe card. The boards on both of our spare chasses had switch 3 switched to ON (the default position).
Today we went back to EY to complete the last hardware swap, install a spare Adnaco chassis with the DIP setting corrected. Inside this chassis we installed the original Dolphin IXH611 card as was in the system from Sep 2018 to Jul 2019.
Here are the details of today's investigation
0. in prep for a changed serial number of the card at EY, changed dishosts.conf and restarted dis_networkmgr
1. disable h1cdsrfm ports and power down computer in MSR
2. At EY, replace Adnaco chassis with the one described above.
3. Power up chassis, power up h1cdsrfm, Link Light!
4. But, same problem. dis_diag suggests EY dolphin is not connected to switch, but this is false as all LEDs are nominal.
5. Somewhat in desperation we moved to a different switch port (like what was done Jul, Aug even though the port failure mode is different). Disabled port 5, moved cdsrfm cable port 5->8
6. dis_diag now sees all nodes at EY. We're seeing a pattern.
7. To see if this persists following a h1cdsrfm reboot we disabled the ports and rebooted h1cdsrfm
8. BAD, we forgot to change ixnodetab to the new port, crashed all EY models
9. dis_diag now does not see EY nodes from port 8.
10. try the port swap one more time, disable port 8, enable port 5, move cable 8->5.
11 nodes are again seen. So disable/enable sequence is needed, but is a cable disconnect/reconnect needed?
12 to test this, disable h1cdsrfm ports (this is port 5 at EY), reboot h1cdsrfm
13 At this point EY nodes are not seen, so disable EY port 5 and then enable it, keeping the cable inserted
14 all nodes are now seen at EY. We have a work around.
15 Restart all the EY models before starting the cdsrfm code. BAD, accidentally restarted h1susex first.
16 after all models are running, started the long-range-dolphin code on h1cdsrfm, all IPC running again!
This work has been a learning experience with several positives take home messages:
New h1cdsrfm restart procedure: until we can fix the Dolphin reinitialization problem on Adapter 2, a disable/enable following reboot for the EY switch port permits IPC data to work correctly.
Spare Adnaco chassis: finding the DIP setting issue now means we our spare chassis can be used at the end station
Adnaco chassis running hot: the original EY chassis (board SN C8610323) was running cool in Jan 2019 (82C) and hot this week (94-97C). We need to investigate why this is before using this as a spare chassis.