It appears that the RFM loop on the managed hub (VMIC 5595) is in the opposite way to which we imagined, from the higher numbered port down to the lower numbered port. Because we put the loop together "backwards" it turned out that for the IOP watchdog on the SUS ETMY frontend to communicate with the SEI BSC6 front end less than a foot away phyiscally, it had to run through 8km of fiber because we extended the loop to the LVEA on Tuesday this week. To run through 8km of fiber optics cable takes about 40uS. The IOP runs at 64KHz, with a processing loop taking about 15uS. So not surprisingly the SEI BSC6 IPC receiver was giving 65538 error per second.

The EY ISC was also going through 8km of fiber to communicate with SUS ETMY. This system runs at 16KHz and has 60uS loop time. 40uS could have been just sufficient, and indeed I did see a non-zero communication error rate, at about 1 error per minute.

I reconfigured the RFM loop at EY to the following

[ISC EY] -> [SUS ETMY] -> [SEI BSC6] -> [LVEA SUS ITMY] -> back to ISC EY

Now the EY IOP watchdog error rate is zero. I'll trend the ISC EY -> SUS ETMY to check it is now truely zero, but I watched it for several minutes last night and it was zero.