Jim, Dave

First the good news, the RFM receive error rates on the end station SUS models have been fixed. Now the bad news, we don't know why what we did would fix the problem.

Prior to today's work, the RFM loop was such that the LSC front end communicated directly with the end station SUS front end. The loop continued on to the end station ISC, then back to the corner through OAF and ASC back to LSC. The error rate at both end station SUS systems for both channels being sent by the OMC model was about one every 5 seconds. The OMC sender was running at 13uS, so we should have zero errors at the end station. Indeed, we added the very same receivers in the PEM model (running on ISC front end) and verified a zero error rate there.

Working at EY, we first removed ISC from the RFM loop. The SUS errors went to zero (even though ISC was not inbetween LSC and SUS, we just shorted the loop by one node).

We reinserted ISC, and the SUS errors came back as before.

We then switched the position of SUS and ISC on the end station RFM switch, so now ISC is between LSC and SUS, and SUS error rate went to zero. We left EY in this configuration.

We then went to EX and repeated the above, with the same results. EX now also has SUS/ISC swapped from this morning.

To check once again that we know the direction the loop takes through the VMIC 5596 RFM switch, we repeated our DTS measurements (see X1 alog  for details). We are sure that prior to the change LSC communicated directly with SUS, and now ISC is inbetween.

After many hour of running we have not had a single RFM IPC error.

Operators should now closely investigate any red blocks on the CDS Overview MEDM. The only ones we would expect are the occassional ADC errors on certain IOP models.

We will try to reproduce the original SUS errors on the DTS to investigate why removing ISC or moving the ISC fixes SUS's problem.