The goal of yesterdays firmware upgrade was to change the behavior of the blinking LED lights and to allow for additional diagnostics of the problem. In the past the LED typically had three states: (i) Off, (ii) Blinking and (iii) On. This upgrade interchanged the states (ii) and (iii). For a typical link light this means: Off indicates no fiber signal, Blinking indicates a fiber signal is present, and On means the timing signal is received and synchronized. Operating without error now means all LEDs are On.

Furthermore, it is possible with the Altium JTAG port to change the behavior of the LED state while running and without disrupting the timing signal. LEDs which are on can be made to blink in rates of 0.5Hz, 1Hz, 2Hz, 4Hz, etc. This should allow us to isolate the coupling site.

The upgrade was successful for all timing slaves, but failed for the timing master/fanouts. Strangely, it failed due to bogus code in svn on how to calculated the CRC-32 (ethernet) checksum. Baffling. More investigations are ongoing.