The Symptoms
============

The laser is off, the chillers are running and the status screen shows either
"Interlock OK" as red, or both "Interlock OK" and "Epics Alarm" as red.



Possible Causes
===============

a) One of the safety relays in the interlock box is faulty.

b) There is a problem with the TwinCAT code controlling the chillers.

c) There is a problem with the TwinSafe code implementation.

d) Network delays in the PSL EtherCAT network accumulate to a point where
TwinSafe initiates a shutdown.

e) The turbine based flow sensor in the chiller gets stuck, registering a drop
in flow which in turn triggers the laser interlock.



The Evidence
============

a) The Dold LG5929 safety relay extension module has a mechanical life time of
20 million switching cycles and a mean time to dangerous failure of 144.3 years.
The contacts are normally open.

    Safety relays fail, they do not fail intermittently.

    Simulating an interlock box failure by switching off the interlock box
results in a sea of red on the status screen.

b) I have gone through the TwinCAT code for both chillers.  The only difference
is that there is an extra variable declared for the diode chiller.  However this
variable is never invoked in the chiller code.  The instruction sets for both the
diode and crystal chiller are the same.

c) I have gone through the TwinSafe function blocks and have not found anything
wrong with it.

d) This possibility game up because each time the laser had tripped, Patrick and
I noticed that there was 1 lost frame in the TwinCAT datastream.  However with
the laser running last week I noticed that something like 201 frames were lost
and the laser was still running.  So network delays were ruled out.

    Simulating a network delay by removing either the Rx or Tx fibre from the
Ethernet switch, results in a sea of red on the status screen.

e) The output of the flow rate sensor goes to the chiller controller via a normally
open switch.  If the flow rate is within the allowed range, the switch remains closed.
If it goes out of range, it opens and remains open until the flow rate is restored.

    For the crystal chiller, opening the flow rate switch does indeed switch off the
chiller.  Closing the flow rate switch does not switch the chiller back on automatically.
A number of fields on the status screen go red and can only be cleared by switching
the chiller back on and pressing the reset button on the status screen.

    For the diode chiller, opening the flow rate switch does switch off the chiller.
Closing the flow rate switch, turns the chiller back on.  What's more all the red
flags on the status screen go green automatically except "Interlock OK".



Conclusion
==========

Of the 5 cases outlined above, the one that seems to reproduce the observed events
is the last one.  In particular it seems that it is the flow rate sensor in the diode
chiller.