The Symptoms ============ The laser is off, the chillers are running and the status screen shows either "Interlock OK" as red, or both "Interlock OK" and "Epics Alarm" as red. Possible Causes =============== a) One of the safety relays in the interlock box is faulty. b) There is a problem with the TwinCAT code controlling the chillers. c) There is a problem with the TwinSafe code implementation. d) Network delays in the PSL EtherCAT network accumulate to a point where TwinSafe initiates a shutdown. e) The turbine based flow sensor in the chiller gets stuck, registering a drop in flow which in turn triggers the laser interlock. The Evidence ============ a) The Dold LG5929 safety relay extension module has a mechanical life time of 20 million switching cycles and a mean time to dangerous failure of 144.3 years. The contacts are normally open. Safety relays fail, they do not fail intermittently. Simulating an interlock box failure by switching off the interlock box results in a sea of red on the status screen. b) I have gone through the TwinCAT code for both chillers. The only difference is that there is an extra variable declared for the diode chiller. However this variable is never invoked in the chiller code. The instruction sets for both the diode and crystal chiller are the same. c) I have gone through the TwinSafe function blocks and have not found anything wrong with it. d) This possibility game up because each time the laser had tripped, Patrick and I noticed that there was 1 lost frame in the TwinCAT datastream. However with the laser running last week I noticed that something like 201 frames were lost and the laser was still running. So network delays were ruled out. Simulating a network delay by removing either the Rx or Tx fibre from the Ethernet switch, results in a sea of red on the status screen. e) The output of the flow rate sensor goes to the chiller controller via a normally open switch. If the flow rate is within the allowed range, the switch remains closed. If it goes out of range, it opens and remains open until the flow rate is restored. For the crystal chiller, opening the flow rate switch does indeed switch off the chiller. Closing the flow rate switch does not switch the chiller back on automatically. A number of fields on the status screen go red and can only be cleared by switching the chiller back on and pressing the reset button on the status screen. For the diode chiller, opening the flow rate switch does switch off the chiller. Closing the flow rate switch, turns the chiller back on. What's more all the red flags on the status screen go green automatically except "Interlock OK". Conclusion ========== Of the 5 cases outlined above, the one that seems to reproduce the observed events is the last one. In particular it seems that it is the flow rate sensor in the diode chiller.