J. Oberling, P. King

The investigation

Today we investigated what appeared to be frozen PSL channel: H1:PSL-OSC_DCHILFLOW.  Upon entering the laser diode room and comparing the data on the PSL Beckhoff computer's CHIL screen for the diode chiller to the front panel of the chiller itself, we found that all of the channels were not reading properly (flow rate, temp set point, actual temp, conductivity).  These data are read into the Beckhoff computer via RS-232 connections; the flow rate is read into the PSL Interlock Control Box, and the rest of the data are read directly into the Beckhoff computer via an add-in PCI RS-232 card.  Digging into the Beckhoff code it appeared that the Beckhoff data channels were not properly reading in the data from the diode chiller.

What we think happened

Back in May we performed several tests of the interlock system for the chillers to see how they behaved when certain cables were unplugged.  See Peter's alog here, under "The Evidence," point e where he talks about opening and closing the interlock switch.  We unplugged the cables to open the interlock and plugged them back in to close.  When this was performed, we did not restart the PSL Beckhoff PC; we should have, as RS-232 is not hot-swappable.  Therefore the channels for the diode chiller got stuck (interestingly enough the crystal chiller's channels have all been fine, and we performed the same test with it).

The solution

To fix this, we restarted the PSL Beckhoff PC.  We first called the control room to inform them of our intentions, since restarting the PC shuts the PSL down.  They put the IFO in a safe configuration (in this case they simply unlocked the IMC) and we shut the laser down and restarted the Beckhoff PC.  Everything came back on just fine, the PSL fired right up without issue.  Looking again at the CHIL screen the diode chiller data now matches that shown on the front panel of the diode chiller.  Problem solved!

One caveat

While we are now reading data on H1:PSL-OSC_DCHILFLOW, we do not trust the actual reported flow rate.  The same holds true for the crystal chiller, H1:PSL-OSC_XCHILFLOW.  Back in May we replaced the turbine style flow sensors in the PSL chillers with vortex style flow sensors (no moving parts in the new sensors).  With the sensor change we also had to change the flow sensor calibration, a number we received from LZH, who had tested these flow sensors (we went from ~550 pules/liter to 970 pulses/liter).  Upon this change the flow rate on the crystal chiller "dropped" from ~18 lpm to 9.7 lpm, a change of almost a factor of 2.  There was no associated drop in the HPO laser head flow rates (each of the 4 laser heads has its own flow sensor), therefore we do not trust this calibration number and therefore do not trust the flow rate being reported by the chillers.  That being said, we can still use the reported flow rate to monitor if there is a change in flow and as an indicator that something might be wrong.  The flow rate should not change, and if it does then something is not working correctly regardless of the flow rate being output by the chillers.  We do, however, trust the flow rates being reported by the HPO laser head flow sensors and by the water cooled power meter flow sensors.  To fix this, we will take the next opportunity when we have to swap out the running chillers for the spare chillers.  Before putting the spare chiller into operation, we will hook up an external flow meter on its own water circuit (this is before hooking the chiller up to the PSL water circuit), measure the flow of the chiller and adjust the pulses/liter calibration number until the chiller reports the same flow rate as the external flow meter.