EdM, JasonjO, JeffB, PatrickT, PeterK, RickS

As the frequency of PSL shutdowns apparently initiated by the PSL Beckhoff interlock system increased, on Thursday morning we decided to take the system down and make some changes.  Note that all Beckhoff interlock triggers were apparently generated by the Diode Chiller flow interlock input.

We have had some experience with PSL chiller flow sensors that use a rotating rotor failing.  Replacement sensors that use optical sensing of eddies induced by a vane in the water flow (no moving parts) have been tested on the PSL reference system in Hannover.  We installed these sensors in the spare PSL chillers (there are two, the Crystal Chiller, and the Diode Chiller).

Because we don't have spare 480V 3-phase circuits available in the Chiller Room, we were not able to test the spare chillers but we tested the installaiton process.  Thursday morning we shut down the PSL and installed the new flow sensors in the chillers that had been operating.  We elected to do this rather than switch to the spare chillers for several reasons, including minimizing down time.

We found that the chillers would not operate with the new flow sensors.  Further, we found that the chiller controllers would not function normally, apparently unrelated to the flow sensors.  After consultation with the Thermotek (manufacturer) reps over the phone, we decided to switch to the spare chillers.

We found that the spare chillers would not operate with the new flow sensors either, though they had been installed and tested in Hannover and detailed instllation instructions had been provided and followed.

We swapped the flow sensors in the spare chillers back to the original-style sensors.   We first found that the Diode Chiller flow sensor appeared to be faulty so we swapped it with a similar sensor.  We eventually found that the chiller-side wiring of the flow sensor in the Diode Chiller is different than in the Crystal Chiller- the black and white wires on the sensor need to be swapped.

While writing an aLog entry Thursday evening, the chiller tripped again, twice, so we swapped out the Diode Chiller sensor one more time.  We left Thursday night with the chiller apparently functioning properly.  However within a few hours, Sheila and Evan gave up commissioning efforts after several shutdowns in quick succession.

Trends from Thursday night, when the chillers were operating but the laser had shut down due to a Beckhoff IL trip seemed to indicate Beckhoff IL events that were not triggered by any of the inputs, including the Diode Chiller flow input.  Having observed Beckhoff IL events from the Diode Chiller only, even with the spare chillers and several different flow sensors (with the Crystal Chillers apparently operating without incident and no indication that there actuall was a flow interruption in the Diode Chillers), we decided to investigate functionality issues with the Beckhoff IL controller.

We attempted to swap in the spare Beckhoff IL module, but found the issues detailed in Patrick Thomas's aLog entry.  We switched back to the original Beckhoff IL controller and eventually Patrick and PeterK were able to get it back on-line late yesterday (Friday) afternoon.

With the chiller apparently operating normally, we decided to leave it running and see if some commissioning work could be accomplished.  We expect that the shutdowns will continue until we remedy the real source of the problems.  However, we removed and re-connected most relevant cables and there is a remote possibility that we could have fixed an intermittent connection.

Next steps:

If we see the shutdowns, attributed to Diode Chiller Flow errors, again, we will swap the Diode Chiller and Crystal Chiller interlock cables to see if the shutdowns follow the cables (cable or Beckhoff IL problem) or chiller (Diode Chiller problem).

We also ordered three new old-style flow sensors that will be delivered today along with two new controllers for the originally-operating chillers.  Next week we will work to get the spare chillers operating, first with the old-style sensors, then maybe with the new-style sensors, so that we will have a backup ready.

We will also discuss the issues we encountered with the Beckhoff IL module replacement with the people at NeoLase in Hannover.