J. Oberling, J. Bartlett, R. Savage, R. McCarthy, C. Soike, T. Guidry, S. Lorimer
Summary
Diode chiller failed due to a broken fitting, causing enough water to leak to trigger the chiller's low water level interlock, thereby bringing the entire PSL down. We installed the backup chiller, but it also failed (crystal chiller wasn't cooling the water). To get the system back up and running again we found a push-to-connect fitting that would work on a temporary basis until we could source a replacement. After some bumps, chiefly clearing air from the system, the PSL is back up and running again.
Details (massive wall of text warning)
As reported in Corey's alog here, the PSL diode chiller failed last night. Upon arriving this morning we found that one of the plastic push-to-connect fittings on the chiller bypass valve had failed, leaking enough water to trip the chiller's low water level interlock; this caused the shutdown of the PSL last night. With the chiller off, Jeff was gently testing the extent of the failure when the entire fitting snapped off at the valve, so it was barely hanging on as it was. The attached pictures show the failed fitting:
- 1st pic: The vavle itself. Jeff's finger is touching the failed fitting, and a stream of water can be seen coming out of the bottom of the fitting, where it seals against the valve. Apologies for the bad focus, best I could do one-handed while leaning over Jeff in a cramped space with an almost 6 year old phone.
- 2nd pic: The valve after the fitting sheared off entirely. The water seen leaking here is residual water from the lines.
- 3rd pic: The other end of the failed fitting. As can be seen, it completely sheared off at the valve.
The quickest way to get back up and running was to swap in the backup chiller, so that's what we did. Once the backup chiller was installed, we powered everything on to test; the diode chiller fired up just fine, but the crystal chiller would not flow water (it would start, but the pump wasn't pumping). Knowing that we generally see a controller failure when swapping chillers, we swapped the crystal chiller controller with a spare (we had 2). This didn't help. Next idea was that the pump was air bound; we had bled it while filling, but maybe there was still some residual air keeping the pump from operating. Turns out this was the problem. After we bled the crystal chiller pump, it fired up without issue.
We then set about setting the operating pressure and flows for the crystal chiller circuits. For future reference, this is done by using the valve installed at the back of the chiller to throttle the flow until the operating pressure (read by the pressure meters mounted to the Al plate below the filters mounted on the wall) is in the range we want, then adjusting the bypass mounted on the wall to achieve the flow we want. At that point it becomes a balancing act between these 2 valves to set the operating pressure and flow for the system. This done, we let the chiller run and observed it to make sure everything was working as intended. At this point we noticed that the crystal chiller wasn't cooling the water; the chiller setpoint is 20.0 °C, the water temperature was ~29.0 °C. Since we had swapped in a new controller, we didn't suspect it as being the problem. With the chiller running, the heat exchanger coils were checked; they should be warm if the chiller is cooling properly. They were ice cold. Jeff checked the sight glass and did not see any refrigerant flowing, indicating that the chiller needs a recharge. So now we have one chiller unit with a busted diode chiller and a backup chiller unit with crystal chiller that wouldn't chill. Wonderful.
At this point we decided that the best course of action was to find a way to replace the failed fitting, either with an exact replacement from another chiller (like our working diode chiller with the not-working crystal chiller) or with fittings we could make work. We checked the bypass valve for the backup diode chiller, and found that the fitting scheme for the valve was completely different from the in-service diode chiller. Instead of plastic fittings with clear tubing, the backup diode chiller used stainless steel fittings, flexible metal tubing, and completely different connections. At some point while manufacturing these chillers for LIGO (the in-service chiller was orignally made for H2, which is now 3IFO; the backup chiller is our original H1 chiller unit. The SNs of these chillers are only a few numbers apart), TermoTek decided to change the internal design for the bypass valve water circuit. Wonderful.
At this point we began a part hunt for fittings we could possibly use to replace the failed one. In the end, the only fitting we could find that worked was a brass 10mm push-to-connect with a male tapered thread on one end. The valve uses straight pipe thread, so we used an adapter to make it work. While brass is not ideal for use with DI water, this is a temporary fix and won't hurt the system in the interim (we are no longer running the HPO, which cooled the laser gain crystals with direct water contact and was the reason for no brass in the system). The fourth picture shows the fix. We got the in-service chiller reinstalled and back up and running, leak checked, and operating pressure and flows set. Right before I was getting ready to start the PSL, the diode chiller gave a loud knock that shook the entire unit; the diode chiller pump was air hammering. It began performing a series of loud air hammers, and while it was doing this the chiller was completely unresponsive to control from the PSL computer; we had to shut it off at the wall. We were able to set the chiller so we could turn it on and off at the control panel (it can be turned on via a number of ways (line, key, remote start), but only one at a time (can't turn on at the control panel if Remote Start is selected)). After a series of starts and stops, the pump stopped air hammering. Our suspicion here is an air bubble got caught in the pump and wouldn't clear, causing the air hammering. Once we shut the chiller off and let it sit for a few minutes the air bubble cleared and the hammering stopped.
With the hammering stopped, we let the chiller run for ~45 minutes to see if the behavior would continue; it never did, so Ed restarted the PSL as reported here, and things are up and running once again. We'll monitor the chiller over the next couple of days to ensure everything is OK. In addition, we'll source a replacement valve and get that installed in a more controlled manner during an upcoming Tuesday maintenance window, thereby removing the brass fitting from the system. We will also get a HVAC tech to look at our backup crystal chiller and get it up and running again so we have a working backup, which we do not right now.
This series of events reminds me of the failure we experienced this last February (in-service chiller failed, then the backup failed upon installation, which required us to pull a chiller from 3IFO to get the PSL running again), and seems to be a running "feature" of these chillers (we almost always have some secondary issue when working on these chillers; generally this is a failure of a chiller controller, but not always, as today and February indicate). Thankfully, we didn't have to dip into 3IFO stock this time to return the system to normal operation. This, to me, is more evidence that we need to move on from these chillers to something smaller and less complex. These chillers were designed with cooling the HPO in mind. Since we no longer run the HPO and are using amplifiers (for O3 and for the upcoming pre-O4 PSL upgrades), these chillers are massive overkill for what we actually need, and almost always end up causing massive headaches with even the simplest of failures. This has been discussed as part of the pre-O4 PSL upgrades, and is a move I wholeheartedly support.
I want to give a huge Thank You! to Richard, Chris, Tyler, and Scott for their assistance this morning, it is much appreciated.
Refer to alog 53864 regarding GC network issues overnight. The edge network ports do not service CDS (CDS maintains a separate switch and connection to the LHO network switching core), and that network connection nor the upstream connection to our ISP went down. Coincidental, but not causal.