J. Oberling, R. Short

Several maintenance activites for PSL today in advance of the holiday break.

Cooling System Investigation

On Saturday at around 9pm PST the PSL tripped off.  Ryan and I poured through signals and have found no obvious cause of the trip, but it was reported by Richard that when he found the PSL off and checked on the chiller he found the Pyrex measuring cup we use to fill the chiller broken on the floor; that cup usually sits on the shelf above the chiller.  Either something kicked the chiller hard enough to knock the cup off or a sustained vibration caused it to walk off.  Unfortunately we see no evidence in our channels of either scenario: no abnormal changes in reported flow, no obvious oscillations in flow rate or temperature, no chiller alarms at the time of the trip (although the chiller did alarm after the trip, but not during it).  Also, Ryan and Camilla reported having difficulty starting the chiller, which would shut off after ~30 seconds running; on the 4th or 5th attempt the chiller stayed on and they were able to recover the PSL.  Based on this, I think the chiller took an air bubble and the pump air bound, tripping the chiller and the rest of the PSL.  This would possibly explain the measuring cup being knocked off the shelf, and does explain the multiple restarts needed to keep the chiller running (we saw this when initially filling the system last year; we had air in the pump which caused the chiller to turn off, and multiple restarts were required to clear the air from the pump).  In addition, after recovering the PSL the chiller was throwing an alarm every 5-10 minutes (but not tripping the chiller off).  Upon investigation this alarm was identical to the one we sometimes get when bleeding air from the system; the air bleeds out, and this causes a temporary drop in flow, which then causes the chiller to alarm.  I sat in the chiller room for 30 minutes or so observing the system, and noticed that upon every alarm the filter cup behind the chiller would look full of water.  Air would slowly begin to accumulate in the filter cup, and at some point the air would leave the cup and the chiller would temporarily alarm.  Where did the air go?  It was clearly not being passed through to the chiller, so my best guess at this point was the pressure balance between the air and water would hit a tipping point and the air would go back up the return line (back towards the PSL enclosure).  Obviously not a great situation.  I also briefly entered the PSL enclosure to ensure we did not have a leak, which we did not; I also confirmed that there was no air in the cooling lines in the enclosure, all return and supply lines were full of water.

With the above in mind, during today's access system work (which took the PSL laser down for the duration but left the chiller running) we temporarily increased the flow rate of the chiller by partiallly closing its internal bypass.  We took the flow rate from ~2.1 lpm to ~4.8 lpm.  We then proceeded to bleed a large amount of accumulated air from the system at the filter cup behind the chiller.  We then walked the lines and found the supply lines completely full, but the return lines had trapped air in the first down-anlge portion of the line in the LVEA (along the south wall directly across from HAM1), and in the large down-angle portion in the CER where the cooling lines drop under the mezzanine.  We left the chiller with this increased flow rate while the access system work was being performed, it ran at this rate for roughly 4 hours (the hope is at the higher flow rate we're continuing to push air from the system).

Where did this air come from?  Short answer, I don't know.  It's entirely possible we did not get all of the air out of the system upon firing things up for the first time in 2021; we did run the chiller at a higher rate of flow for a few days to try to push as much air as possible out of the system (this chiller maxes out around 8 lpm).  However, I don't think that explains everything.  We have been occasionally bleeding air from the system throughout 2022 and yet more air remains, more than I would expect given the amount of air observed in the return lines today (although it's entirely likely that the increased flow and pressure we temporarily used today is compressing the amount of air, making appear less than it actually is).  My suspicion is that we have a small leak somewhere that is sucking air into the system (recall that we run this system at a much lower flow and pressure than in previous PSL configuration; in the past we've run as high as 20 lpm flow and ~70 psi of inlet pressure, currently we are running at ~2.1 lpm flow and ~12 psi of inlet pressure).  I have no evidence for this, unfortunately, this is only a suspicion right now.  We'll have to keep observing the system, as it's entirely possible that this chiller is not strong enough to push all of the air out of the lines (many vertical pipes in the system).

Once the access system work completed we returned the chiller flow to its usual value of ~2.1 lpm and restarted the PSL.  We let the PSL warm up for about 30 minutes and then proceeded with our remaining maintenance tasks.  We also checked the cooling lines and confirmed that we had removed the vast majority of the built up air from the system.  Unfortunately some small air pockets still remain, I'm starting to suspect this chiller isn't quite strong enough to push it all out.

PMC & RefCav Beam Alignment & Amplifier Pump Diode Current Increase

We remotely tweaked the beam into the PMC and the RefCav, starting with the PMC.  The PMC realignment was done with the ISS OFF.  When we started we had ~14 W reflected and ~106.4 W transmitted; when we stopped we had ~13 W reflected and ~107.5 W transmitted.

At this point we decided to bump up the pump diode currents, as the total system output had dropped from 138.5 W to ~129 W over the last year (first time we've done this with the new system, 11 months in).  We first tried to increase the currents on Amp1, which didn't really net us much beyond worsening PMC mode matching so we reverted back to their starting value of 9 A.  We then increased the currents for Amp2 and the results at the PMC were immediate.  While we didn't see much increase in the measured output power (measured by a PD using the leakage from the first turning mirror after the amplifier), the PMC output jumped up and the reflected power decreased markedly (indicating that while we weren't seeing a big power increase (I want to check that witness PD at some point), we were changing the beam quality, returning it to where we had it at the end of system install in January '22).  In the end we increased the diode currents by 0.2 A, from 8.5 A to 8.7 A; the PMC output jumped from ~107.5W to ~110.8 W, and the reflected power dropped from 13 W to ~11 W.  With the ISS back ON we now have 10.8 W reflected and 112.0 W transmitted; we had to change the ISS RefSignal from -1.80 V to -1.89 V to get the diffracted power percentage back to ~2.2% (increase in PMC output power caused the diffracted power % to jump to ~6% at the -1.80 V RefSignal).

We then proceeded to remotely tweak the FSS RefCav alignment.  When we started we had a RefCav TPD of 0.711 V, and had a TPD of 0.98 V upon completion.

We will continue to monitor the cooling system for air bubbles.  We think that the chiller alarm channel might be a decent witness for when air bubbles are becoming a problem (frequency of chiller alarms appears to pick up when air bubbles get bad, like after recovering from the trip yesterday), so we are going to add that channel to our weekly PSL trends and see how well that works out.