Sheila, Terra, Craig
The PSL is off, is has been since about 17:30 UTC. The chillers don't seem to be tripped. Confusingly the laser screen indicates that the chillers are fine, (two green boxes) while the PSL_STATUS screen has a red box for the crystal chiller (screenshot attached). Jason will come out to the site to investigate/restart the laser.
We also noted that the temperature trends for the PSL have been usual since thursday morning's incursion. (2nd screnshot) I went to the controller box and saw that the north AS unit was on, which was probably unintentional (the south unit was off, and they are normally both off in science mode). I turned it off at noon local time. Terra noted that the PSL microphone has seen an elevated level of noise in the last few days, which went back to normal as soon as the AC unit was off. (In the third attached screenshot, blue traces are from the time when the AC was on). The montors on the RF AM stabilization also changed when we turned off the AC, and some channels on the AM stabilization box seem to have been sensitive to some kind of switching of the PSL HVAC over the last few days.
It seems like we need some kind of a better way to monitor if the PSL environment settings are correct, maybe adding them to diag main if we can find a good set of tests to write. It also is suprising to me that our RF system seems to be so sensitive to accoustic pick up in the PSL. Has anyone in detchar looked at PSL PEM monitors to see if glitches there are correlated to the "RF45" glitches seen durring O1?
The TCS chillers are also tripped, with the same DAC problem we have been having. (27435) This happened about 36 hours ago.
Bottom chiller screen; flashing between 'temperature' and 'warning'
The laser was up and running this morning when I checked it around 6 am (local).
The gibberish message on the diode chiller controller I've never seen before
and is most likely a controller malfunction.
To fix the problem, I would try (in order):
- power cycling the chiller with the power switch located at the rear of the chiller
- replacing the chiller controller (if Jeff Bartlett happens to have a spare handy)
- install the spare chiller (which will take a bit of work because ... )
* the turbine flow sensors need to be replaced with the vortex ones
* the 3-phase power plug needs to be installed
* some filters need to be removed
* the coolant lines will need to have any air pockets removed
The problem with the first solution is that it is hard to gauge how long the
"fix" might be valid for before the laser could trip out again.
We used Sheila's very instructive alog to kill and restart all the models on the OAF machine, reset the TCS chillers and restart the TCS laser.
J. Oberling, S. Dwyer
Attempted to bring the PSL back up but were ultimately unsuccessful. Came in and found the crystal chiller running and the diode chiller off, although the Laser MEDM screen indicated the diode chiller was up and running. EPICS channels frozen again?
The diode chiller turned on without an issue, although the weirdness on the main screen, seen in Terra's photos above, did not go away. We let the chiller run for several minutes and then attempted to power on the HPO. Approximately midway during the pump diode power up everything stopped and we found the diode chiller shut off. To see if it was a coincidence we reset the interlocks and attempted to turn the HPO on again, this time monitoring the chillers. The HPO got to its second stability range and the diode chiller immediately shut off. We power cycled the diode chiller (which by the way cleared the funky front panel issue seen in the photos above). This time the HPO acheived the second stability range for 10 whole seconds before the diode chiller shut off again; it almost seems as if the chiller is shutting off as soon as it sees a heat load. During all this the crystal chiller remained up and running without issue.
At this time I'm out of ideas, although the chiller behavior copuled with the front screen weirdness makes me think we may have a control panel problem with the diode chiller (as Peter mentions above); I seem to recall that when we had the chiller flow sensor issues last year (April/May 2015) we also had some weird issues with the chiller (the one we just recently removed from service) that were solved by replacing the control panels. I left the PSL off; the diode chiller is also off and the crystal chiller is running. Please do not attempt to turn it on, we will investigate more fully tomorrow morning.
Filed FRS #5605.