Peter, Jeff B, Dave, Kiwamu, Nutsinee
Quick conclusion: They work now.
Details: This morning we troubleshoot the cause of TCS chillers failure last night. All the channels in the chiller servo was requesting the right number (20C, no crazy gains). We went to the chillers and realized that the set point was 5 degree C for both TCSX and Y chillers instead of its nominal (20C). We tried to reset the set point at the chiller and it would come right back to 5C. This suggested that the front end was sending out some bad number to the chillers. This was confirmed further by measuring the voltage between pin 6 (ground) and pin 7 (temperature control) at the connector pinout and we were able to change the temperature setpoint when we unplugged this connector. Later we found out that H1OAF DAC went bad yesterday 19:18 PT. This forced the IOP to send out 0 values and caused the chiller setpoint to change to the lowest temperature possible. 10 minutes after DAC failed the chiller interlocks tripped because the chiller temperature was to low. Accordning to Sheila this happened when the temperature reached about 15 deg C. Below I included the plot of CO2 laser temperature, which is slightly higher than the actual chiller temperature but it's a good indicator of what happened to the chiller temperature (I couldn't find a channel that would report the actual chiller temperature).
We don't know what caused this DAC error. Dave also mentioned that this is quite rare.
Totally unrelated note to what happened, TCSY chiller is really nasty. I've attached a picture of inside the chiller Peter took earlier today.
Hi, can you post a better explanation of what is "nasty" about the TCS-Y chiller. Is the filter dirty? I can't tell from that photo. The whole system was flushed when we were there around the start of the year. Was the AOM head connected back into the water system again after it was inspected? Once we remove the AOMs we will want to inspect fully and if the water seems contaminated then we will need to flush it again.
As far as the chillers shutting down - this has been a problem whenever the DAC goes to zero. We now have summing boards that can be installed, and after an initial failure to understand why we couldn't get them to work at LHO, we have installed them succesfully at LLO. We now realize that the problem is a setting in the front end code to limit the actuation range of this channel. So we should be getting these put in soon and these will solve the zero-volts-shutdown problem.