Happened a third time: 2020-03-17 05:04:09 UTC. Still much reduced since the chiller swap.
Looked back at this after TJ suggested we should check that both chillers are keeping constant temperatures and flow rates. Temperatures in ITMX and ITMY cCO2 chillers seems to be stable. See plot over last 2 years here. As the laser hasn't ben on recently we cannot check how stable it is.
Flow rate reported by ITMX chiller seems to regularly drop from the nominal 3.8 to 2 GPM to up to 10 times an hour. However these drops only last one data point which doesn't effect the laser controller safety controls. See zoomed plot here. The chiller has been like for the past 3.5 years (as far back as I can easily see in EPICS).
2021/09/21 Tuesday 17:30UTC Both ITMX and ITMY chiller temperatures decreased from 22 degC to 19.5 degC. The set point is still at 22degC but the channel H1:TCS-{ITMY/ITMX}_CO2_CHILLER_OUT_GAIN_OUT16 dropped to zero Simulink for h1tcscs . Ndscope trace here. This apears to be the day the h1tcscs model was installed on h1oaf0 - see alog 60002. Tagging cds.
ECR for upgrading the CO2 controller is here: https://dcc.ligo.org/E1600312
Latest concept for redesigning CO2 controller is attached. Project is stalled right now due to lack of manpower.
Daniel fixed this setpoint problem by finding the filters had been removed (see alog 60660).
Once the filters were added back in ~19.15UTC 11/16/2021, the filters seems to be maxed out. Daniel turned off the laser power servo loop (sitemap > TCS > CO2{X/Y} > LASER > ON/OFF for PZT_SERVO_GAIN) and they behaved as expected. These values are not recorded in SDF. It doesn't make sence for the chillers to be tied to the laser when the laser is off so will investigate if the matrix changed.
The PZT_SERVO_GAIN OUTPUT we turned OFF 2021/11/16 20UTC must have not being svn saved and turned back on on 2021/11/17 22UTC. See attached trend.
The design is to stabilize the laser PZT error signal by controlling the chiller temperature (E1300233p20). When the laser is off this seems to break down and change the have the setpoint given to the chiller (CHILLER_SETPT1) far from the ~20deg setpoint offset H1:TCS-ITM{}_CO2_CHILLER_SET_POINT_OFFSET (seen in TCS CO2 LASER screen). See table below.
TJ and I are looking into these settings. Maybe the matrix gain values or PZT gains are incorrect or the loop should be switched off when the laser is off.
| Current values | CHILLER_SET_POINT_OFFSET | PZT_SERVO_GAIN | CHILLER_SETPT1 (setpoint sent to chiller) |
| CO2X | 20.6 | -111 | 16.6* |
| CO2Y | 20.5 | 55 | 22.4 |
*Documentation states it's important not to let the chillers be +/- 5deg from 20deg otherwise there may be condensation formed in the laser. The chillers have limits to shut off if they get outside this range but only compared to the CHILLER_SETPT1 value.
The PZT_SERVO_GAIN OUTPUT is turned off in the CO2 Guardian's DOWN state (well done guardian), running DOWN fixed the problem.
Now the lasers think the set point given to the chillers is 20deg but the chillers are receiving set point request of 22degC. So another issue to solve.