Reports until 05:53, Saturday 08 February 2020
H1 TCS
camilla.compton@LIGO.ORG - posted 05:53, Saturday 08 February 2020 - last comment - 14:40, Friday 03 December 2021(54980)
TCSy CO2 Laser has only Kicked us out of Observing Twice since Chiller Swap
Following on from FRS 13095.  Before October we were pushed out of Observing a few times by the TCSy CO2 laser loosing lock. TCSY chiller was swapped 2019-09-18 alog 51996. Flow meter was swapped 2019-10-10 alog 52406.
 
By comparing H1:GRD-TCS_ITMY_CO2_STATE_N with our observation bit and checking logs, I have found only 2 cases since the swap (5months ago) where the TCSy CO2 laser has pushed us out of observing by loosing lock (2019-09-29 23:57 UTC and 2019-12-05 05:41 UTC). The first time being before the flow meter was swapped.
There were 15 times it pushed us out of observing from July 1st to the chiller swap (all between 2019-07-10 and 2019-08-31). Unless something else changed at the end of August I think we can confirm that the original chiller or new flow meter solved our problem.
Attached is the H1:GRD-TCS_ITMY_CO2_STATE_N from May 2019 to now (t1 line = chiller swap. t2 line = flow meter swap). The C02 laser now looses lock to search for a new lock point less regularly and seems to be able to wait until H1 is unlocked. 
Images attached to this report
Comments related to this report
camilla.compton@LIGO.ORG - 22:14, Monday 16 March 2020 (55638)
Link

Happened a third time: 2020-03-17 05:04:09 UTC. Still much reduced since the chiller swap.

camilla.compton@LIGO.ORG - 11:28, Tuesday 02 November 2021 (60493)CDS
Link

Looked back at this after TJ suggested we should check that both chillers are keeping constant temperatures and flow rates. Temperatures in ITMX and ITMY cCO2 chillers seems to be stable. See plot over last 2 years here. As the laser hasn't ben on recently we cannot check how stable it is.

Flow rate reported by ITMX chiller seems to regularly drop from the nominal 3.8 to 2 GPM to up to 10 times an hour. However these drops only last one data point which doesn't effect the laser controller safety controls. See zoomed plot here. The chiller has been like for the past 3.5 years (as far back as I can easily see in EPICS).

2021/09/21 Tuesday 17:30UTC Both ITMX and ITMY chiller temperatures decreased from 22 degC to 19.5 degC. The set point is still at 22degC but the channel H1:TCS-{ITMY/ITMX}_CO2_CHILLER_OUT_GAIN_OUT16 dropped to zero Simulink for h1tcscs . Ndscope trace here. This apears to be the day the h1tcscs model was installed on h1oaf0 - see alog 60002. Tagging cds.

Images attached to this comment
aidan.brooks@LIGO.ORG - 10:46, Wednesday 03 November 2021 (60511)
Link

ECR for upgrading the CO2 controller is here: https://dcc.ligo.org/E1600312

Latest concept for redesigning CO2 controller is attached. Project is stalled right now due to lack of manpower.

 

Non-image files attached to this comment
camilla.compton@LIGO.ORG - 12:59, Tuesday 16 November 2021 (60663)
Link

Daniel fixed this setpoint problem by finding the filters had been removed (see alog 60660).

Once the filters were added back in ~19.15UTC 11/16/2021, the filters seems to be maxed out. Daniel turned off the laser power servo loop (sitemap > TCS > CO2{X/Y} > LASER > ON/OFF for PZT_SERVO_GAIN) and they behaved as expected. These values are not recorded in SDF. It doesn't make sence for the chillers to be tied to the laser when the laser is off so will investigate if the matrix changed. 

 

camilla.compton@LIGO.ORG - 10:07, Friday 03 December 2021 (60842)
Link

The PZT_SERVO_GAIN OUTPUT we turned OFF 2021/11/16 20UTC must have not being svn saved and turned back on on 2021/11/17 22UTC. See attached trend.

The design is to stabilize the laser PZT error signal by controlling the chiller temperature (E1300233p20). When the laser is off this seems to break down and change the have the setpoint given to the chiller (CHILLER_SETPT1) far from the ~20deg setpoint offset H1:TCS-ITM{}_CO2_CHILLER_SET_POINT_OFFSET (seen in TCS CO2 LASER screen). See table below.

TJ and I are looking into these settings. Maybe the matrix gain values or PZT gains are incorrect or the loop should be switched off when the laser is off.

Current values CHILLER_SET_POINT_OFFSET PZT_SERVO_GAIN CHILLER_SETPT1 (setpoint sent to chiller) 
CO2X 20.6 -111 16.6*
CO2Y 20.5 55 22.4

*Documentation states it's important not to let the chillers be +/- 5deg from 20deg otherwise there may be condensation formed in the laser. The chillers have limits to shut off if they get outside this range but only compared to the CHILLER_SETPT1 value. 

Images attached to this comment
camilla.compton@LIGO.ORG - 14:40, Friday 03 December 2021 (60850)
Link

The PZT_SERVO_GAIN OUTPUT is turned off in the CO2 Guardian's DOWN state (well done guardian), running DOWN fixed the problem.

Now the lasers think the set point given to the chillers is 20deg but the chillers are receiving set point request of 22degC. So another issue to solve.