ryan.short@LIGO.ORG - posted 06:59, Friday 12 January 2024 - last comment - 16:06, Friday 12 January 2024(75331)
CER Temperature Increase; Failed Chiller; No SQZ
H1 called for help this morning at 6:15 PST when it could not go to observing because of the SQZ system not functioning properly. Eventually I saw that the Slow Controls section of the CDS overview was red, so I called in Dave for assistance. So far we have not found evidence of an actualm hardware failure, but Richard noted that a chiller in the CER had failed this morning, causing temperatures to rapidly rise there, potentially creating this issue with the SQZ Beckhoff terminals. Richard has turned on a fan and both chillers in the CER to bring the temperatures back down.
SQZ slow control Beckhoff is reporting errors in most systems: laser, fiber, pmc, shg, opo, clf and adf.
fiber lock has a "temperature feedback limits" error which triggered when the CER went above 26C at 05:44 PST this morning.
As troubleshooting was ongoing, I went through the ObservationWithOrWithoutSqueezing wiki to make the changes necessary for observing without squeezing, since SQZ seems to have been the only system impacted at the moment and our range without squeezing was hovering just above 120Mpc. I did not have the chance to actually start observing as H1 lost lock, but the intent bit was green and I could have begun observing.
To allow the IFO to observe without squeezing, I took the following actions:
All of this must be reverted once the SQZ Beckhoff issue is resolved to observe with squeezing.
This morning the ciller bank that feeds the CER quit chilling. Quite cold outside. Normailly this would not have been a major issue but the secondary bank of chillers was shut down for noise issues in the IFO. With the second bank of chillers off, the loss of the running bank allowed for the temerature to rise quickly and to a point we had not seen before. (30c)
I reset the failed bank and turned the secondary bank on as well. A fan was also added to bring the temperature down quickly. The fan has been shut off.
The power supply that feeds the 5 Beckhoff squeezer chassis tripped off. I have not found a reason for this trip as this power supply is les than 50% loaded and the fan had already been replaced so temperature no different than the other units around it.
I will get with Marc P. to have him measure temperatures of this supply.
Requested SQZ_MANGER to SQZ_READY_IFO, as expected it got suck on TTFSS error. Took H1:SQZ-LASER_HEAD_CRYSTALFREQUENCY from where it had railed at 200 to 40MHz. This took Beatnote close to nominal 160MHz. Still not locking with Temperature Feedback Error. I trended TTFSS screen and after trying H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_RESET (no effect) toggled H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_ON Off and On (in the Slow Frequency Servo" box. TTFSS then locked fine.
All SQZ are in their nominal SQZ_READY_IFO states and I'll revert Ryan's "Observing with no sqz" changes.
Reverted all Ryan's changes and added new GRDs to wiki, we'll need to accept sdf's once we get to NLN. I touched pitch sliders in FC1 (mainly) and FC2, following SQZ Wiki: Issues with SQZ FC Guardian to get FC to lock green.
I checked the supply in question, H1-VDC-C2 U34 RHS (H1-ISC-SQZ SC). This supply originally drew 24V 5A, but we have added more Slow Controls (SC) chassis to this rail since, so it looks to be drawing 10A.
Temperature at the front of the supply is 95F, rear of the supply is 104F, only 9F delta. The ambient air is cool and moving around averaging 68F. Air flow is not restricted on this supply, and it has a new fan.
I do not feel good airflow through this supply like I do others, I would expect a higher temperature delta, most supplies in that area are sitting at 75F front 105F rear, delta of 30F.
If this supply trips again I recommend replacement. In either case, it will be replaced during the vent.
SDF's for Observing with SQZ accepted.