Fri Jan 12 10:10:30 2024 INFO: Fill completed in 10min 26secs
Gerardo confirmed a good fill curbside. Starting TCs were high, trip temp was increased to -60C for this fill. TCmins = -77C,-72C.
NLN Lockloss :
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1389100118
The Lockloss tags said Windy, and the wind is gusting up to about 30MPH so this makes sense.
Lockloss plots and Log attached.
Interesting set of glitches that happened around the same time as the lockloss seems pretty interesting. I'm not sure how to investigate this any further though.
TITLE: 01/12 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 23mph Gusts, 14mph 5min avg
Primary useism: 0.07 μm/s
Secondary useism: 0.34 μm/s
QUICK SUMMARY:
When I came in H1's ISC_LOCK was in LOCKING_ALS after a Lockloss from NOMINAL_LOW_NOISE .
I noticed the Earthquake, check to see if Mr.Short had been called, and saw his alog .
The Chiller in the CER seems to have been fixed and the temperature is coming back down.
Ryan call and gave me the Short story of the last few hours of the OPS OWL Shift, and did a wonderful handoff after calling Dave and Camilla in for help.
Camilla just arrived and will be helping resolve the SQZ issues.
My plan is to try to lock while she is resolving issues and hopfully SQZer will be ready before INJECT_SQZING.
Taking H1 Initial_alignment since it's having a hard time locking beyond Find IR.
I am seeing a rise in LVEA ZONE 1A temperature that has not turned around yet.
Reminder that there is a python script which calculates the current windchills:
david.barker@opslogin0: windchill
EX temp 06.6(degF) wind 11.0(mph) windchill -8.5(degF) -22.5(degC)
MX temp 06.7(degF) wind 11.0(mph) windchill -8.4(degF) -22.4(degC)
CS temp 07.1(degF) wind 19.0(mph) windchill -12.2(degF) -24.6(degC)
MY temp 06.6(degF) wind 07.0(mph) windchill -5.1(degF) -20.6(degC)
EY temp 06.3(degF) wind 28.0(mph) windchill -16.7(degF) -27.0(degC)
wind chill average over site -10.2 degF -23.4 degC
H1 called for help this morning at 6:15 PST when it could not go to observing because of the SQZ system not functioning properly. Eventually I saw that the Slow Controls section of the CDS overview was red, so I called in Dave for assistance. So far we have not found evidence of an actualm hardware failure, but Richard noted that a chiller in the CER had failed this morning, causing temperatures to rapidly rise there, potentially creating this issue with the SQZ Beckhoff terminals. Richard has turned on a fan and both chillers in the CER to bring the temperatures back down.
SQZ slow control Beckhoff is reporting errors in most systems: laser, fiber, pmc, shg, opo, clf and adf.
fiber lock has a "temperature feedback limits" error which triggered when the CER went above 26C at 05:44 PST this morning.
As troubleshooting was ongoing, I went through the ObservationWithOrWithoutSqueezing wiki to make the changes necessary for observing without squeezing, since SQZ seems to have been the only system impacted at the moment and our range without squeezing was hovering just above 120Mpc. I did not have the chance to actually start observing as H1 lost lock, but the intent bit was green and I could have begun observing.
To allow the IFO to observe without squeezing, I took the following actions:
All of this must be reverted once the SQZ Beckhoff issue is resolved to observe with squeezing.
This morning the ciller bank that feeds the CER quit chilling. Quite cold outside. Normailly this would not have been a major issue but the secondary bank of chillers was shut down for noise issues in the IFO. With the second bank of chillers off, the loss of the running bank allowed for the temerature to rise quickly and to a point we had not seen before. (30c)
I reset the failed bank and turned the secondary bank on as well. A fan was also added to bring the temperature down quickly. The fan has been shut off.
The power supply that feeds the 5 Beckhoff squeezer chassis tripped off. I have not found a reason for this trip as this power supply is les than 50% loaded and the fan had already been replaced so temperature no different than the other units around it.
I will get with Marc P. to have him measure temperatures of this supply.
Requested SQZ_MANGER to SQZ_READY_IFO, as expected it got suck on TTFSS error. Took H1:SQZ-LASER_HEAD_CRYSTALFREQUENCY from where it had railed at 200 to 40MHz. This took Beatnote close to nominal 160MHz. Still not locking with Temperature Feedback Error. I trended TTFSS screen and after trying H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_RESET (no effect) toggled H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_ON Off and On (in the Slow Frequency Servo" box. TTFSS then locked fine.
All SQZ are in their nominal SQZ_READY_IFO states and I'll revert Ryan's "Observing with no sqz" changes.
Reverted all Ryan's changes and added new GRDs to wiki, we'll need to accept sdf's once we get to NLN. I touched pitch sliders in FC1 (mainly) and FC2, following SQZ Wiki: Issues with SQZ FC Guardian to get FC to lock green.
I checked the supply in question, H1-VDC-C2 U34 RHS (H1-ISC-SQZ SC). This supply originally drew 24V 5A, but we have added more Slow Controls (SC) chassis to this rail since, so it looks to be drawing 10A.
Temperature at the front of the supply is 95F, rear of the supply is 104F, only 9F delta. The ambient air is cool and moving around averaging 68F. Air flow is not restricted on this supply, and it has a new fan.
I do not feel good airflow through this supply like I do others, I would expect a higher temperature delta, most supplies in that area are sitting at 75F front 105F rear, delta of 30F.
If this supply trips again I recommend replacement. In either case, it will be replaced during the vent.
SDF's for Observing with SQZ accepted.
H1 is back to observing as of 12:58 UTC. I accepted two SDF diffs on EY_ISC (these were accepted by Austin during his shift, so I've essentially undone his accepting).
TITLE: 01/12 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
SEI_ENV state: EARTH_QUAKE
Wind: 23mph Gusts, 20mph 5min avg
Primary useism: 1.06 μm/s
Secondary useism: 0.90 μm/s
QUICK SUMMARY:
Austin called and reported on a large nearby earthquake unlocking H1 and tripping ISIs for all four QUADs, BS, HAM7, and HAM8. There was also a typo in the SEI_ENV Guardian which caused it to go into error at this time, but this has been fixed.
Since the earthquake is still shaking, I'll wait until it has fully passed and SEI_ENV goes back to CALM to untip the ISIs, then I'll start an initial alignment and have H1 try to lock. The wind is elevated, but not terrible (20-30mph gusts), so there's potential for that to cause issues.
TITLE: 01/12 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:
- EX saturation @ 1:08
- Lockloss @ 5:53 - cause unknown
- Airhandler alert on alarm handler for MY REHEAT channel - looks like there has been a sharp spike over the past hour, Tagging FMCS
- Relocking was mostly straightforward, I did give ETMY a little tap during ALS to get it to catch but locking was otherwise automated
- Back to NLN @ 7:17/OBSERVE @ 7:24
- Following SDFs on ALS Y popped up, preventing me from going into observing, accepting for now for them to be looked at later, Tagging ISC
- Lockloss @ 7:49 - caused by 5.9 EQ from Alaska
- Called Ryan S. and let him know of the situation, ground motion is already starting to settle and he intends to untrip the WDs and begin an initial alignment soon after
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 23:51 | PCAL | Rick, Tony | PCAL lab | local | Maintenance | 01:23 |
| 00:19 | VAC | Janos | MX | N | VAC checks | 01:09 |
Lockloss @ 7:49, caused by a 5.9 EQ from Alaska.
Lockloss @ 5:53 - cause unknown. Potentialy see a slight rise in oscillations in the DSOFT YAW loops that starts around ~35 seconds before the lockloss. Looking at the LSC loops, it looks like LSC DARM sees the motion first.
Tonight has been uneventful this far. H1 is going strong, 28 hours into its current lock. Secondary microseism has stabilized at 0.5 um/s, but the wind has been steadily on the rise, now ~30 mph here on site.
TITLE: 01/12 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 26mph Gusts, 20mph 5min avg
Primary useism: 0.10 μm/s
Secondary useism: 0.42 μm/s
QUICK SUMMARY:
- H1 still locked, just hit 24 hours and appears to still be stable
- Microseism continues to trend down
- CDS/DMs ok
TITLE: 01/11 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY: Locked for 24 hours, quiet day.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 16:43 | FAC | Karen | Opt lab | n | Tech clean | 17:04 |
| 16:51 | FAC | Kim | H2 | n | Tech clean | 17:05 |
| 18:56 | FAC | Randy | EY | n | Wind fence checks | 19:21 |
| 20:34 | - | Camilla | Yarm | n | Running | 21:04 |
| 21:39 | VAC | Travis | FCTE | n | Looking for parts | 21:40 |
| 23:51 | PCAL | Rick, Tony | PCAL lab | local | Maintenance | 01:51 |
