Vicky, Naoki
In the last lock, the FC IR kept unlocking. We first thought the timer issue similar to 76280 so we removed the timer, but it did not help. We checked the FC IR OLG and it was fine. Then we found that the FC ASC input signal moved a lot. We turned off FC ASC and the stability was improved. We decided to go to observe without FC ASC tonight.
H1's Observing was short-lived.
Squeezer Issues:
First issues was the Squeezer dropping us from Observing multiple times because of the SQZ_FC guardian which would unlock (this happened about 3x, and then H1 lost lock). Naoki was investigating at the time of the H1 lockloss. He mentioned finding an issue and has updated the guardian and will make an alog.
Additionally: WIND
About 2hrs ago winds picked up very fast (from max speeds of 10mph up to current 25mph). See attached.
Currently trying to get through Green Arms---X-arm was not good and INCREASE FLASHES was needed a few times.
Vicky also taking a look at Squeezer World, too.
Also having gusts close to 40mph. H1 couldn't get past FINDIR, so running another alignment (crossing fingers for SRC this time).
After being down about 4.5hrs, H1 is finally back post-south-pacific EQ (there was a short Observing stretch, but it only lasted a few minutes after a lockloss as the world was still rumbling---Picket Fence was noisy).
Ran an Initial Alignment due to DRMI locking issues. SRC OFFLOADED had to be skipped since SRY would not acquire. But H1 locked up on its first attempt after the partial Initial Alignment.
TITLE: 04/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 10mph Gusts, 6mph 5min avg
Primary useism: 0.09 μm/s
Secondary useism: 0.14 μm/s
QUICK SUMMARY:
Walked in as TJ was almost done recovering H1 from the 3-Detector-Shaking south pacific earthquake. Nuc33 had another DTT for SQZ crash (will rerun the startup script again, but TJ mentioned it has been crashing fairly regularly).....and....H1 just went back to OBSERVING!
TITLE: 04/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Two lock losses, one unknown and one from an earthquake. Relocking has been straight forward. The first relock I touch SRM to lock DRMI and to avoid an initial alignment. The second relock that is finishing up has been full auto.
LOG:
Lock loss from a 6.5M Earthquake near Papua New Guinea. Looks like it was just over what we could handle.
Sun Apr 14 10:13:38 2024 INFO: Fill completed in 13min 34secs
No obvious reason for the lock loss that ended the lock just shy of 26 hours. The lock loss tool doesn't seem to be working at the moment.
Back to Observing at 1655.
I intervened at DRMI locking to avoid an initial alignment by moving SRM to get DRMI to catch.
TITLE: 04/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY: Locked for 25.5 hours. Some great triple coincidence over the last 24 hours.
TITLE: 04/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
H1's been locked for ~17.5hrs (and currently observing for ~12hrs).
LOG:
H1 is humming along with a lock over 13.5hrs (and a 6.5+hr triple coincidence stretch). Winds are low and microseism has been hovering over the 50th percentile the last couple shifts.
TITLE: 04/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 1mph Gusts, 0mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY:
H1's been locked for 9.5+hrs, and it was Saturday Tour day, but TJ said the last tour has completed.
TITLE: 04/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Locked for 9.5 hours. We went out of Observing at 1830 UTC to run a calibration sweep in coordination with LLO, but went quickly back in Observing at 1901UTC.
LOG:
Ran a broadband and simulines sweep in coordination with LLO at 1830UTC and back to Observing at 1901UTC. Simulines info:
Start:
PDT: 2024-04-13 11:39:00.895189 PDT
UTC: 2024-04-13 18:39:00.895189 UTC
GPS: 1397068758.895189
End:
PDT: 2024-04-13 12:00:31.547582 PDT
UTC: 2024-04-13 19:00:31.547582 UTC
GPS: 1397070049.547582
2024-04-13 19:00:31,472 | INFO | File written out to: /ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20240413T
183901Z.hdf5
2024-04-13 19:00:31,479 | INFO | File written out to: /ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_202
40413T183901Z.hdf5
2024-04-13 19:00:31,485 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_202
40413T183901Z.hdf5
2024-04-13 19:00:31,490 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_202
40413T183901Z.hdf5
2024-04-13 19:00:31,494 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_202
40413T183901Z.hdf5
ICE default IO error handler doing an exit(), pid = 2063220, errno = 32
Sat Apr 13 10:13:33 2024 INFO: Fill completed in 13min 29secs
TITLE: 04/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 1mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.20 μm/s
QUICK SUMMARY: Locked for 1.5 hours, automated relock.
TITLE: 04/12 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
Nice shift with H1 eventually locking up on its own with no intervention; a GW candidate mid-shift; and H1 running on a 7+hr lock (and 5+ hrs of triple coincidence).
LOG:
H1's been locked for almost 3.25hrs (just under 3hrs of observing) and we have been in triple coincidence for much of this shift except for H1 being down at the beginning & Virgo down for a lockloss 2hrs ago.
As a separate issue (I think) than FC not locking, the pump ISS control voltage was dropping below 2V and losing lock. I aligned the fiber polarization, but more importantly sent more power from the pump aom to the fiber, aka reduced the rejected power on H1:SQZ-SHG_REJECTED_DC_POWERMON, to keep pump ISS locked. On a Tuesday with extra time, might be worth checking if the pump path alignment (pump aom + fiber) is still good.
Once we got back to NLN, continued to have the issue with SQZ_FC unlocking. Naoki & Vicky continued to work on it.
Currently we have SQUEEZING.
To get to OBSERVING, I had to ACCEPT some Diffs (see attached).
If the SQZ_FC unlocks again, Operators should take steps to put H1 in NLN + NO SQUEEZING (I'm set up to follow the procedure to do this if it happens in the next 45min, otherwise, I sent a text to the OWL shift operator.)
See LHO:77188 - these FC locking issues seem correlated with >30-40 mph winds, likely moving FC2 SUS too much to hold lock. We had seen the FCGS trans beam (on SQZT8 in FCES) moving a lot on the camera, which suggested the FC cavity axis was moving a lot. This is why we disabled FC ASC this night. If that helped, it's b/c FC2 SUS, aka the FC cavity axis, was moving due to winds. We traced this down the following night 77188 when higher winds prevented even FC green VCO locking, and we saw the FCGS control signal was railing and correlated with FC2 SUS motion with no control signal.
Both nights (this and following one) - FC locked overnight once winds calmed down, see screenshot.