The Lockloss tool hasn't updated with any of the Locklosses since 4-13-24 12:49UTC
TITLE: 04/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 8mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.13 μm/s
QUICK SUMMARY:
I powercycled ITMY camera (h1cam23) and restarted its server process on h1digivideo2. All looks good now, the camera went down at 04:33 Sun 14th PDT.
TITLE: 04/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
A bit of a rough shift with a rumbling planet for the first half of the shift. Wind storm (20-30mph) for 4hrs of the shift. Mid-shift marked by beginning of FC IR-locking issues for Squeezer. And also had issues with SRC portion of alignment for 2-alignments from the night.
If SQZ_FC unlocks, we should transition to NLN + NO SQUEEZING (via procedure in wiki).
LOG:
Vicky, Naoki
In the last lock, the FC IR kept unlocking. We first thought the timer issue similar to 76280 so we removed the timer, but it did not help. We checked the FC IR OLG and it was fine. Then we found that the FC ASC input signal moved a lot. We turned off FC ASC and the stability was improved. We decided to go to observe without FC ASC tonight.
As a separate issue (I think) than FC not locking, the pump ISS control voltage was dropping below 2V and losing lock. I aligned the fiber polarization, but more importantly sent more power from the pump aom to the fiber, aka reduced the rejected power on H1:SQZ-SHG_REJECTED_DC_POWERMON, to keep pump ISS locked. On a Tuesday with extra time, might be worth checking if the pump path alignment (pump aom + fiber) is still good.
Once we got back to NLN, continued to have the issue with SQZ_FC unlocking. Naoki & Vicky continued to work on it.
Currently we have SQUEEZING.
To get to OBSERVING, I had to ACCEPT some Diffs (see attached).
If the SQZ_FC unlocks again, Operators should take steps to put H1 in NLN + NO SQUEEZING (I'm set up to follow the procedure to do this if it happens in the next 45min, otherwise, I sent a text to the OWL shift operator.)
See LHO:77188 - these FC locking issues seem correlated with >30-40 mph winds, likely moving FC2 SUS too much to hold lock. We had seen the FCGS trans beam (on SQZT8 in FCES) moving a lot on the camera, which suggested the FC cavity axis was moving a lot. This is why we disabled FC ASC this night. If that helped, it's b/c FC2 SUS, aka the FC cavity axis, was moving due to winds. We traced this down the following night 77188 when higher winds prevented even FC green VCO locking, and we saw the FCGS control signal was railing and correlated with FC2 SUS motion with no control signal.
Both nights (this and following one) - FC locked overnight once winds calmed down, see screenshot.
H1's Observing was short-lived.
Squeezer Issues:
First issues was the Squeezer dropping us from Observing multiple times because of the SQZ_FC guardian which would unlock (this happened about 3x, and then H1 lost lock). Naoki was investigating at the time of the H1 lockloss. He mentioned finding an issue and has updated the guardian and will make an alog.
Additionally: WIND
About 2hrs ago winds picked up very fast (from max speeds of 10mph up to current 25mph). See attached.
Currently trying to get through Green Arms---X-arm was not good and INCREASE FLASHES was needed a few times.
Vicky also taking a look at Squeezer World, too.
Also having gusts close to 40mph. H1 couldn't get past FINDIR, so running another alignment (crossing fingers for SRC this time).
After being down about 4.5hrs, H1 is finally back post-south-pacific EQ (there was a short Observing stretch, but it only lasted a few minutes after a lockloss as the world was still rumbling---Picket Fence was noisy).
Ran an Initial Alignment due to DRMI locking issues. SRC OFFLOADED had to be skipped since SRY would not acquire. But H1 locked up on its first attempt after the partial Initial Alignment.
TITLE: 04/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 10mph Gusts, 6mph 5min avg
Primary useism: 0.09 μm/s
Secondary useism: 0.14 μm/s
QUICK SUMMARY:
Walked in as TJ was almost done recovering H1 from the 3-Detector-Shaking south pacific earthquake. Nuc33 had another DTT for SQZ crash (will rerun the startup script again, but TJ mentioned it has been crashing fairly regularly).....and....H1 just went back to OBSERVING!
TITLE: 04/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Two lock losses, one unknown and one from an earthquake. Relocking has been straight forward. The first relock I touch SRM to lock DRMI and to avoid an initial alignment. The second relock that is finishing up has been full auto.
LOG:
Lock loss from a 6.5M Earthquake near Papua New Guinea. Looks like it was just over what we could handle.
Sun Apr 14 10:13:38 2024 INFO: Fill completed in 13min 34secs
No obvious reason for the lock loss that ended the lock just shy of 26 hours. The lock loss tool doesn't seem to be working at the moment.
Back to Observing at 1655.
I intervened at DRMI locking to avoid an initial alignment by moving SRM to get DRMI to catch.
TITLE: 04/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY: Locked for 25.5 hours. Some great triple coincidence over the last 24 hours.
TITLE: 04/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
H1's been locked for ~17.5hrs (and currently observing for ~12hrs).
LOG:
H1 is humming along with a lock over 13.5hrs (and a 6.5+hr triple coincidence stretch). Winds are low and microseism has been hovering over the 50th percentile the last couple shifts.
Jennie W, Sheila, Jenne
We did steps of 1 up in PITCH for camera offset servo 2 which controls the beam spot on ETMX by moving ITMX and ETMX, as suggested by this overnight test Camilla and I did.
This improved the build-ups and we went past the optimum point which we found to be CAM_PIT2_OFFSET = -173, so we set the offset to this. The optimum point is shown by the right most cursor in this image.
We did steps of 1 up in YAW for camera offset servo 3 which controls the beam spot on ETMY by moving ITMY and ETMY, as suggested by this overnight test Camilla and I did.
This improved the build-ups and we went past the optimum point which we found to be CAM_YAW3_OFFSET = -349.5, so we set the offset to this. The optimum point is shown by the right most cursor in this image. There was confusion as the PITCH and YAW degrees are cross-coupled (this confused us as one must wait for the pitch servo to converge while changing the yaw degree of freedom).
We then tried to optimise our last DOF for the camera servos: PITCH for servo 3. We first moved the offset up and then down, but made the build-ups worse each time. By this point we had increased thje gain of the camera servo filter banks by 2 to speed up the converging. CAM_PIT3_OFFSET should be set to its nominal at -230 counts.
Then we ran the A2L gain scripts we used the other day successfully to optimise thse loops to match the new arm alignment. During running of this script we lost lock, not sure why.
We did not add these new offsets to the guardian because of the lock loss, we will do them together with the A2L optimisation during the nest commissioning period.
The A2L script is is userapps/isc/h1/scripts and is called run_all_a2l.sh
Looking at the lockloss while running the A2L script:
It completed the excitations and gain adjustment for ITMX and ITMY yaw, but lost lock during the Y2L exctitation for ETMX. Next time we get a chance to try this, we could try running ETMX Yaw with a much smaller excitation and ask the code not to set the gain automatically.
The ndscope template used to make this screenshot is in sheila.dwyer/ndscope/ASC/A2L_script.yml
Returned to Observing at 16:10 UTC. The high frequency squeezing doesn't look great but it will get better as we thermalize?
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1397224003