TITLE: 07/06 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

Lockloss 1:06 UTC
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1404263217
Lockloss Screenshots attached

Relocking:
After the Lockloss I had pretty small flashes on X arm.
I allowed Increase flashes to run and it didn't get me better than 0.3.
I then touched it up by hand and could not get it better than 3.1, Trending back I think the goal is above 1.
I then tried to get better alignment by rolling back to an alignment from the beggining of the last lock.

I tried the Alignment after the Last initial alignment.

Im going to try to move just PR2 now.
Revert all movements back to right after we lost lock.

Sheila did a small Pico motor move in HAM3.
Pico Motor ALS/Pop steering HAM1 ( Actually in HAM3).
H1:ALS-C_TRX_A_LF_GAIN was increased temporarily to make the X arm WFS run.

And Sheila did another move of ALS/Pop steering once the WFS were running.

Note the H1:ALS-X_FIBR_A_DEMOD_RFMON beat note dropped down to -38 and the threshold was lowered to -43.

Once this was done we could do an Initial Alignment, BUT we did not have anything on AS Air
Moved IM4 & PRM to get light on AS Air and Refl PRM cam.
Sheila used IM4 PRM & PR2 Osems to match prior OSEM values to do a "manual WFS releave past" to Move PR2 which gave us increased IR flashes.

Touched up PRM in Yaw to lock PRX

Finished Initial Alignment at 5:19 UTC

Locking was being difficult and would lockloss at FIND IR and Locking ALS.
I tried giving another Initial Alignment after it failed a number of time because we did touch it up from hand.
Even after that IA  it was still locklossing at Find IR and Lockling ALS. Paused in Locking ALS to allow the WFS to calm down.
Yeah ALS WFS DOF 2 is pulling it away for some reason. But even trying to allow the WFS to melow out, it still catches a Lockloss.
Finally got past DRMI !!! YAY!!!

LOCKLOSS!? From MAX POWER!?
7:30 UTC Random HEPI HAM1 Watchdog trip.
IOP SUS56, 34, & 23 all had a IOP DACkill trip at the same time.
Seems like sush2a Had DAC error calling in CDS team.

CDS team is resetting all the SUS front end models because everything from HAM1 to HAM6 tripped in this timing glitch. ****

LOG:
Sheila remotely Helped get me a good Alignment and got me through a rouch IA
Dave B & Erik helped restart all the Front Ends.
Jim was also called due to a HEPI trip and he was next on the call list.

Every one has been cycled to the bottom of the list.

Tony, Jim, Erik, Dave:

We had a timing error which caused DACKILLs on h1susb123, h1sush34, h1sush56 and DAC_FIFO errors on h1sush2a.

There was no obvious cause of the timing error which caused the Dolphin glitch, we noted that h1calcs was the only model with a DAQ CRC error (see attached).

After diag-reset and crc-reset only the SUS dackill and fifo error persisted. 

We restarted all the models on h1susb123, h1sush2a, h1sush34, h1sush56 after bypassing the SWWD SEI systems for BSC1,2,3 and HAM2,3,4,5,6.

SUS models came back OK, we removed the SEI SWWD bypasses and handed the system over to Tony.

TITLE: 07/05 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 9mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY:
H1 has been locked and Observing for over 2 hours. Everything seems to be functioning well.

TITLE: 07/05 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

Most of today was supposed to be a straightforward 3hrs of commissioning, but an earthqake lockloss and another random lockloss forced a majority of the shift to be locking/commissioning time.  For commissioning there was some PR2 Spot moves, and sheila made a note about possible/probable PR2 & IM4 (pit) adjustments needed for INPUT ALIGN part of initial alignment.
LOG:

• 1601-1605 Removing black glass from HAM3 for PR2 spot move (Robert)
  • 2120 Reinstalling black glass (robert)
• 1604 PR2 spot move (sheila)
• 1626-1812 Isaiah running X & then Y arms
• 1637 LOCKLOSS due to M5.0 BC coast earthquake
  • 1817 back to NLN
• 1813-1839 Checking if power meter set-up will work (robert)
• 1829 PR2 spot move attempt#2 (sheila)
• 1829 LOCKLOSS (Sheila had NOT started PR2 move, Robert was out on the floor with his power meter set-up and was doing lesser invasive actions compared to other work he had done.  So not sure on this lockloss.)
  • xxxx Back to NLN
• 1923 Power meter set-up again (robert)
• 2055 Pcal lab work (francisco)
• 2100 Keita shared a photo of a head/skull of an animal (first guess was a young badger?)
• 2121 Squeezing in scan squeezer alignment at the end of the commissioning (camilla)
• 2130 Back to OBSERVING
• 2159-2205 Squeezer dropped us out of Observing.  Took the Manager DOWN & back to FDS.  I was slow to address this as I stepped out and then was chatting with someone when I returned and did not hear Verbal.
• 2232 Optics lab work (camilla)

Robert, Sheila, Camilla, Corey

Today we moved the spot on PR2, which reduced the power measured at the HAM3 viewport and the appearance of the 48 Hz peak in DARM when the black glass is removed from that viewport.

In our first lock of the day, Robert removed black glass: 16:08 UTC, we took ~20 minutes of quiet time with the black glass removed, before loosing lock due to an earthquake.

We decided to move 42 urad based on the first screenshot shows the May move described in 77949, the arm powers improved in the first 42urad of PR3 yaw move  After relocking we moved PR2 by 42 urad, while the IFO was thermalizing (screenshot).  Ran the A2L script and got a few minutes of quiet time there, starting at 20:20-20:31 UTC.  We then decided to take another step to see if we could reduce the power in the HAM2 beam further, we moved an additional 10urad on PR3 yaw, and saw a small decrease in the beam power.  We re-ran A2L here (added values to lscparams and loaded guardian), and took some time with the illuminator on the viewport to check the height of the peak.  Camilla noticed that the squeezing didn't look optimal, so she's running SQZ alignment and angle scans.  There is some LSC coherence, mostly with PRCL.  We will plan to add PRCL FF next week, and re tune MICH and SRCL.

Robert measured power in the beam exiting the HAM3 viewport at a few steps:  the original position: 47 mW, -20 urad yaw: 28 mW; -42 urad 19 mW; -52 urad: 17mW.

Old alogs about this move:

• Attempt to move closer to center while unlocked (reverted): 77968
• First move, when the squeezer whistles/FC motion coincidentally appeared and made DARM noise bad: 77870 
• reverting that move because of SQZ issue: 77895
• In lock move of PR2 spot: 77949
• reverting PR2 spot move: 78012

Operator note:  We moved these beams using a script that moves sliders on several optics, so that ideally we should be able to relock without doing any significant alignment.  For yaw, the script was recently adjusted so that it really should work well, for pitch there may be some manual alignment of PR2 + IM4 needed to get initial alignment to lock the X arm in IR. 

First run at 10:00 did not complete, its thermocouples reached the trip temp of -130C before we got a flow of LN2.

Fri Jul 05 10:17:33 2024 INFO: Fill completed in 17min 29secs

We re-ran the fill at 12:00 with a lowered trip temp of -140C, this run was successful

Fri Jul 05 12:04:36 2024 INFO: Fill completed in 4min 33secs

Trending the trip temps since this system was installed two years ago, we have never had to run with trip temps as low as -140C (summer settings have been -130C).

Tomorrow we will run the fill at the earlier time of 08:00 in case outside temps were a factor in today's issue. I have set the trip temp back to -130C.

Commissioning has had a rough time due to a big Earthquake (and then a mystery lockloss).  H1 is currently recovering from the 2nd lockloss of this morning.  The hope is to return to Commissioning if timing works out (with the main task being PR2 spot moves).  
As for locking, it's been fairly straightforward.

Richard, Keita, Camilla

Richard questioned a 4e-9 pressure rise when we heated up OM2 in June 78573 (plot). The first time we heated OM2 after the O3/O4 vent in November 2022 (plot) the spike was larger 9e-9. Future times we heated up OM2 in O4a (July and September) the rise was smaller. 

Keita says that it is expected that the first time you heat something up after a vent the pressure can rise as particles from venting are burnt off, future OM2 heating we'd expect smaller pressure rises.

H1 has recovered from the recent BC aftershock.  One SDF diff which came up was an LSC X-Arm TRAMP.  Reverted back to the setpoint of 0.0 (from 1.0).  Not really sure why this changed---at the time, H1 was Check Mich Fringing.

Monthly FAMIS Check (#26491)

T240 Centering Script Output:

Averaging Mass Centering channels for 10 [sec] ...
2024-07-05 10:59:44.259990

There are 15 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -0.56 [V]
ETMX T240 2 DOF Y/V = -0.401 [V]
ETMX T240 2 DOF Z/W = -0.481 [V]
ITMX T240 1 DOF X/U = -1.362 [V]
ITMX T240 1 DOF Y/V = 0.317 [V]
ITMX T240 1 DOF Z/W = 0.415 [V]
ITMX T240 3 DOF X/U = -1.42 [V]
ITMY T240 3 DOF X/U = -0.713 [V]
ITMY T240 3 DOF Z/W = -1.737 [V]
BS T240 1 DOF Y/V = -0.386 [V]
BS T240 3 DOF Y/V = -0.343 [V]
BS T240 3 DOF Z/W = -0.485 [V]
HAM8 1 DOF X/U = -0.312 [V]
HAM8 1 DOF Y/V = -0.481 [V]
HAM8 1 DOF Z/W = -0.774 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.132 [V]
ETMX T240 1 DOF Y/V = -0.101 [V]
ETMX T240 1 DOF Z/W = -0.157 [V]
ETMX T240 3 DOF X/U = -0.098 [V]
ETMX T240 3 DOF Y/V = -0.226 [V]
ETMX T240 3 DOF Z/W = -0.091 [V]
ETMY T240 1 DOF X/U = 0.04 [V]
ETMY T240 1 DOF Y/V = 0.091 [V]
ETMY T240 1 DOF Z/W = 0.153 [V]
ETMY T240 2 DOF X/U = -0.094 [V]
ETMY T240 2 DOF Y/V = 0.158 [V]
ETMY T240 2 DOF Z/W = 0.064 [V]
ETMY T240 3 DOF X/U = 0.165 [V]
ETMY T240 3 DOF Y/V = 0.062 [V]
ETMY T240 3 DOF Z/W = 0.095 [V]
ITMX T240 2 DOF X/U = 0.122 [V]
ITMX T240 2 DOF Y/V = 0.214 [V]
ITMX T240 2 DOF Z/W = 0.203 [V]
ITMX T240 3 DOF Y/V = 0.109 [V]
ITMX T240 3 DOF Z/W = 0.118 [V]
ITMY T240 1 DOF X/U = 0.037 [V]
ITMY T240 1 DOF Y/V = 0.044 [V]
ITMY T240 1 DOF Z/W = -0.066 [V]
ITMY T240 2 DOF X/U = 0.049 [V]
ITMY T240 2 DOF Y/V = 0.193 [V]
ITMY T240 2 DOF Z/W = 0.03 [V]
ITMY T240 3 DOF Y/V = 0.023 [V]
BS T240 1 DOF X/U = -0.204 [V]
BS T240 1 DOF Z/W = 0.089 [V]
BS T240 2 DOF X/U = -0.09 [V]
BS T240 2 DOF Y/V = 0.009 [V]
BS T240 2 DOF Z/W = -0.162 [V]
BS T240 3 DOF X/U = -0.202 [V]

Assessment complete.

STS Centering Script Output:

Averaging Mass Centering channels for 10 [sec] ...

2024-07-05 11:02:32.261843
There are 2 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -4.008 [V]
STS EY DOF Z/W = 2.765 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.507 [V]
STS A DOF Y/V = -0.728 [V]
STS A DOF Z/W = -0.647 [V]
STS B DOF X/U = 0.377 [V]
STS B DOF Y/V = 0.94 [V]
STS B DOF Z/W = -0.492 [V]
STS C DOF X/U = -0.655 [V]
STS C DOF Y/V = 0.894 [V]
STS C DOF Z/W = 0.344 [V]
STS EX DOF X/U = -0.06 [V]
STS EX DOF Y/V = 0.017 [V]
STS EX DOF Z/W = 0.087 [V]
STS EY DOF Y/V = 0.025 [V]
STS FC DOF X/U = 0.239 [V]
STS FC DOF Y/V = -1.056 [V]
STS FC DOF Z/W = 0.644 [V]

Assessment complete.

For FAMIS 26263:
Laser Status:
    NPRO output power is 1.818W (nominal ~2W)
    AMP1 output power is 66.81W (nominal ~70W)
    AMP2 output power is 136.9W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 2 days, 1 hr 53 minutes
    Reflected power = 20.62W
    Transmitted power = 104.8W
    PowerSum = 125.4W

FSS:
    It has been locked for 0 days 0 hr and 44 min
    TPD[V] = 0.6824V

ISS:
    The diffracted power is around 2.5%
    Last saturation event was 0 days 0 hours and 44 minutes ago

Possible Issues:
    PMC reflected power is high
    FSS TPD is low

Bruco ran for last night's 159MPc range (instructions from Elenna), using command below. Results here.

python -m bruco --ifo=H1 --channel=GDS-CALIB_STRAIN_CLEAN --gpsb=1404222629 --length=1000 --outfs=4096 --fres=0.1 --dir=/home/camilla.compton/public_html/brucos/GDS_CLEAN_1404222629 --top=100 --webtop=20 --plot=html --nproc=20 --xlim=7:2000 --excluded=/home/elenna.capote/bruco-excluded/lho_DARM_excluded.txt

Can see:

• CHARD_P is bad 10-30Hz and CHARD_Y is very bad < 25Hz
• PRCL bad coherence 10-20Hz 
• SRCL looks good and MICH only has a little coherence 10-20Hz 
• OMC ASC signals dominate <16Hz, maybe this is expected....  
• HAM1 signals are high 10-30Hz, from PRCL coupling?

Commissioning started at 1600utc/9amlocal, but at 1637utc a magnitude 5.0 earthquake near British Columbia knocked h1 out of lock (this is where we have been having earthquakes the last day or so).

TITLE: 07/05 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

H1's been locked 2.25hrs with 2-3hr lock stretches starting about 15hrs ago (with issues before that as noted in previous logs & mainly post-tues-Maintenance this week).  Seeing small EQ spikes in recent hours after the big EQ about 22hrs ago.  Low winds and microseism.  On the drive in, there was a thin layer of smoke on the horizon in about all directions post-4th of July with no active/nearby plumes observed.

H1 called for assistance at 08:45 UTC because it was able to lock up to NLN, but could not inject squeezing due to an error with the SQZ TTFSS. The specific error it reported was "Fiber trans PD error," then on the fiber trans screen it showed a "Power limit exceeded" message. The input power to the TTFSS (SQZ-FIBR_TRANS_DC_POWER) was indeed too high at 0.42mW where the high power limit was set at 0.40mW. Trending this back a few days, it seems that the power jumped up in the morning on July 3rd (I suspect when the fiber pickoff in the PSL was aligned) and it has been floating around that high power limit ever since. I'm not exactly sure why this time it was an issue, as we've had several hours of observing time since then.

I raised the high power limit from 0.40mW to 0.45mW, the TTFSS was able to lock without issue, SQZ_MANAGER brought all nodes up, and squeezing was injected as usual. I then accepted the new high power limit in SDF (attached) for H1 to start observing at 09:20 UTC.

Since this feels like a Band-Aid solution just to get H1 observing tonight, I encourage someone with more knowledge of the SQZ TTFSS to look into it as time allows.

TITLE: 07/05 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: An earthquake lockloss then a PI lockloss. Currently at MAX_POWER

Lock1:

• 23:25 UTC reacquired NLN, back in Observing at 23:28 after accepting some SDF diffs for ALSX
• 01:19, 01:37, and 02:12 UTC Dropped out of observing for a few seconds due to 2 SDF diffs on syscssqz (from DIAG_SDF log), I didn't catch what the channels were.
  • It seems like its a problem with the SQZ_FIBR_TRANS (Tagging SQZ)
  • It started happening again at 04:14 UTC and I was able to catch the channels, a trend of the {FAST,COM}GAINS. I unmonitored them for now
• 02:13 UTC lockloss from another earthquake, a 2.6 from Alaska?

Lock2:

• XARM is having fiber polarization issues now, H1:ALS-X_FIBR_LOCK_FIBER_POLARIZATIONPERCENT can't get below 14 (ALS_OVERVIEW -> POLARIZATION CORRECTION BOX), Sheila suggested H1:ALS-X_FIBR_LOCK_LOGIC_FORCE which did the trick for now until the root issue can be solved
• Lost it at ALS then I didn't have the issue the 2nd try
• 05:32 UTC superevents20240405at, didn;t get a verbal callout for this
• 06:37 UTC lockloss from a PI ringup

Lock3:

• ALSX fiber error is back, toggling force on the PLL doesn't do anything now. I toggled force on the PDH as well which locked it
• GRD couldnt get the POL below 22 after a bit of trying, I took over (paused ALS_XARM) and got it to 15 somehow, I unpaused it and it was finally able to get out of CHANGE_POL / FAULT
• 07:23 DRMI lockloss at CARM_150_PICO

To recap for SQZ, I have unmonitored 3 SQZ channels on syscssqz (H1:SQZ-FIBR_SERVO_COMGAIN, H1:SQZ-FIBR_SERVO_FASTGAIN, H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_ON) that keep dropping us out of observing for now until their root issue can be fixed (Fiber trans PD error, too much power on FIBR_TRANS?). I noticed that each time the GAINS change it also drops our cleaned range

Keita Daniel

We found that the transimpedance gain of the ALS-X_FIBR_A_DC PD was wrong (changed it from 20k to 2k). In turn, this meant that 20mW of light was on this PD.

After looking at the beatnote amplitude directly at the PD and found it to be way too small, we decided to swap the PD with a spare (new PD S/N S1200248, old PD S/N S1200251). However, this did not improve the beatnote amplitude. (The removed PD was put back into the spares cabinet.)

We then looked for clipping and found that the beam on the first beam sampler after the fiber port was close to the side. We moved the sampler so the beam is closer to the center of the optics. We also found the beam on the polarizing cube in the fiber path to be low. We moved the cube downwards to center the beam. After aligning the beam back to the broadband PD, the beatnote amplitude improved drastically. This alignment seems very sensitive.

We had to turn the power from the laser in the beat note path down from 20mW to about 6mW on the broadband PD.

This required a recalibration of the ALS-X_LASER_IR_PD photodiode. The laser output power in IR is about 60mW.

The beatnote strength as read by the medm screens is now 4-7dBm. Still seems to vary.