Monthly FAMIS Check (#26491)

T240 Centering Script Output:

Averaging Mass Centering channels for 10 [sec] ...
2024-07-05 10:59:44.259990

There are 15 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -0.56 [V]
ETMX T240 2 DOF Y/V = -0.401 [V]
ETMX T240 2 DOF Z/W = -0.481 [V]
ITMX T240 1 DOF X/U = -1.362 [V]
ITMX T240 1 DOF Y/V = 0.317 [V]
ITMX T240 1 DOF Z/W = 0.415 [V]
ITMX T240 3 DOF X/U = -1.42 [V]
ITMY T240 3 DOF X/U = -0.713 [V]
ITMY T240 3 DOF Z/W = -1.737 [V]
BS T240 1 DOF Y/V = -0.386 [V]
BS T240 3 DOF Y/V = -0.343 [V]
BS T240 3 DOF Z/W = -0.485 [V]
HAM8 1 DOF X/U = -0.312 [V]
HAM8 1 DOF Y/V = -0.481 [V]
HAM8 1 DOF Z/W = -0.774 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.132 [V]
ETMX T240 1 DOF Y/V = -0.101 [V]
ETMX T240 1 DOF Z/W = -0.157 [V]
ETMX T240 3 DOF X/U = -0.098 [V]
ETMX T240 3 DOF Y/V = -0.226 [V]
ETMX T240 3 DOF Z/W = -0.091 [V]
ETMY T240 1 DOF X/U = 0.04 [V]
ETMY T240 1 DOF Y/V = 0.091 [V]
ETMY T240 1 DOF Z/W = 0.153 [V]
ETMY T240 2 DOF X/U = -0.094 [V]
ETMY T240 2 DOF Y/V = 0.158 [V]
ETMY T240 2 DOF Z/W = 0.064 [V]
ETMY T240 3 DOF X/U = 0.165 [V]
ETMY T240 3 DOF Y/V = 0.062 [V]
ETMY T240 3 DOF Z/W = 0.095 [V]
ITMX T240 2 DOF X/U = 0.122 [V]
ITMX T240 2 DOF Y/V = 0.214 [V]
ITMX T240 2 DOF Z/W = 0.203 [V]
ITMX T240 3 DOF Y/V = 0.109 [V]
ITMX T240 3 DOF Z/W = 0.118 [V]
ITMY T240 1 DOF X/U = 0.037 [V]
ITMY T240 1 DOF Y/V = 0.044 [V]
ITMY T240 1 DOF Z/W = -0.066 [V]
ITMY T240 2 DOF X/U = 0.049 [V]
ITMY T240 2 DOF Y/V = 0.193 [V]
ITMY T240 2 DOF Z/W = 0.03 [V]
ITMY T240 3 DOF Y/V = 0.023 [V]
BS T240 1 DOF X/U = -0.204 [V]
BS T240 1 DOF Z/W = 0.089 [V]
BS T240 2 DOF X/U = -0.09 [V]
BS T240 2 DOF Y/V = 0.009 [V]
BS T240 2 DOF Z/W = -0.162 [V]
BS T240 3 DOF X/U = -0.202 [V]

Assessment complete.

STS Centering Script Output:

Averaging Mass Centering channels for 10 [sec] ...

2024-07-05 11:02:32.261843
There are 2 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -4.008 [V]
STS EY DOF Z/W = 2.765 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.507 [V]
STS A DOF Y/V = -0.728 [V]
STS A DOF Z/W = -0.647 [V]
STS B DOF X/U = 0.377 [V]
STS B DOF Y/V = 0.94 [V]
STS B DOF Z/W = -0.492 [V]
STS C DOF X/U = -0.655 [V]
STS C DOF Y/V = 0.894 [V]
STS C DOF Z/W = 0.344 [V]
STS EX DOF X/U = -0.06 [V]
STS EX DOF Y/V = 0.017 [V]
STS EX DOF Z/W = 0.087 [V]
STS EY DOF Y/V = 0.025 [V]
STS FC DOF X/U = 0.239 [V]
STS FC DOF Y/V = -1.056 [V]
STS FC DOF Z/W = 0.644 [V]

Assessment complete.

For FAMIS 26263:
Laser Status:
    NPRO output power is 1.818W (nominal ~2W)
    AMP1 output power is 66.81W (nominal ~70W)
    AMP2 output power is 136.9W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 2 days, 1 hr 53 minutes
    Reflected power = 20.62W
    Transmitted power = 104.8W
    PowerSum = 125.4W

FSS:
    It has been locked for 0 days 0 hr and 44 min
    TPD[V] = 0.6824V

ISS:
    The diffracted power is around 2.5%
    Last saturation event was 0 days 0 hours and 44 minutes ago

Possible Issues:
    PMC reflected power is high
    FSS TPD is low

Bruco ran for last night's 159MPc range (instructions from Elenna), using command below. Results here.

python -m bruco --ifo=H1 --channel=GDS-CALIB_STRAIN_CLEAN --gpsb=1404222629 --length=1000 --outfs=4096 --fres=0.1 --dir=/home/camilla.compton/public_html/brucos/GDS_CLEAN_1404222629 --top=100 --webtop=20 --plot=html --nproc=20 --xlim=7:2000 --excluded=/home/elenna.capote/bruco-excluded/lho_DARM_excluded.txt

Can see:

• CHARD_P is bad 10-30Hz and CHARD_Y is very bad < 25Hz
• PRCL bad coherence 10-20Hz 
• SRCL looks good and MICH only has a little coherence 10-20Hz 
• OMC ASC signals dominate <16Hz, maybe this is expected....  
• HAM1 signals are high 10-30Hz, from PRCL coupling?

Commissioning started at 1600utc/9amlocal, but at 1637utc a magnitude 5.0 earthquake near British Columbia knocked h1 out of lock (this is where we have been having earthquakes the last day or so).

TITLE: 07/05 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

H1's been locked 2.25hrs with 2-3hr lock stretches starting about 15hrs ago (with issues before that as noted in previous logs & mainly post-tues-Maintenance this week).  Seeing small EQ spikes in recent hours after the big EQ about 22hrs ago.  Low winds and microseism.  On the drive in, there was a thin layer of smoke on the horizon in about all directions post-4th of July with no active/nearby plumes observed.

H1 called for assistance at 08:45 UTC because it was able to lock up to NLN, but could not inject squeezing due to an error with the SQZ TTFSS. The specific error it reported was "Fiber trans PD error," then on the fiber trans screen it showed a "Power limit exceeded" message. The input power to the TTFSS (SQZ-FIBR_TRANS_DC_POWER) was indeed too high at 0.42mW where the high power limit was set at 0.40mW. Trending this back a few days, it seems that the power jumped up in the morning on July 3rd (I suspect when the fiber pickoff in the PSL was aligned) and it has been floating around that high power limit ever since. I'm not exactly sure why this time it was an issue, as we've had several hours of observing time since then.

I raised the high power limit from 0.40mW to 0.45mW, the TTFSS was able to lock without issue, SQZ_MANAGER brought all nodes up, and squeezing was injected as usual. I then accepted the new high power limit in SDF (attached) for H1 to start observing at 09:20 UTC.

Since this feels like a Band-Aid solution just to get H1 observing tonight, I encourage someone with more knowledge of the SQZ TTFSS to look into it as time allows.

TITLE: 07/05 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: An earthquake lockloss then a PI lockloss. Currently at MAX_POWER

Lock1:

• 23:25 UTC reacquired NLN, back in Observing at 23:28 after accepting some SDF diffs for ALSX
• 01:19, 01:37, and 02:12 UTC Dropped out of observing for a few seconds due to 2 SDF diffs on syscssqz (from DIAG_SDF log), I didn't catch what the channels were.
  • It seems like its a problem with the SQZ_FIBR_TRANS (Tagging SQZ)
  • It started happening again at 04:14 UTC and I was able to catch the channels, a trend of the {FAST,COM}GAINS. I unmonitored them for now
• 02:13 UTC lockloss from another earthquake, a 2.6 from Alaska?

Lock2:

• XARM is having fiber polarization issues now, H1:ALS-X_FIBR_LOCK_FIBER_POLARIZATIONPERCENT can't get below 14 (ALS_OVERVIEW -> POLARIZATION CORRECTION BOX), Sheila suggested H1:ALS-X_FIBR_LOCK_LOGIC_FORCE which did the trick for now until the root issue can be solved
• Lost it at ALS then I didn't have the issue the 2nd try
• 05:32 UTC superevents20240405at, didn;t get a verbal callout for this
• 06:37 UTC lockloss from a PI ringup

Lock3:

• ALSX fiber error is back, toggling force on the PLL doesn't do anything now. I toggled force on the PDH as well which locked it
• GRD couldnt get the POL below 22 after a bit of trying, I took over (paused ALS_XARM) and got it to 15 somehow, I unpaused it and it was finally able to get out of CHANGE_POL / FAULT
• 07:23 DRMI lockloss at CARM_150_PICO

To recap for SQZ, I have unmonitored 3 SQZ channels on syscssqz (H1:SQZ-FIBR_SERVO_COMGAIN, H1:SQZ-FIBR_SERVO_FASTGAIN, H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_ON) that keep dropping us out of observing for now until their root issue can be fixed (Fiber trans PD error, too much power on FIBR_TRANS?). I noticed that each time the GAINS change it also drops our cleaned range

PI ring up lockloss? PI28 and 29 were unable to be damped down by the GRD and we eventually lost lock.

https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1404196646

Closes FAMIS25997, last checked in alog78550

ITMX_ST2_CPSINF_H1 has gotten noisier at high frequency

Everything else looks the same as previously.

Lost lock from an earthquake

It seems earthquakes causing similar magnitudes of movement on-site may or may not cause lockloss. Why is this happening? Should expect to always or never cause lockloss for similar events. One suspicion is that common or differential motion might lend itself better to keeping or breaking lock.

- Lockloss is defined as H1:DRD-ISC_LOCK_STATE_N going to 0 (or near 0).
- I correlated H1:DRD-ISC_LOCK_STATE_N with H1:ISI-GND_STS_CS_Z_EQ_PEAK_OUTMON peaks between 500 and 2500 μm/s.
- I manually scrolled through the data from present to 2 May 2024 to find events.
    - Manual, because 1) wanted to start with a small sample size and quickly see if there was a pattern, and 2) because I need to find events that caused loss, then go and find similarly sized events we kept lock.
- Channels I looked at include:
    - IMC-REFL_SERVO_SPLITMON
    - GRD-ISC_LOCK_STATE_N
    - ISI-GND_STS_CS_Z_EQ_PEAK_OUTMON ("CS_PEAK")
    - SEI-CARM_GNDBLRMS_30M_100M
    - SEI-DARM_GNDBLRMS_30M_100M
    - SEI-XARM_GNDBLRMS_30M_100M
    - SEI-YARM_GNDBLRMS_30M_100M
    - SEI-CARM_GNDBLRMS_100M_300M
    - SEI-DARM_GNDBLRMS_100M_300M
    - SEI-XARM_GNDBLRMS_100M_300M
    - SEI-YARM_GNDBLRMS_100M_300M
    - ISI-GND_STS_ITMY_X_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_Y_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_Z_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_X_BLRMS_100M_300M
    - ISI-GND_STS_ITMY_Y_BLRMS_100M_300M
    - ISI-GND_STS_ITMY_Z_BLRMS_100M_300M
    - SUS-SRM_M3_COILOUTF_LL_INMON
    - SUS-SRM_M3_COILOUTF_LR_INMON
    - SUS-SRM_M3_COILOUTF_UL_INMON
    - SUS-SRM_M3_COILOUTF_UR_INMON
    - SUS-PRM_M3_COILOUTF_LL_INMON
    - SUS-PRM_M3_COILOUTF_LR_INMON
    - SUS-PRM_M3_COILOUTF_UL_INMON
    - SUS-PRM_M3_COILOUTF_UR_INMON

        - ndscope template saved as neil_eq_temp2.yaml

- 26 events; 14 lockloss, 12 locked (3 or 4 lockloss event may have non-seismic causes)

- After, usiing CS_PEAK to find the events, I, so far, used the ISI channels to analyse the events.
    - The SEI channels were created last week (only 2 events captured in these channels, so far).

- Conclusions:
    - There are 6, CS_PEAK events above 1,000 μm/s in which we *lost* lock;
        - In SEI 30M-100M
            - 4 have z-axis dominant motion with no motion or strong z-motion or no motion in SEI 100M-300M
            - 2 have y-axis dominated motion with a lot of activity in SEI 100M-300M and y-motion dominating some of the time.
    - There are 6, CS_PEAK events above 1,000 μm/s in which we *kept* lock;
        - In SEI 30M-100M
            - 5 have z-axis dominant motion with only general noise in SEI 100M-300M
            - 1 has z-axis dominant noise near the peak in CS_PEAK and strong y-axis domaniated motion starting 4 min prior to the CS_PEAK peak; it too only has general noise in SEI 100M-300M. This x- or y-motion which starts about 4 min before the peak in CS_PEAK has been observed in 5 events -- Love waves precede Rayleigh waves, could be Love waves?
    - All events below 1000 μm/s which lose lock seem to have a dominant y-motion in either/both SEI 30M-100M / 100M-300M. However, the sample size is not large enough to convince me that shear motion is what is causing lockloss. But it is large enough to convince me to find more events and verify. (Some plots attached.)

TITLE: 07/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

Today H1 was down for ALS maintenance and replacement of ALS X PD as described in Daniel's alog (78864) . 
once they returned I started an Initial_Alignment and then started locking.
Observing was reached at 23:28 UTC.
There have been a number of earthquakes right off the coast of Victoria Island B.C. today.

LOG:                                                                                                                                                                                                                                                                                                 

TITLE: 07/04 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 4mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s 
QUICK SUMMARY:

• Relocking at POWER_25W
• We just finished a IA before this lock attempt and everything seems to be going well
• 2nd order violins look a little high

Keita Daniel

We found that the transimpedance gain of the ALS-X_FIBR_A_DC PD was wrong (changed it from 20k to 2k). In turn, this meant that 20mW of light was on this PD.

After looking at the beatnote amplitude directly at the PD and found it to be way too small, we decided to swap the PD with a spare (new PD S/N S1200248, old PD S/N S1200251). However, this did not improve the beatnote amplitude. (The removed PD was put back into the spares cabinet.)

We then looked for clipping and found that the beam on the first beam sampler after the fiber port was close to the side. We moved the sampler so the beam is closer to the center of the optics. We also found the beam on the polarizing cube in the fiber path to be low. We moved the cube downwards to center the beam. After aligning the beam back to the broadband PD, the beatnote amplitude improved drastically. This alignment seems very sensitive.

We had to turn the power from the laser in the beat note path down from 20mW to about 6mW on the broadband PD.

This required a recalibration of the ALS-X_LASER_IR_PD photodiode. The laser output power in IR is about 60mW.

The beatnote strength as read by the medm screens is now 4-7dBm. Still seems to vary.

6 channels were accepted in the SDF diffs after the ALS adjustments done today.