J. Kissel, L. Dartez, F. Llamas

As Louis and I were wracking our brains for all the changes to the IFO that might have impacted the IFO's calibration in 2024 in order to update / maintain T2300297, we realized that Francisco's work -- changing the PCAL "inner beam" spot position of PCALX to explore and validate models of systematic errors the PCAL system -- would potentially impact the measurements that are constantly running on PCALX -- the so-called "roaming lines" which is a really long duration, repeating sweep of the sensing function between 1 and 5 kHz. Because this long-duration sweep relies on nominal low noise data, the speed at which the sweep frequency advances is dependent on the IFO's duty cycle, and is thus irregular; sometimes complete in 6 days, other times taking 10 days or more.

To facilitate this research, I plot the trend of the PCALX Oscillator Frequency that's used to define the sweep frequency over time and overlay Francisco, et. al's PCALX spot moves.

And just so you can join us in our worry -- while Francisco's work (see latest in LHO:78964) is showing 0.03%-level changes in the comparison between PCALX and PCALY at 283.91 and 284.01 Hz respectively (via plots like this), the roaming lines measurement is only PCALX and the sweep is from 1-5 kHz, where it's been known for a long time that PCAL spot position changes of ~5 [mm] at RX, or ~2.5 [mm] scale at the Test Mass, can cause 1-to-10%-level errors in reported displacement values at 3, 4, 5 kHz (largest at highest frequencies) -- see, for example, figures 2.30 through 2.35 in T2300381.

Stay tuned!

This aLOG does not suggest we've done the leg work to quantify that we've seen the a problem, but the start of doing such a study.

Ibrahim

Today, I went into the staging building to see if there were any visible or otherwise apparent reasons for our L, P, Y M1 Transfer Function incoherence.

Here's what I did:

• Backed off all EQ stops, none were too close to touching, but the temp drif sag was higher yesterday than today so I just made sure they were all the way back.
• Re-did the foil, double triple checking nothing was touching, rubbing or in contact
• Checked every BOSEM Flag

Here's what I found:

• M1 is visibly pitched - this is corroborated by the 8-day drift of the F1 OSEM counts (as opposed to the daily sag and rebound of the LF-RT OSEMs. I've copied over the F1 section of the drift screenshot from yesterday's alog 79042 on this topic, since it seems the pitch instability suggestion from that alog was the right hypotehesis (F1 is used in P and is above the M1 center of mass). Image 1 shoes this clearly.
  • This could either be differential sagging of the Top Stage blades causing it to pitch in one direction over another.
  • It could also be a pitch hysteresis issue we had with the Top Mass during assembly - this was fixed by setting the heights of the blades to a different nominal position.
  • It could also be that the the Top Mass blades are differentially sagging too, causing more pull on one side than the other.
  • Or a combo, or something else.
• The M1 flags are surprisingly not visibly sagging that much - I guess this makes sense since 25 microns (our measured offset from centered position last week) is only 0.025mm.
• The M2 flags are much more visibly sagging, which means that the Top Mass blades are more affected by the temperature drift. This sort of corroborates the idea that the Top Mass blade sag is affecting the pitch moreso than the top stage blades or a previous ptich hysteresis blade height problem. Image 2 shoes one of the 4 M2 BOSEMs substantially lower than their centered positon.

Here's what I'm going to to:

I'm going to set an offset in order to "fake" center the BOSEMs and then take data but I expect that with a Pitch instability the results won't be too coherent either. We'll see. We're at least now ensured that there won't be any rubbing or contact.

What I'm going to do later:

We will fix the percieveable Roll issue on Monday when there are more hands on deck. I'll also consider re-centering the OSEMs at noon and measuring the temp at that time so we have some sort of informed reference.

Laser Status:
    NPRO output power is 1.821W (nominal ~2W)
    AMP1 output power is 65.27W (nominal ~70W)
    AMP2 output power is 138.5W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 2 days, 23 hr 37 minutes
    Reflected power = 17.94W
    Transmitted power = 107.8W
    PowerSum = 125.8W

FSS:
    It has been locked for 0 days 0 hr and 49 min
    TPD[V] = 0.7949V

ISS:
    The diffracted power is around 3.2%
    Last saturation event was 0 days 0 hours and 49 minutes ago

Possible Issues:
    ISS diffracted power is high, seems like its been high since early morning this past maintenance.

16:44 UTC lockloss. PRCL was oscillating at about ~3.6 Hz.

Ryan C, Sheila D

15:32 to 16:01 UTC we dropped Observing to do some range checks/investigations.

Sheila and I did some range checks following the wiki. Running the coherence check showed high coherence with CHARD and the SQZer BLRMs did not look as good as previous locks, particularly in the 10-20, 20-34, and 60-100 HZ bandwidths. So we decided to drop out of observing since LLO was down to run the SQZ alignment and angle scans and I concurrently ran the A2L_min script (TJs alog78552). After these finished we also took 5 minutes of NO_SQZ time starting at 15:49 UTC which showed that the extra noise is not coming from the SQZer. We gained a few Mpcs from the SQZ scan and A2L_min script.

Coherence comparison before and after SQZ and A2L checks

Ran the (userapps)/isc/h1/scripts/a2l/a2l_min_multi.py script for all four quads in both P and Y to try and help our range. Minimal improvements, this was done in tandem with the SQZ scans which gained us about 2 Mpc.

ETMX P
Initial:  3.12
Final:    3.09
Diff:     -0.03

          ETMX Y
Initial:  4.79
Final:    4.81
Diff:     0.02

          ETMY P
Initial:  4.48
Final:    4.49
Diff:     0.01

          ETMY Y
Initial:  1.13
Final:    1.26
Diff:     0.13

          ITMX P
Initial:  -1.07
Final:    -1.02
Diff:     0.05

          ITMX Y
Initial:  2.72
Final:    2.79
Diff:     0.07

          ITMY P
Initial:  -0.47
Final:    -0.43
Diff:     0.04

          ITMY Y
Initial:  -2.3
Final:    -2.36
Diff:     -0.06

Fri Jul 12 08:08:03 2024 INFO: Fill completed in 8min 0secs

Jordan confirmed a good fill curbside.

TITLE: 07/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

• We've been locked and observing for 4:20, since 10:17 UTC
  • The lockloss overnight (08:24 UTC) was from an earthquake
• Lots of earthquakes overnight
• Range is ok, SQZing doesn't look the best from the trace on NUC33

We were only about 2 and half hours into lock when I did this test due to our earthquake lockloss this morning.

I ran the

python auto_darm_offset_step.py

in /ligo/gitcommon/labutils/darm_offset_step

Starting at GPS 1404768828

See attached image.

Analysis to follow.

Returned DARM offset H1:OMC-READOUT_X0_OFFSET to 10.941038 (nominal) at 2024 Jul 11 21:47:58 UTC (GPS 1404769696)

DARM offset moves recorded to 
data/darm_offset_steps_2024_Jul_11_21_33_30_UTC.txt

We're running a patched version of simuLines during the next calibration measurement run. The patch (attached) was provided by Erik to try and get around what we think are awg issues introduce (or exacerbated) by the recent awg server updates (mentioned in LHO:78757).

Operators: there's is nothing special to do. just follow the normal routine as I applied the patch changes in place. Depending on the results of this test, I will either roll them back or work with Vlad to make them permanent (at least for LHO).

We cannot make a reasonable assessment of energy deposited in HAM6 when we had the pressure spikes (the spikes themselves are reported in alogs 78346, 78310 and 78323, Sheila's analysis is in alog 78432), or even during regular lock losses.

This is because all of the relevant sensors saturate badly, and the ASC-AS_C is the worst in this respect because of heavy whitening. This happens each and every time the lock is lost. This is our limitation in configuration. I made a temporary change to partly mitigate this in a hope that we might obtain useful knowledge for regular lock losses (but I'm not entirely hopeful), which will be explained later.

Anyway, look at the 1st attachment, which is the trend at around the pressure spike incident at 10W (other spikes were at 60W, so this is the mildest of all). You cannot see the pressure spike because it takes some time for the puffs of gass molecules to reach the pirani.

Important points to take:

• Marker shows when the Fast Shutter was actually fired, which is witnessed by GS13 (bottom left).
• ASC-AS_C cannot be relied on. It started saturating badly even before the FS fired (look at INMONs, 3rd from the top, left), and it kept saturating for a long time.
• OMC QPDs cannot be relied on. They started saturating at about the same time as AS_C though they seem to have stopped saturating earlier  (2nd from the bottom, left).
• AS_B and AS_C cannot be relied on. They're much better than AS_C and OMC QPDs, they started railing some milliseconds earlier than the FS, but they railed (right middle for SUM, one below that for INMONs).
  • But the beam was NOT clipped by the Fast Shutter after it closed as the power dropped to zero, we can see that after WFS stopped saturating.

This is understandable. Look at the second attachment for a very rough power budget and electronics description of all of these sensors. QPDs (AS_C  and OMC QPDs) have 1kOhm raw transimpedance, 0.4:40 whitening that is not switchable on top of two stages of 1:10 that are switchable. WFSs (AS_A and AS_B) have 0.5k transimpedance with a factor of 10 gain that is switchable, and they don't have whitening.

This happens with regular lock losses, and even  with 2W RF lock losses (third attachment), so it's hard to make a good assessment of the power deposited for anything. At the moment, we have to accept that we don't know.

We can use AS_B or AS_A data even though they're railed and make the lower bound of the power, thus energy. That's what I'll do later.

(Added later)

After TJ locked the IFO, we saw strange noise bump ffrom ~20 to ~80 or so Hz. Since nobody had any idea, and since my ASC SUM connection to the PEM rack is an analog connection from the ISC rack that also has the DCPD interface chassis, I ran to the LVEA and disconnected that.

Seems like that wasn't it (it didn't get any better right after the disconnection), but I'm leaving it disconnected for now. I'll connect it back when I can.

J. Kissel, E. von Reis, D. Sigg

Circa March 2023, Daniel had installed some never-used first attempts at further filtering the high frequency noise present in the 524 kHz OMC DCPD channels (never aLOGged because it was never used, but I call them out after I found the work in LHO:68098). 

Now, because I'd like to characterize the existing, in-use, digital AA filtering, running into some unknown noise (LHO:78516), and hoping to install 2 to 4 more parallel filter banks that would also be quite full of filters (LHO:78956), there is worry that there won't be enough computation time in the 524 kHz system.

Remember, that "the 524 kHz system" is actually a modified "standard" 65 kHz system, which is reading out 8 samples from the 524 kHz ADC each 65 kHz clock cycle and computing everything at 65 kHz. Thus, in principle, the max turn around time is
    1 / (2^16 Hz) = (1 / 65536) [sec] = 1.5258789e-5 [sec] = 15.3 [usec]

However, we *think* the practical limit is somewhat less that this. I don't think I understand what those limitations are enough to say definitively and/or to quote a limit quantitatively, but I think they're related to "the copy of the OMC DCPD channels which are demodulated at high frequency to create PI channels that are shipped to the end station -- in other words, the IPC sending demands a bit of computational time, and if there isn't enough turnaround time left in the OP to write to the IPC network, then the end-station SUS PI models throw an IPC timing error."

Anyways -- this morning, I looked at the 524 kHz system's computation time as is before doing anything (via the channel H1:FEC-179_CPU_METER), and it's sitting at 9 [usec] (out of the ideal 15 [usec]), occasionally popping up to 10 [usec].

But -- this led me to remember that -- regardless of whether the filter is turned ON -- the front-end computes the output of the filter -- sucking up computation time.
So, I've removed these unused prototype notch filters from the DCPD A0 and B0 filter banks.
In addition, I've also removed the old "V2A" filter from a previous version of the digital compensation for the OMC DCPD transimpedance amplifier response.

Removing the notch filters drops the computation time from "9 [sec] occasionally bopping up to 10 [usec]."
See attached time series of the CPU meter.

These filters are, of course, available in the filter_archive, under the latest previous archived file before today's work:
    /opt/rtcds/lho/h1/chans/filter_archive/h1iopomc0/
        H1IOPOMC0_1401558760.txt
but for ease of use, I copy them here.

FM3 :: Notches1
    ellip("BandStop",3,0.5,30,12800,13200)
    notch(10216,50,30)
    ellip("BandStop",3,0.5,30,10380,10465)
    ellip("BandStop",3,0.5,30,12900,13100)

FM5 :: Notches2
    ellip("BandStop",3,0.5,30,8100,8200)
    notch(9101,50,30)notch(9337,200,30)
    notch(9463,50,20)
    ellip("BandStop",3,0.5,30,9750,9950)

FM8 :: Notches3
    ellip("BandStop",5,0.5,40,14384,18384)
    ellip("BandStop",5,0.5,40,30768,34768)

FM6 :: V2A
    zpk([5.699+i*22.223;5.699-i*22.223;32.73],[2.549;2.117;6.555],0.00501187,"n")gain(0.971763)
I've also posted a plot of the magnitude of these notch filters -- mostly just to demonstrate how many second order sections that these filters had -- sucking up computation time.

It seems earthquakes causing similar magnitudes of movement on-site may or may not cause lockloss. Why is this happening? Should expect to always or never cause lockloss for similar events. One suspicion is that common or differential motion might lend itself better to keeping or breaking lock.

- Lockloss is defined as H1:DRD-ISC_LOCK_STATE_N going to 0 (or near 0).
- I correlated H1:DRD-ISC_LOCK_STATE_N with H1:ISI-GND_STS_CS_Z_EQ_PEAK_OUTMON peaks between 500 and 2500 μm/s.
- I manually scrolled through the data from present to 2 May 2024 to find events.
    - Manual, because 1) wanted to start with a small sample size and quickly see if there was a pattern, and 2) because I need to find events that caused loss, then go and find similarly sized events we kept lock.
- Channels I looked at include:
    - IMC-REFL_SERVO_SPLITMON
    - GRD-ISC_LOCK_STATE_N
    - ISI-GND_STS_CS_Z_EQ_PEAK_OUTMON ("CS_PEAK")
    - SEI-CARM_GNDBLRMS_30M_100M
    - SEI-DARM_GNDBLRMS_30M_100M
    - SEI-XARM_GNDBLRMS_30M_100M
    - SEI-YARM_GNDBLRMS_30M_100M
    - SEI-CARM_GNDBLRMS_100M_300M
    - SEI-DARM_GNDBLRMS_100M_300M
    - SEI-XARM_GNDBLRMS_100M_300M
    - SEI-YARM_GNDBLRMS_100M_300M
    - ISI-GND_STS_ITMY_X_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_Y_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_Z_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_X_BLRMS_100M_300M
    - ISI-GND_STS_ITMY_Y_BLRMS_100M_300M
    - ISI-GND_STS_ITMY_Z_BLRMS_100M_300M
    - SUS-SRM_M3_COILOUTF_LL_INMON
    - SUS-SRM_M3_COILOUTF_LR_INMON
    - SUS-SRM_M3_COILOUTF_UL_INMON
    - SUS-SRM_M3_COILOUTF_UR_INMON
    - SUS-PRM_M3_COILOUTF_LL_INMON
    - SUS-PRM_M3_COILOUTF_LR_INMON
    - SUS-PRM_M3_COILOUTF_UL_INMON
    - SUS-PRM_M3_COILOUTF_UR_INMON

        - ndscope template saved as neil_eq_temp2.yaml

- 26 events; 14 lockloss, 12 locked (3 or 4 lockloss event may have non-seismic causes)

- After, usiing CS_PEAK to find the events, I, so far, used the ISI channels to analyse the events.
    - The SEI channels were created last week (only 2 events captured in these channels, so far).

- Conclusions:
    - There are 6, CS_PEAK events above 1,000 μm/s in which we *lost* lock;
        - In SEI 30M-100M
            - 4 have z-axis dominant motion with no motion or strong z-motion or no motion in SEI 100M-300M
            - 2 have y-axis dominated motion with a lot of activity in SEI 100M-300M and y-motion dominating some of the time.
    - There are 6, CS_PEAK events above 1,000 μm/s in which we *kept* lock;
        - In SEI 30M-100M
            - 5 have z-axis dominant motion with only general noise in SEI 100M-300M
            - 1 has z-axis dominant noise near the peak in CS_PEAK and strong y-axis domaniated motion starting 4 min prior to the CS_PEAK peak; it too only has general noise in SEI 100M-300M. This x- or y-motion which starts about 4 min before the peak in CS_PEAK has been observed in 5 events -- Love waves precede Rayleigh waves, could be Love waves?
    - All events below 1000 μm/s which lose lock seem to have a dominant y-motion in either/both SEI 30M-100M / 100M-300M. However, the sample size is not large enough to convince me that shear motion is what is causing lockloss. But it is large enough to convince me to find more events and verify. (Some plots attached.)