16:44 UTC lockloss. PRCL was oscillating at about ~3.6 Hz.

Ryan C, Sheila D

15:32 to 16:01 UTC we dropped Observing to do some range checks/investigations.

Sheila and I did some range checks following the wiki. Running the coherence check showed high coherence with CHARD and the SQZer BLRMs did not look as good as previous locks, particularly in the 10-20, 20-34, and 60-100 HZ bandwidths. So we decided to drop out of observing since LLO was down to run the SQZ alignment and angle scans and I concurrently ran the A2L_min script (TJs alog78552). After these finished we also took 5 minutes of NO_SQZ time starting at 15:49 UTC which showed that the extra noise is not coming from the SQZer. We gained a few Mpcs from the SQZ scan and A2L_min script.

Coherence comparison before and after SQZ and A2L checks

Ran the (userapps)/isc/h1/scripts/a2l/a2l_min_multi.py script for all four quads in both P and Y to try and help our range. Minimal improvements, this was done in tandem with the SQZ scans which gained us about 2 Mpc.

ETMX P
Initial:  3.12
Final:    3.09
Diff:     -0.03

          ETMX Y
Initial:  4.79
Final:    4.81
Diff:     0.02

          ETMY P
Initial:  4.48
Final:    4.49
Diff:     0.01

          ETMY Y
Initial:  1.13
Final:    1.26
Diff:     0.13

          ITMX P
Initial:  -1.07
Final:    -1.02
Diff:     0.05

          ITMX Y
Initial:  2.72
Final:    2.79
Diff:     0.07

          ITMY P
Initial:  -0.47
Final:    -0.43
Diff:     0.04

          ITMY Y
Initial:  -2.3
Final:    -2.36
Diff:     -0.06

Fri Jul 12 08:08:03 2024 INFO: Fill completed in 8min 0secs

Jordan confirmed a good fill curbside.

TITLE: 07/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

• We've been locked and observing for 4:20, since 10:17 UTC
  • The lockloss overnight (08:24 UTC) was from an earthquake
• Lots of earthquakes overnight
• Range is ok, SQZing doesn't look the best from the trace on NUC33

TITLE: 07/12 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Mostly uneventful evening, but after the lockloss while trying to relock (hadn't done an IA), DRMI unlocked twice while we were in ENGAGE_DRMI_ASC, and got beamsplitter saturations both times, with the second time also getting a BS ISI saturation (1st time ndscope, 2nd time ndscope). Not sure what that was about, as I ran an IA and when I tried locking again it was fine. After being in Observing for a little bit I decided that I wanted to adjust sqz because it was really bad, and I had some issues after running it the first time (clicked things out of order even though it shouldn't matter??), but eventually I was able to get a lot more squeezing and a much better sensitivity at the higher frequencies.
LOG:

23:00 Relocking and at PARK_ALS_VCO
23:35 NOMINAL_LOW_NOISE
23:42 Started running simulines measurement to check if simulines is working
00:05 Simulines done
00:13 Observing

00:44 Earthquake mode activated due to EQ in El Salvador
01:04 Seismic to CALM

04:32 Lockloss
Relocking
    - 17 seconds into ENGAGE_DRMI_ASC, BS saturated and then LL
    - BS saturation twice in ENGAGE_DRMI_ASC, then ISI BS saturation, then LL
05:08 Started an initial alignment
05:27 IA done, relocking
06:21 NOMINAL_LOW_NOISE
06:23 Observing

06:38 Out of Observing to try and make sqz better because it's really bad
07:24 Observing                                                                                                            

Lockloss @ 07/12 04:32 UTC

Thu Jul 11 08:11:57 2024 INFO: Fill completed in 11min 53secs

late entry from this morning

Observing at 157Mpc and have been locked for 3.5  hours. We rode out a 5.2 earthquake from El Salvador earlier. Wind is low and going down.

Ibrahim, Oli

Attached are the most recent (07-10-2024) BBSS Transfer Functions following the most recent RAL visit and rebuild. The Diaggui screenshots show the first 01-05-2024 round of measurements as a reference. The PDF shows these results with respect to expectations from the dynamical model. Here is what we think so far:

• Results are less coherent in certain orders of magnitude, namely L, P, Y.
• The percieveable roll we have not fixed in M3 does not seem to change the Roll TF measurement
• The F3 OSEM sways more than the others as a response to the TF excitations and is in general, more reactive than the others (attached gif)
• Pitch has unexpected and shifted peaks, which seem to cross couple into the other TFs
• Most percieveeable in Y, there is increased signal noise from around 1.5-6hz
• The F1, LF and RT OSEMs have drifted from their nominal centered positions (this will be a seperate alog).

Thoughts:

The nicest sounding conclusion here is that something is wrong with the F3 OSEM because it is the only OSEM and/or flag involved in L, P, Y (less coherent measurements) but not in the others; F3 fluctuates and reacts much more irratically than the others, and in Y, the F3 OSEM has the greatest proportion of actuation than P and a higher magnitude than L, so if there were something wrong with F3, we'd see it in Y the loudest. This is exactly where we see the loudest ring-up. I will take spectra and upload this in another alog. This would account for all issues but the F1, LF and RT OSEM drift, which I will plot and share in a seperate seperate alog.

We're running a patched version of simuLines during the next calibration measurement run. The patch (attached) was provided by Erik to try and get around what we think are awg issues introduce (or exacerbated) by the recent awg server updates (mentioned in LHO:78757).

Operators: there's is nothing special to do. just follow the normal routine as I applied the patch changes in place. Depending on the results of this test, I will either roll them back or work with Vlad to make them permanent (at least for LHO).

J. Kissel, E. von Reis, D. Sigg

Circa March 2023, Daniel had installed some never-used first attempts at further filtering the high frequency noise present in the 524 kHz OMC DCPD channels (never aLOGged because it was never used, but I call them out after I found the work in LHO:68098). 

Now, because I'd like to characterize the existing, in-use, digital AA filtering, running into some unknown noise (LHO:78516), and hoping to install 2 to 4 more parallel filter banks that would also be quite full of filters (LHO:78956), there is worry that there won't be enough computation time in the 524 kHz system.

Remember, that "the 524 kHz system" is actually a modified "standard" 65 kHz system, which is reading out 8 samples from the 524 kHz ADC each 65 kHz clock cycle and computing everything at 65 kHz. Thus, in principle, the max turn around time is
    1 / (2^16 Hz) = (1 / 65536) [sec] = 1.5258789e-5 [sec] = 15.3 [usec]

However, we *think* the practical limit is somewhat less that this. I don't think I understand what those limitations are enough to say definitively and/or to quote a limit quantitatively, but I think they're related to "the copy of the OMC DCPD channels which are demodulated at high frequency to create PI channels that are shipped to the end station -- in other words, the IPC sending demands a bit of computational time, and if there isn't enough turnaround time left in the OP to write to the IPC network, then the end-station SUS PI models throw an IPC timing error."

Anyways -- this morning, I looked at the 524 kHz system's computation time as is before doing anything (via the channel H1:FEC-179_CPU_METER), and it's sitting at 9 [usec] (out of the ideal 15 [usec]), occasionally popping up to 10 [usec].

But -- this led me to remember that -- regardless of whether the filter is turned ON -- the front-end computes the output of the filter -- sucking up computation time.
So, I've removed these unused prototype notch filters from the DCPD A0 and B0 filter banks.
In addition, I've also removed the old "V2A" filter from a previous version of the digital compensation for the OMC DCPD transimpedance amplifier response.

Removing the notch filters drops the computation time from "9 [sec] occasionally bopping up to 10 [usec]."
See attached time series of the CPU meter.

These filters are, of course, available in the filter_archive, under the latest previous archived file before today's work:
    /opt/rtcds/lho/h1/chans/filter_archive/h1iopomc0/
        H1IOPOMC0_1401558760.txt
but for ease of use, I copy them here.

FM3 :: Notches1
    ellip("BandStop",3,0.5,30,12800,13200)
    notch(10216,50,30)
    ellip("BandStop",3,0.5,30,10380,10465)
    ellip("BandStop",3,0.5,30,12900,13100)

FM5 :: Notches2
    ellip("BandStop",3,0.5,30,8100,8200)
    notch(9101,50,30)notch(9337,200,30)
    notch(9463,50,20)
    ellip("BandStop",3,0.5,30,9750,9950)

FM8 :: Notches3
    ellip("BandStop",5,0.5,40,14384,18384)
    ellip("BandStop",5,0.5,40,30768,34768)

FM6 :: V2A
    zpk([5.699+i*22.223;5.699-i*22.223;32.73],[2.549;2.117;6.555],0.00501187,"n")gain(0.971763)
I've also posted a plot of the magnitude of these notch filters -- mostly just to demonstrate how many second order sections that these filters had -- sucking up computation time.

It seems earthquakes causing similar magnitudes of movement on-site may or may not cause lockloss. Why is this happening? Should expect to always or never cause lockloss for similar events. One suspicion is that common or differential motion might lend itself better to keeping or breaking lock.

- Lockloss is defined as H1:DRD-ISC_LOCK_STATE_N going to 0 (or near 0).
- I correlated H1:DRD-ISC_LOCK_STATE_N with H1:ISI-GND_STS_CS_Z_EQ_PEAK_OUTMON peaks between 500 and 2500 μm/s.
- I manually scrolled through the data from present to 2 May 2024 to find events.
    - Manual, because 1) wanted to start with a small sample size and quickly see if there was a pattern, and 2) because I need to find events that caused loss, then go and find similarly sized events we kept lock.
- Channels I looked at include:
    - IMC-REFL_SERVO_SPLITMON
    - GRD-ISC_LOCK_STATE_N
    - ISI-GND_STS_CS_Z_EQ_PEAK_OUTMON ("CS_PEAK")
    - SEI-CARM_GNDBLRMS_30M_100M
    - SEI-DARM_GNDBLRMS_30M_100M
    - SEI-XARM_GNDBLRMS_30M_100M
    - SEI-YARM_GNDBLRMS_30M_100M
    - SEI-CARM_GNDBLRMS_100M_300M
    - SEI-DARM_GNDBLRMS_100M_300M
    - SEI-XARM_GNDBLRMS_100M_300M
    - SEI-YARM_GNDBLRMS_100M_300M
    - ISI-GND_STS_ITMY_X_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_Y_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_Z_BLRMS_30M_100M
    - ISI-GND_STS_ITMY_X_BLRMS_100M_300M
    - ISI-GND_STS_ITMY_Y_BLRMS_100M_300M
    - ISI-GND_STS_ITMY_Z_BLRMS_100M_300M
    - SUS-SRM_M3_COILOUTF_LL_INMON
    - SUS-SRM_M3_COILOUTF_LR_INMON
    - SUS-SRM_M3_COILOUTF_UL_INMON
    - SUS-SRM_M3_COILOUTF_UR_INMON
    - SUS-PRM_M3_COILOUTF_LL_INMON
    - SUS-PRM_M3_COILOUTF_LR_INMON
    - SUS-PRM_M3_COILOUTF_UL_INMON
    - SUS-PRM_M3_COILOUTF_UR_INMON

        - ndscope template saved as neil_eq_temp2.yaml

- 26 events; 14 lockloss, 12 locked (3 or 4 lockloss event may have non-seismic causes)

- After, usiing CS_PEAK to find the events, I, so far, used the ISI channels to analyse the events.
    - The SEI channels were created last week (only 2 events captured in these channels, so far).

- Conclusions:
    - There are 6, CS_PEAK events above 1,000 μm/s in which we *lost* lock;
        - In SEI 30M-100M
            - 4 have z-axis dominant motion with no motion or strong z-motion or no motion in SEI 100M-300M
            - 2 have y-axis dominated motion with a lot of activity in SEI 100M-300M and y-motion dominating some of the time.
    - There are 6, CS_PEAK events above 1,000 μm/s in which we *kept* lock;
        - In SEI 30M-100M
            - 5 have z-axis dominant motion with only general noise in SEI 100M-300M
            - 1 has z-axis dominant noise near the peak in CS_PEAK and strong y-axis domaniated motion starting 4 min prior to the CS_PEAK peak; it too only has general noise in SEI 100M-300M. This x- or y-motion which starts about 4 min before the peak in CS_PEAK has been observed in 5 events -- Love waves precede Rayleigh waves, could be Love waves?
    - All events below 1000 μm/s which lose lock seem to have a dominant y-motion in either/both SEI 30M-100M / 100M-300M. However, the sample size is not large enough to convince me that shear motion is what is causing lockloss. But it is large enough to convince me to find more events and verify. (Some plots attached.)

[Dave, Erik]

Dave found that DACs in h1sush2a were in a FIFO HIQTR state since 2024-04-09 11:33 UTC.

FIFO HIQTR means that DAC buffers had more data than expected.  DAC latency would be proportionally higher than expected.

The models were restarted, which fixed the issue.