The low freq ~4% error seems to agree with what's modeled in the uncertainty estimates (e.g., see attached) -- that should be a known error.

Naoki, Vicky

Re-aligning FC2 P/Y sliders manually was able to bring squeezer back. Steps taken just now, which seemed to work (tagging OpsInfo). Screenshot of some relevent medms.

• Paused SQZ_FC Guardian manually
• sitemap > SQZ > SQZ Overview > FC SERVO (medms)
  --> manually locked FC VCO GREEN by engaging (closing) the servo (red switch) on the common mode board. Can also increase common gain slider from e.g. 5 -->15/20 if alignment has trouble holding. (GRD will reset everything, so adjust gains however to hold a lock)
  --> could see a weak and flashing TEM00 spot on FC GREEN TRANS camera, so adjusted FC2 P/Y sliders to maximize FC GREEN TRANS spot on the camera. This also maximized FC GREEN transmission flashes, like on H1:SQZ-FC_TRANS_C_LF_OUTPUT.
• Trends screenshot shows trends when squeezer dropped. FC2 was aligned, and you can see flashes on the bright green FC transmission PD (H1:SQZ-FC_TRANS_C_LF_OUTPUT) increase. For alignment, you can use green transmission flashes on the PD, or watch the spot on the camera, which got brighter to what you see in the screenshot (note also the location of the beam spot on the camera; this is where the cavity should be). 
• Un-paused SQZ-FC guardian. Then from SQZ_MANAGER, re-selected FREQ_DEP_SQZ. The squeezer went back fine after this.
• Accepted FC2 SUS SDF diffs after squeezer was back to FDS.

Will investigate why the alignment had to be manually walked (we haven't done this in 1-2 months; ASC typically takes care of FC alignment fine). Likely similar issue that Corey had last night LHO:72050; if this happens again, we can try aligning FC2 in this way.

00:05:12 Returned to Observing

We found that SQZ ASC has been OFF since 6 days ago (first attached figure). We took SQZ data 6 days ago in alog71902. During this measurement, we turned off SQZ ASC for alignment of anti squeezing. We changed the SQZ ASC flag in sqzparams to False and set back to True after the measurement, but we may forgot to reload the SQZ_MANAGER guardian. We asked Oli to reload the SQZ_MANAGER guardian when lockloss happens. We went to commissioning and manually turned on the SQZ ASC switch and went back to observing in a minute.

We also found that the SQZ ASC switch (H1:SQZ-ASC_WFS_SWITCH) and FC ASC switch (H1:SQZ-FC_ASC_SWITCH) are not monitored. We monitored them as shown in the second, third attached figure.

Regarding 2), we maximized the +1st order beam, which is left side of 0th order beam looking from the laser side of SQZT0.

FYI, 
$ gpstime Aug 05 2023 12:30 UTC
    PDT: 2023-08-05 05:30:00.000000 PDT
    UTC: 2023-08-05 12:30:00.000000 UTC
    GPS: 1375273818.000000
so... this behavior seems to have started at 5:30a local time on a Saturday. Therefore *very* unlikely that the start of this issue is intentional / human change driven.

The investigation continues....
making sure to tag CAL.

Other facts and recent events:
- Attached are 2 screenshots that show the actual *digital* excitation is not changing with time in anyway.
    :: 2023-08-08_H1PCALEX_OSC7_102p13Hz_Line_3mo_trend.png shows the specific oscillator, --- PCALX's OSC7 which drives the 102.13 Hz line's EPICs channel version of its output. The minute trend shows the max, min, and mean of the output, and there's no change in amplitude.
    :: 2023-08-08_H1PCALEX_EXC_SUM_3mo_trend.png shows a trend of the total excitation sum from PCAL X. This also shows *no* change in time in amplitude.

Both trends show the Aug 02 2023 change in amplitude kerfuffle I caused that Corey found and a bit later rectified -- see LHO:71894 and subsequent comments, but that was done, over with an solved, definitely by Aug 03 2023 UTC and unrelated to the start up of this problem.

It's also well after I installed new oscillators and rebooted the PCALX, PCALY, and OMC models on Aug 01 2023 (see LHO:71881).

The front-end version of the calibration's systematic error at 102.13 Hz also shows the long, time-dependent issue -- this will allow us to trend the issue against other channels

Folks in the calibration group have found that the online monitoring system for the
    -  overall DARM response function systematic error
    - (absolute reference) / (Calibrated Data Product) [m/m] 
    - ( \eta_R ) ^ (-1) 
    - (C / 1+G)_pcal / (C / 1+G)_strain
    - CAL-DELTAL_REF_PCAL_DQ / GDS-CALIB_STRAIN
(all different ways of saying the same thing; see T1900169) in calibration at each PCAL calibration line frequency -- the "grafana" pages -- are showing *huge* amounts of systematic error during these times when the amplitude of the line is super loud.

Though this metric is super useful because it's dreadfully obvious that things are going wrong -- this metric is not in any normal frame structure, so you can't compare it against other channels to find out what's causing the systematic error.

However -- remember -- we commissioned a front-end version of this monitoring during ER15 -- see LHO:69285.

That means the channels 
     H1:CAL-CS_TDEP_PCAL_LINE8_COMPARISON_OSC_FREQ        << the frequency of the monitor
    H1:CAL-CS_TDEP_PCAL_LINE8_SYSERROR_MAG_MPM        << the magnitude of the systematic error
    H1:CAL-CS_TDEP_PCAL_LINE8_SYSERROR_PHA_DEG        << the phase of the systematic error 

tell you (what's supposed to be***) equivalent information.

*** One might say that "what's suppose to be" is the same as "roughly equivalent" due to the following reasons: 
    (1) because we're human, the one system is displaying the systematic error \eta_R, and the other is displaying the inverse ( \eta_R ) ^ (-1) 
    (2) Because this is early-days in the front-end system, it uses the "less complete" calibrated channel CAL-DELTAL_EXTERNAL_DQ rather than the "fully correct" channel GDS-CALIB_STRAIN

But because the problem is so dreadfully obvious in these metrics, even though they're only *roughly* equivalent, you can see the same thing.
In the attached screenshot, I show both metrics for the most recent observation stretch, between 10:15 and 14:00 UTC on 2023-Aug-09.

Let's use this front-end metric to narrow down the problem via trending.

There appears to be no change in the PCALX analog excitation monitors either.

Attached is a trend of some key channels in the optical follower servo -- the analog feedback system that serves as intensity stabilization and excitation power linearization for the PCAL's laser light that gets transmitted to the test mass -- the actuator of which is an acousto-optic modulator (an AOM). There seems to be no major differences in the max, min, and mean of these signals before vs. after these problems started on Aug 05 2023.

H1:CAL-PCALX_OFS_PD_OUT_DQ
H1:CAL-PCALX_OFS_AOM_DRIVE_MON_OUT_DQ

I believe this is caused by the presence of another line very close to the 102.13 Hz pcal line.  This second line is present at the start of a lock stretch but seems to go away as the lock stretch continues.  I have attached a plot showing a zoom-in on an ASD around 102.1-102.2 Hz right after a lock stretch (orange), where the second peak is evident, and well into a lock stretch (blue) where the PCAL line is still present, but the second peak right below it in frequency is gone.  This ASD is computed using an hour of data for each curve, so we can get the needed resolution for these two peaks.

I don't know the origin of this second line.  However, a quick fix to the issue could be moving the PCAL line over by about a Hz.  The second attached plot shows that the spectrum looks pretty clean from 101-102 Hz, so somewhere in there would be probably be okay for a new location of the PCAL line.

Since it looks like the additional noise is at 102.12833 Hz, I did a quick check in Fscan data from Aug 5 for channels where there is high coherence with DELTAL_EXTERNAL at 102.12833 but *not* at 102.13000 Hz. This narrows down to just a few channels:

• H1:PEM-EX_MAG_EBAY_SEIRACK_{Z,Y}_DQ .
• H1:PEM-EX_ADC_0_09_OUT_DQ
• H1:ASC-OMC_A_YAW_OUT_DQ. Note that other ASC-OMC channels (Fscan tracks A,B and PIT,YAW) see high coherence at both frequencies.

(lines git issue opened as we work on this.)

As a result of Ansel's discovery, and conversation on the CAL call today -- I've moved the calibration line frequency from 102.13 to 104.23 Hz. See LHO:72108.

This line may have appeared in the previous lock the day before (Aug 4). The daily spectrogram for Aug 4 shows a line near 100 Hz starting at 21:00 UTC. 

Looking at alogs leading up to the time Derek notes above, I noticed that Gabriele retuned and tested new LSC FF. This change may be related to this new peak. Remembering some issues we had recently where DHARD filter impulses were ringing up violin modes, I checked the new LSC FF filters and how they are engaged in the guardian. Some of them have no ramp time, and the filter bank is turned on immediately along with the filters in the guardian. I have no idea why that would cause a peak at 102 Hz, but I updated those filters to have a 3 second ramp.

Reloaded the H1LSC model to load in Elenna's filter changes

Now that the calibration line has been moved, the comb-like structure at the calibration line frequency is no longer present (checked in the CLEAN channel).

We can also see the shape of the 102.12833 Hz line much more clearly without the overlapping calibration line. I have attached a plot for reference on the width and shape.

As discussed in todays commissioning meeting, I checked TMSX and ETMX movement for a kick during locking and couldn't see anything suspicious. I did find some increase motion/noise every 8Hz in TMSX 1s into ENGAGE_SOFT_LOOPS when ISC_LOCK isn't explicitly doing anything, plot attached. However this noise was present prior to Aug 4th, (July 30th attached).

TMS is suspicious as Betsy found that TMS's have violin modes ~103-104Hz.

Jeff draws attendtion to 38295, showing modes of quad blade springs above 110Hz, and 24917 showing quad top wire modes above 300Hz.

Elenna's notes with calibration lines off (as we are experimenting with for current lock) we can see this 102Hz peak at ISC_LOCK state ENGAGE_ASC_FOR_FULL_IFO. We were mistaken.

To preserve documentation, this problem has now been solved, with more details in 72537, 72319, and 72262.

The cause of this peak was a spurious, narrow, 102 Hz feature in the SRCL feedforward that we didn't catch when the filter was made. This has been been fixed, and the cause of the mistake has been documented in the first alog listed above so we hopefully don't repeat this error.

DAQ missing full frames GPS times (no overlap between fw0 and fw1 lists) (missing because of crash highlighted)

Scanning directory: 13755...
FW0 Missing Frames [1375554944, 1375555008]
FW1 Missing Frames [1375554752, 1375554816, 1375555072, 1375555136, 1375555200]
 

This morning new filtermodules were added to h1susauxh56 to readout the ZM4,5 quadrants. The RCG starts new filtermodules in an inactive state, namely with INPUT=OFF, OUTPUT=OFF, GAIN=0.0. It can be a bit time consuming to manually activate the MEDM switches by hand.

I wrote a script to activate new filtermodules, called activate_new_filtermodules. It takes the filtermodule name as its argument.

Here is an example using a h1pemmx filtermodule:

david.barker@opslogin0: activate_new_filtermodule PEM-MX_CHAN_12
H1:PEM-MX_CHAN_12_SW1 => 4
H1:PEM-MX_CHAN_12 => ON: INPUT
H1:PEM-MX_CHAN_12_SW2 => 1024
H1:PEM-MX_CHAN_12 => ON: OUTPUT
H1:PEM-MX_CHAN_12_GAIN => 1.0
 

------------------- DAQ CHANGES: ------------------------

REMOVED:

No channels removed from the DAQ frame

ADDED:

+8 fast channels added (all at 256Hz)

< H1:SUS-ZM4_M1_VOLTMON_LL_OUT_DQ 4 256
< H1:SUS-ZM4_M1_VOLTMON_LR_OUT_DQ 4 256
< H1:SUS-ZM4_M1_VOLTMON_UL_OUT_DQ 4 256
< H1:SUS-ZM4_M1_VOLTMON_UR_OUT_DQ 4 256
< H1:SUS-ZM5_M1_VOLTMON_LL_OUT_DQ 4 256
< H1:SUS-ZM5_M1_VOLTMON_LR_OUT_DQ 4 256
< H1:SUS-ZM5_M1_VOLTMON_UL_OUT_DQ 4 256
< H1:SUS-ZM5_M1_VOLTMON_UR_OUT_DQ 4 256
 

+112 slow channels added

Tue08Aug2023
LOC TIME HOSTNAME     MODEL/REBOOT
11:31:41 h1susauxh56  h1susauxh56 <<< Correct ZM6, add ZM4 and ZM5

11:32:59 h1daqdc1     [DAQ] <<< 1-leg restart
11:33:10 h1daqfw1     [DAQ]
11:33:11 h1daqnds1    [DAQ]
11:33:11 h1daqtw1     [DAQ]
11:33:19 h1daqgds1    [DAQ]

11:33:53 h1susauxb123 h1edc[DAQ] <<< EDC restart for CDSACPWR

11:34:25 h1daqgds1    [DAQ] <<< 2nd gds1 restart needed

11:36:06 h1daqdc0     [DAQ] <<< 0-leg restart
11:36:17 h1daqfw0     [DAQ]
11:36:17 h1daqtw0     [DAQ]
11:36:18 h1daqnds0    [DAQ]
11:36:25 h1daqgds0    [DAQ]
11:37:14 h1daqgds0    [DAQ] <<< 2nd gds0 restart needed

11:39:10 h1daqfw1     [DAQ] <<< FW1 crash!!
 

The new VOLTMON channels for ZM4 and ZM5 have been initialized, accepted, and monitored in the susauxh56 SDF table, screenshot attached.

The above aLOG covers another *solution* top the on-going studies about the ~5-10% systematic error in the calibration -- namely, what's unique to LHO and *left over* after the flaw in GDS filters that was fixed in LHO:71787.

The filter was loaded by 2023-08-07 17:15 UTC.

This change has been added to the LHO record of calibration pipeline changes for O4, DCC:T2300297

Correction to the timing of this filter update -- The filter was loaded by 2023-08-07 17:15 PDT -- i.e. 2023-08-08 00:15 UTC