Reducing the SR2 M1 DAMP P gain from -0.5 to -0.2 and the SRM M1 DAMP L gain from -0.5 to -0.25 did reduce the noise in SRCL and DARM between 3 and 8 Hz.

The main improvement was obtained reducing SR2 damping gain, When reducing the SRM L gain, the 1.3 Hz line appears to get larger. It's worth taking a look at the SRM damping loop and its interaction with the ASC and LSC loops to ttry to track donw the 1.3 Hz peak.

So SRCL and DARM are limited by SR2 P damping noise between 3 and 8 Hz. This change reduced DARM RMS by 20%.

J. Kissel, A. Neunzert, E. Goetz, V. Bossilkov

As we continue the investigation in understanding why the noise the region around 102.13 Hz gets SUPER loud at the beginning of nominal low noise segments, and the calibration line seems to be reporting a huge amount of systematic error (see investigations in LHO:72064), Ansel has found that some new electronics noise has appeared in the end station as of Saturday Aug 5 2023 around 05:30a PT at frequency extremely, and unluckily close to the 102.13000 Hz calibration line -- something at 102.12833 Hz; see LHO:72105.

While we haven't yet ID'd the cause, and thus no have a solution -- we can still change the calibration frequency to move it away from this feature in hopes that there're not beating together terribly like that are now.

I've changed the calibration frequency line frequency to 104.23 Hz as of 21:13 UTC on Aug 09 2023.

This avoids 
    (a) LLO's similar frequency at 101.63 Hz, and 
    (b) because the former frequency, 102.13 Hz was near the upper edge of the 9.33 Hz wide [92.88, 102.21) Hz pulsar spin down, "non-vetoed" band, this new frequency 104.23 Hz skips up to the next 18.55 Hz wide "non-veto" band between [104.22, 122.77) Hz according to LHO:68139.

Stay tuned 
   - to see if this band-aid fix actually helps, or just spreads out the spacing between the comb, and
   - as we continue to investigate the issue of from where this thing came.

Other things of note: 

Since
   - this feature is *not* related to the calibration line itself, 
   - this calibration line is NOT used to generate any time-dependent correction factors and thus the calibration pipeline itself, nor the data it produces is affected
   - this calibration line is used only to *monitor* the calibration systematic error, 
   - this feature is clearly identified in an auxiliary PEM channel -- and that same channel *doesn't* see the calibration line
we conclude that there *isn't* some large systematic error that is occuring, it's just the calculation that's getting spoiled and misreporting large systematic error.
Thus, we make NO plan to do anything on further with the calibration or systematic error estimate side of things from this.

We anticipate that this now falls squarely into the noise subtraction pipeline's shoulders. Given that this 102.12833 Hz noise has a clear witness channel, and the noise creates non-linear nastiness, I expect this will be an excellent candidate for offline non-linear / NONSENS cleaning.


Here's the latest list of calibration lines:
Freq (Hz)   Actuator                   Purpose                      Channel that defines Freq             Changes Since Last Update (LHO:69736)     
15.6        ETMX UIM (L1) SUS          \kappa_UIM excitation        H1:SUS-ETMY_L1_CAL_LINE_FREQ          No change
16.4        ETMX PUM (L2) SUS          \kappa_PUM excitation        H1:SUS-ETMY_L2_CAL_LINE_FREQ          No change
17.1        PCALY                      actuator kappa reference     H1:CAL-PCALY_PCALOSC1_OSC_FREQ        No change
17.6        ETMX TST (L3) SUS          \kappa_TST excitation        H1:SUS-ETMY_L3_CAL_LINE_FREQ          No change
33.43       PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC4_OSC_FREQ        No change
53.67         |                            |                        H1:CAL-PCALX_PCALOSC5_OSC_FREQ        No change
77.73         |                            |                        H1:CAL-PCALX_PCALOSC6_OSC_FREQ        No change
104.23        |                            |                        H1:CAL-PCALX_PCALOSC7_OSC_FREQ        FREQUENCY CHANGE; THIS ALOG
283.91        V                            V                        H1:CAL-PCALX_PCALOSC8_OSC_FREQ        No change
284.01      PCALY                      PCALXY comparison            H1:CAL-PCALY_PCALOSC4_OSC_FREQ        No change
410.3       PCALY                      f_cc and kappa_C             H1:CAL-PCALY_PCALOSC2_OSC_FREQ        No change
1083.7      PCALY                      f_cc and kappa_C monitor     H1:CAL-PCALY_PCALOSC3_OSC_FREQ        No change
n*500+1.3   PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC1_OSC_FREQ        No change (n=[2,3,4,5,6,7,8])

This is a quick investigation about 1.6611Hz comb in h(t) which might be related to OM2 heater (alog 71108 and 71801 ).

During Tuesday maintenance, I measured the OM2 heater driver at the driver chassis (https://dcc.ligo.org/LIGO-D2000211) using a breakout board while both the Beckhoff-side cable and vacuum-side cable were attached to the chassis and the driver was heating the OM2 (H1:AWC-OM2_TSAMS_DRV_VSET=21.448W). How the chassis, Beckhoff module and in-vac OM2 are connected are in https://dcc.ligo.org/LIGO-E2100049.

Voltages I looked at are:

1. Across the positive and negative pin of the Beckhoff drive output that comes into the driver (pin 6-19 on the back DB25).
2. Beckhoff positive drive and the driver ground (pin 6-13 on the back DB25).
3. Beckhoff negative drive and the driver ground (pin 19-13 on the back DB25).
4. Across the driver positive and negative output (pin1-14 on the front DB25).
5. Driver positive output and the driver ground (pin 14-13 on the front DB25).
6. Driver negative output and the driver ground (pin 1-13 on the front DB25).

I used SR785 (single BNC input, floating mode, AC coupled) for all of the above, and in addition used a scope to look at 2. and 3.

1., 4., 5. and 6. didn't look suspicious at all at low frequency, but 2. and 3. showed a forest of peaks comprising ~1.66Hz and its odd-order harmonics. See the first picture attached, 1.66Hz was derived from 18.25/11, not from the fundamental peak as the fft resolusion was not fine enough for that. (FWIW, frequency of the peaks read off of SR785 are 1.625, 5.0, 8.3125, 11.625, 14.9375 and 18.25Hz). Note that the amplitude is not huge (see the y-axis scale).

For 2. and 3., if you look at wider frequency range, it was apparent that the forest extended to much higher frequency (at least ~100Hz before it starts to get smaller), and something was glitching at O(1Hz) (attached video). No, the glitch is not SR785 auto-adjusting the frontend gain, it's in the signal.

Each glitch was pretty fast (2nd picture, see t~ negative 24ms from the center where there was a glitch of ~2ms or so width) and it's not clear if it's related to the comb. I could not see 1.66Hz periodicity using the scope but, as I already wrote, the comb is not huge in amplitude.

Anyway, these all suggest that the comb is between Beckhoff ground and the driver ground (i.e. it's common mode for Beckhoff positive and negative output). Differential receiver inside the driver seems to have rejected that, we're not pounding the heater element of OM2 with this comb (3rd picture, this corresponds to the measurement 4.).

I cannot say at this point if this is the cause of the comb in h(t) but it certainly looks as if Beckhoff is somehow related.

I haven't repeated this with zero drive voltage from Beckhoff though probably I should have. If the comb is not present in 2. and 3. with zero drive voltage from Beckhoff (I doubt that but it's still possible), we could use an adjustable power supply as an alternative.

The heater driver chassis is mounted on the same rack as the DCPD interface. Maybe we could relocate the driver just in case Beckhoff ground is somehow coupling into the rest of the chassis on the same rack.

It would also be useful to know if something happened to the Beckhoff world at the same time the comb appeared in h(t) apart from that we started heating OM2.

J. Kissel

As a result of recent findings that the collection of lines used to probe the low frequency sensing function at 
    PCALY_EXC  8.925, 11.575, 15.275, 24.5 [Hz]
    DARM_EXC   8.825, 11.475, 15.175, 24.4 [Hz]

are causing excess non-linearities found when looking over long time scales --- see LHO:71964 --- I've turned OFF these calibration lines as of 2023-08-09 16:40 UTC. We were already out of observing for PEM injections anyways, and Robert relinquished a moment for me to quickly turn these off.

I've accepted the changes in both the PCALY (h1caley) and OMC (h1omc) front-end model's SDFs against both the safe and OBSERVE.snaps such that this sticks.



Here's the latest list of calibration lines:
Freq (Hz)   Actuator                   Purpose                      Channel that defines Freq             Changes Since Last Update (LHO:69736)     
8.825       DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC1_OSC_FREQ              Turned OFF; THIS aLOG
8.925       PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC5_OSC_FREQ        Turned OFF; THIS aLOG
11.475      DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC2_OSC_FREQ              Turned OFF; THIS aLOG
11.575      PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC6_OSC_FREQ        Turned OFF; THIS aLOG
15.175      DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC3_OSC_FREQ              Turned OFF; THIS aLOG
15.275      PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC7_OSC_FREQ        Turned OFF; THIS aLOG
24.400      DARM (via ETMX L1,L2,L3)   Live DARM OLGTFs             H1:LSC-DARMOSC4_OSC_FREQ              Turned OFF; THIS aLOG
24.500      PCALY                      Live Sensing Function        H1:CAL-PCALY_PCALOSC8_OSC_FREQ        Turned OFF; THIS aLOG
15.6        ETMX UIM (L1) SUS          \kappa_UIM excitation        H1:SUS-ETMY_L1_CAL_LINE_FREQ          No change
16.4        ETMX PUM (L2) SUS          \kappa_PUM excitation        H1:SUS-ETMY_L2_CAL_LINE_FREQ          No change
17.1        PCALY                      actuator kappa reference     H1:CAL-PCALY_PCALOSC1_OSC_FREQ        No change
17.6        ETMX TST (L3) SUS          \kappa_TST excitation        H1:SUS-ETMY_L3_CAL_LINE_FREQ          No change
33.43       PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC4_OSC_FREQ        No change
53.67         |                            |                        H1:CAL-PCALX_PCALOSC5_OSC_FREQ        No change
77.73         |                            |                        H1:CAL-PCALX_PCALOSC6_OSC_FREQ        No change
102.13        |                            |                        H1:CAL-PCALX_PCALOSC7_OSC_FREQ        No change
283.91        V                            V                        H1:CAL-PCALX_PCALOSC8_OSC_FREQ        No change
284.01      PCALY                      PCALXY comparison            H1:CAL-PCALY_PCALOSC4_OSC_FREQ        Off briefly between 2023-08-01 22:02 - 22:11 UTC, back on as of 22:16 UTC
410.3       PCALY                      f_cc and kappa_C             H1:CAL-PCALY_PCALOSC2_OSC_FREQ        No Change
1083.7      PCALY                      f_cc and kappa_C monitor     H1:CAL-PCALY_PCALOSC3_OSC_FREQ        No Change
n*500+1.3   PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC1_OSC_FREQ        No Change (n=[2,3,4,5,6,7,8])

Wed Aug 09 10:21:40 2023 INFO: Fill completed in 21min 35secs

Gerardo confirmed a good fill curbside.

Fire Alarm

Since L1 is not OBSERVING due to logging activity;
H1 has intentionally dropped out of OBSERVING early for PEM Injections and other commissioning.
 

23:42:12UTC Pushed out of Observing by SQZ_MANAGER

We got multiple SDF Diffs appearing (attachment1 and 2), but they kept changing so the ones in the attached screenshot were the earliest ones we could get. Attachment3 shows the SQZ_MANAGER log from that time and Attachment4 shows the SQZ_FC log from that time.

Naoki and Vicky were able to relock the squeezer by manually moving the beam spot, but are running some tests before we go back into Observing.

Benoit, Ansel, Derek

Benoit noticed that for recent locks, the 102.13 Hz calibration line is much louder than typical for the first few hours of the lock. An example of this behavior is shown in the attached spectrogram of H1 strain data on August 5 - this is the first day this behavior appeared. Ansel noted that this feature includes a comb-like structure around the line that is only present in the H1:GDS-CALIB_STRAIN_NOLINES channel and not H1:GDS-CALIB_STRAIN (see spectra for CALIB_STRAIN and CALIB_STRAIN_NOLINES on Aug 5). This issue also visible in the PCAL trends for the 102.13 Hz line. 

We are not sure if the excess noise near 102.13 Hz is from the calibration line itself or another noise source that is near the line. However, the behavior has been present for every lock since 12:30 UTC on August 5 2023. 

WP11359 Consolidated power control IOC software

Erik:

Erik installed new IOC code for the control of Pulizzi and Tripplite network controlled power switches. No DAQ restart was needed.

WP11358 Add new end station Tripplite EPICS channels to the DAQ

Dave:

I modified H1EPICS_CDSACPWR.ini to add the new end station Tripplite control boxes. I found that this file was very much out of date, and only had the MSR Pulizzi channels. I added the missing seven units to the file. DAQ + EDC restart was needed.

WP11357 Add missing ZM4,5 SUSAUX channels to DAQ

Jeff, Rahul, Fil, Marc, Dave:

Jeff discovered that the FM4,5  M1 VOLTMON channels were missing from the SUSAUX model. In this case these are being read by h1susauxh56. It was found that the existing ZM6 channels were actually reading ZM5's channels. h1susauxh56.mdl was modified to:

Change the ADC channels being read by ZM6 from 20-23 to 24-27

Add ZM4 reading ADC channels 16-19

Add ZM5 reading ADC channels 20-23

DAQ Restart was needed.

DAQ Restart

Dave:

Another messy DAQ restart. The sequence was:

1. Restart the 1-leg
2. Restart the EDC to include the power control channels
3. gds1 needed a second restart
4. When fw1 was back up and had written its first full frame:
5. Restart the 0-leg
6. gds0 needed a second restart
7. After fw0 had written its first full frame, fw1 spontaneously crashed!

This was an interesting data point, last week I restarted the DAQ in the opposite order of 0-leg then 1-leg, and as fw1 was coming back fw0 spontaneously crashed, which in that case resulted in some full frame files not being written by either fw.

Could it be that starting the second fw impacts the first fw's disk access speed (perhaps LDAS gap checker switching between file systems)?

As we have found to be always the case, once the errant fw has crashed once, it does not crash again.

A set of extra calibration lines were turned on for a 1-week test starting 2023-07-25 (alog 71706). Part of the purpose of this test was to assess any impacts on CW data quality. Unfortunately, it looks like these lines do pollute low frequencies in a way that would be impactful to CW searches if left on.

Figure 1 shows a normalized H1:CAL-DELTAL_EXTERNAL spectrum for the week starting July 26. The gray dots indicate lines which correspond to intermodulation products of the calibration lines. For this plot, I computed a bunch of intermodulation products with up to 3 lines, and integer multiplicative factors for each line as large as +/- 3 as long as the total order was under 5. The plot intersects this list with the results of the automated linefinder.

These lines are new; their appearance corresponds to the extra lines being turned on. Spot checks of daily Fscan spectra confirm this. For comparison, I have attached an equivalent plot for the previous week (figure 2). Interactive versions of the plots can be found here: fig 1 interactive, fig 2 interactive. The product for each marked line can be seen in the hover text in those versions.

C.Swain, L.Dartez, J.Kissel

Finished characterizing the spare whitening chassis, OMC DCPD S2300004.

• Transfer functions and residuals shown in:
  • 2023-06-29_OMCDCPDWhitening_report.pdf 
• Output Noise Measurements shown in:
  • DCPDA: 2023-07-07_OMC_DCPDA_Whitening_Chassis_S2300004_Output_Referred_Voltage_Noise.pdf 
  • DCPDB: 2023-07-07_OMC_DCPDB_Whitening_Chassis_S2300004_Output_Referred_Voltage_Noise.pdf
• Noise comparison between S2300004 and S2300003, gathered by Jeff Kissel - S2300003 Noise Data - shown in:
  • 2023-07-10_OMC_DCPD_Whitening_Chassis_Noise_Floor_Comparison_Results.pdf

The last two files are text documents containing updated notes on gathering the transfer function measurement data and noise measurement data of the whitening chassis, respectively.