The safe and observe files for SR2 and SR3 are now unlinked. The higher gain values are saved in the safe.snaps and the lower gains have been accepted in the observe.snaps.

Relocking is struggling, theres some ringup during DRMI_LOCKED_CHECK_ASC that keeps killing it.

These are recent measurements by Gabriele of DHARD Y and CHARD Y, plotted with the error shading as well.

CHARD Y has a UGF of 3.3 Hz and 44 deg of phase margin. DHARD Y has a UGF of 5 Hz and a phase margin of 22 deg.

DHARD Y has additional UGFs at 2.1 and 1.5 Hz. CHARD Y has some interesting peak features that cross zero a few times between 2.3 and 3.3 Hz. There also appear to be additional UGFs around 1.2, 0.9, 0.5, 0.4 and 0.3 Hz.

Taking another look at these measurements, DHARD P has about 6 dB gain margin at the highest UGF. DHARD Y's gain margin about 3 or 4 dB. Tagging CAL because this comment might be useful for DARM loop investigations.

Elenna, Sheila

We ran the noise budget code for this no squeezing time. 

• We updated the optical gain (Calibratations.py, in the closedLoopSenseing function) using the calibration report here which Jeff pointed us to.  (The most recent one with the exported tag is the current calibration.)  This is the same as the inverse of the gain in H1:CAL-CS_DARM_CFTD_ERR filter called O4 gain if you have the correct calibration report.  Elenna also found the ratio of DCPD mA to DARM err counts by running the template that measures their transfer function aligoNB/H1/couplings/check_DARM_gain.xml for the braodband pcal injection from alog 72047.  These two steps are what we needed to do to get the calibration right so that the semiclassical shot noise caluculation seems correct.  
• We are using CAL_DELTAL, since this noise budget was run before the calibration fix of last week that channel is more accurate.  We are also using the TDCFs kappa_c and the cavity pole to calculate the shot noise. The radiation pressure noise is calculated with the arm transmitted power monitors. 
• There is no DAC noise included in this budget, as we aren't confident in our usual DAC noise projection right now.  68869  With recent improvements in the low frequency noise it would be good for us to revisit the PUM DAC noise projections soon, by running in the higher noise state and running the noise monitor estimate for that higher noise state. 
• We have changed the frequency noise in this budget. We measure frequency noise by switching CARM onto one sensor, (REFL B), then using that sensor as a out of loop frequency noise sensor.  There was a mistake in the code where the psd was instead multiplied by sqrt(2).  We have now divided the PSD by 2, which should give us the correct frequency noise projection where we are frequency noise limited.  Interestingly, this correctly predicts the darm noise at 4-5kHz where it should be an underestimate by sqrt(2) in the ASD.
• Vicky is working on modeling the quantum noise with Kevin, we are just doing the no squeeze budget here and will return to the noise budget with squeezing with Vicky.  
• Also attached are the sub budgets for laser, asc and length noise, with the DARM noise plotted from a time with squeezing.
• Laser noise: The 120Hz peak is well predicted by the IMC PZT yaw injection.  The frequency noise has the factor described above applied, and there may be some double counting at low frequency although we aren't sure.  
• ASC: CSOFT P, and the hard loops are the dominant contribution.  CHARD Y in particular is large, this is because of Gabriele's recent loop change which helped reduce the RMS and improve some nonlinear noise in DARM. 71927 Elenna is looking at if we can roll that off to keep the low frequency gain without this additional noise at 40Hz. 
• LSC: MICH is the dominant contribution to LSC, it is similar to the level of the ASC total. 

This is all commited as 0f9ffe0e

Sheila, Vicky - we have re-run the noise budget for following times:

Noise budget with squeezing. Changes here: using GDS instead of CAL-DELTAL, closer thermalized FDS time to no-sqz, using updated IFO gwinc parameters related to quantum noise calculation.
(Edit: was a glitch in the old time; updated to an FDS time without glitches. All plots updated.)

PDT: 2023-08-10 08:45:00.000000 PDT
UTC: 2023-08-10 15:45:00.000000 UTC
GPS: 1375717518.000000

PDT: 2023-08-10 09:35:52.000000 PDT
UTC: 2023-08-10 16:35:52.000000 UTC
GPS: 1375720570.000000

Noise budget with no squeezing. Same time as above, now calculates using gwinc quantum noise calculation instead of semiclassical calculation used previously.

PDT: 2023-08-10 10:18:11.000000 PDT
UTC: 2023-08-10 17:18:11.000000 UTC
GPS: 1375723109.000000

Both sqz & no-sqz noise budgets now use the correlated quantum noise calculation from gwinc, instead of semiclassical calculations for SN & QRPN. The gwinc budget parameters related to quantum noise calculation are consistent with the recent sqz data set (8/2, alog 72565), with readout losses evenly split between IFO output losses that influence optical gain (20%) and SQZ injection losses (20%), parameters in plot title here. This is high on SQZ injection losses, and slightly conservative on IFO output losses. This updated FDS time is thermalized and closer to the No-SQZ time; the time used previously was several hours earlier near the start of lock, w/ ifo not yet thermalized.

Unlike before, both budgets now show GDS-CALIB STRAIN, which on 8/10 was more accurately calibrated (see Louis's alog on Aug 8, LHO:72075, comparing CAL-DELTAL and GDS vs. PCAL sweep, and his record from 72531). CAL-DELTAL was previously overestimating range due to calibration inaccuracies. We got GDS-CALIB_STRAIN data from nds servers, and at first weren't able to get input jitter data from nds, due to the sampling rate change of IMC-WFS channels from 2k to 16k, 71242. Jonathan H. helped us fix this issue, so we can now pull GDS data and input jitter data from nds.ligo-wa.caltech.edu:31200 -- thank you Jonathan!! With this, the input jitter sub-budget is kind of interesting, looks to be mostly IMC-WFS in YAW.

A quick thought on discrepancy between expected and measured DARM between below several hundred Hz-- I don't know if this could be related to the recent update to gwinc CTN parameters (high/low index loss angles), related to quantum noise, or mystery noise. The recent gwinc CTN update seemed to have dropped the calculated CTN level slightly (maybe 10-15% or so). In April 2023, Kevin helped update CTN parameters LHO:68499 to reconcile H1 budget with the official gwinc parameters, while Evan made a correlated noise measurement 68482 where noise in the bucket seems more consistent with the older CTN estimate from gwinc (or very slightly higher). Another idea is that it could be related to quantum noise, such as SRCL detuning or sqz angle which could've changed since the sqz dataset, as quantum noise can also affect the noise in this region.

All pushed as git commit 28cf2664.

Edit: All pushed again as git commit 33ffd60b.

Added noise budgets with squeezing for HVAC off time on August 17 from alogs 72308, 72297.

When comparing this HVAC off time on Aug 17 with the noise budget from above on Aug 10, it's interesting to note the broadband difference in input jitter (Aug 10 vs Aug 17, HVAC off). Between these times, worth noting that I think there were several additional improvements (like LSC FF or SUS-related) as well.

Edit: updated 8/10 input jitter budget to the less glitchy noise budget time.

Much of the gap between expected DARM (black traces) and measured DARM (red traces) in the noise budget looks compatible with elevating the CTN trace. Budget plots with 100 Hz CTN @ 1.45e-20 m/rtHz are attached below for the no-HVAC times. This is almost 30% higher than the new gwinc nominal CTN at 100 Hz (i.e., 1.128e-20 m/rtHz --> 1.45e-20 m/rtHz). Compared to the old gwinc estimate of 1.3e-20, this is ~11% higher. Quantum noise calculation unchanged here.

This CTN level is similar to the 30% of excess correlated noise that Evan H. observed in April 2023, see LHO:68482. His cross-correlation measurement sees ~30% excess correlated noise around 100 Hz after subtracting input jitter noise, where that "30%" is using the newer gwinc CTN estimate of 1.128e-20 m/rtHz @ 100 Hz. This elevated correlated noise, if attributed to CTN, corresponds to CTN @ 100 Hz of about 1.3*1.128 = 1.46e-20 m/rtHz. See this git merge request for the gwinc CTN update ; this update lowered the expected CTN at 100 Hz by ~15%, from 1.3e-20 (old) to 1.1e-20 m/rtHz (new), based on updated MIT measurements.

For reference, I have plotted these various CTN levels as dotted traces in the thermal sub-budget.

To elevate CTN levels by 30% in the budget code, I scaled both high+low index loss angles by a factor of 1.8, specifically Philhighn 3.89e-4 --> 7e-4 ; Phillown 2.3e-5 --> 4.14e-5. It seems like much higher than this level ~1.45e-20 might be difficult to reconcile with the full budget.

Noteworthy w.r.t. squeezing: from the laser noise sub-budget, laser frequency noise looks within 33% of squeezed shot noise with ~3.7dB of squeezing. By contrast, the L1 noise budget from Aug 2023 (LLO:66532) shows laser noise at the ~20% level of squeezed shot noise with 5.3 dB of squeezing -- i.e. a lower laser noise floor past shot noise.

The following plots can be found in /ligo/gitcommon/NoiseBudget/aligoNB/out/H1/lho_all_noisebudgets_081723_noHVAC_elevatedCTN, and not yet commited to the git repo.

Plots with higher CTN are attached here for the SQZ / no-SQZ proper noise budget times from 8/10, when injections were run.

Comparing the sqz vs. no-sqz budgets suggests there might be more to understand here, to tease apart the contributions from coating thermal noise (CTN) vs. quantum noise in the bucket. In particular, something disturbing that stands out, is that I imagined that if elevated CTN is the physical effect we're missing, it would reconcile both NBs with and without squeezing. However, there is still some discrepancy in the un-squeezed budget, which was not resolved by CTN, and seems to have a consistent shape. I'm wondering if this is related to the IFO configuration as it affects the quantum noise without squeezing. I think this could result from a non-zero but small SRCL detuning since it looks like elevated noise, with a clear shape, that increases below the DARM pole. Simply elevating CTN to match the no-sqz budget would put us in conflict with squeezed darm, so I don't think it makes sense to elevate CTN further. The budget currently has 0 SRCL detuning as it "seems small-ish", but this parameter is somewhat unconstrained in the quantum noise models.

In models, the readout angle is upper-bounded by Sheila's contrast defect measurement, though in principle it could probably be anything lower than that too, which could be worth exploring. It might be helpful to have an external measurement of the thermalized physical SRCL detuning, or in the models allowing the SRCL detunings to vary, to explore how it fits or is constrained by the fuller noise budget picture.

Plots with squeezing can be found in /ligo/gitcommon/NoiseBudget/aligoNB/out/H1/lho_all_noisebudgets. No squeezing plots are in /ligo/gitcommon/NoiseBudget/aligoNB/out/H1/lho_darm_nosqz_noisebudget.

I pushed to git commit 70ca191c without elevated CTN and the associated extra traces. The relevant parameters are left commented out at the bottom of the QuantumParams file, and relevant code to plot the extra traces is commented out in the lho_all_noisebudgets script.

FYI, 
$ gpstime Aug 05 2023 12:30 UTC
    PDT: 2023-08-05 05:30:00.000000 PDT
    UTC: 2023-08-05 12:30:00.000000 UTC
    GPS: 1375273818.000000
so... this behavior seems to have started at 5:30a local time on a Saturday. Therefore *very* unlikely that the start of this issue is intentional / human change driven.

The investigation continues....
making sure to tag CAL.

Other facts and recent events:
- Attached are 2 screenshots that show the actual *digital* excitation is not changing with time in anyway.
    :: 2023-08-08_H1PCALEX_OSC7_102p13Hz_Line_3mo_trend.png shows the specific oscillator, --- PCALX's OSC7 which drives the 102.13 Hz line's EPICs channel version of its output. The minute trend shows the max, min, and mean of the output, and there's no change in amplitude.
    :: 2023-08-08_H1PCALEX_EXC_SUM_3mo_trend.png shows a trend of the total excitation sum from PCAL X. This also shows *no* change in time in amplitude.

Both trends show the Aug 02 2023 change in amplitude kerfuffle I caused that Corey found and a bit later rectified -- see LHO:71894 and subsequent comments, but that was done, over with an solved, definitely by Aug 03 2023 UTC and unrelated to the start up of this problem.

It's also well after I installed new oscillators and rebooted the PCALX, PCALY, and OMC models on Aug 01 2023 (see LHO:71881).

The front-end version of the calibration's systematic error at 102.13 Hz also shows the long, time-dependent issue -- this will allow us to trend the issue against other channels

Folks in the calibration group have found that the online monitoring system for the
    -  overall DARM response function systematic error
    - (absolute reference) / (Calibrated Data Product) [m/m] 
    - ( \eta_R ) ^ (-1) 
    - (C / 1+G)_pcal / (C / 1+G)_strain
    - CAL-DELTAL_REF_PCAL_DQ / GDS-CALIB_STRAIN
(all different ways of saying the same thing; see T1900169) in calibration at each PCAL calibration line frequency -- the "grafana" pages -- are showing *huge* amounts of systematic error during these times when the amplitude of the line is super loud.

Though this metric is super useful because it's dreadfully obvious that things are going wrong -- this metric is not in any normal frame structure, so you can't compare it against other channels to find out what's causing the systematic error.

However -- remember -- we commissioned a front-end version of this monitoring during ER15 -- see LHO:69285.

That means the channels 
     H1:CAL-CS_TDEP_PCAL_LINE8_COMPARISON_OSC_FREQ        << the frequency of the monitor
    H1:CAL-CS_TDEP_PCAL_LINE8_SYSERROR_MAG_MPM        << the magnitude of the systematic error
    H1:CAL-CS_TDEP_PCAL_LINE8_SYSERROR_PHA_DEG        << the phase of the systematic error 

tell you (what's supposed to be***) equivalent information.

*** One might say that "what's suppose to be" is the same as "roughly equivalent" due to the following reasons: 
    (1) because we're human, the one system is displaying the systematic error \eta_R, and the other is displaying the inverse ( \eta_R ) ^ (-1) 
    (2) Because this is early-days in the front-end system, it uses the "less complete" calibrated channel CAL-DELTAL_EXTERNAL_DQ rather than the "fully correct" channel GDS-CALIB_STRAIN

But because the problem is so dreadfully obvious in these metrics, even though they're only *roughly* equivalent, you can see the same thing.
In the attached screenshot, I show both metrics for the most recent observation stretch, between 10:15 and 14:00 UTC on 2023-Aug-09.

Let's use this front-end metric to narrow down the problem via trending.

There appears to be no change in the PCALX analog excitation monitors either.

Attached is a trend of some key channels in the optical follower servo -- the analog feedback system that serves as intensity stabilization and excitation power linearization for the PCAL's laser light that gets transmitted to the test mass -- the actuator of which is an acousto-optic modulator (an AOM). There seems to be no major differences in the max, min, and mean of these signals before vs. after these problems started on Aug 05 2023.

H1:CAL-PCALX_OFS_PD_OUT_DQ
H1:CAL-PCALX_OFS_AOM_DRIVE_MON_OUT_DQ

I believe this is caused by the presence of another line very close to the 102.13 Hz pcal line.  This second line is present at the start of a lock stretch but seems to go away as the lock stretch continues.  I have attached a plot showing a zoom-in on an ASD around 102.1-102.2 Hz right after a lock stretch (orange), where the second peak is evident, and well into a lock stretch (blue) where the PCAL line is still present, but the second peak right below it in frequency is gone.  This ASD is computed using an hour of data for each curve, so we can get the needed resolution for these two peaks.

I don't know the origin of this second line.  However, a quick fix to the issue could be moving the PCAL line over by about a Hz.  The second attached plot shows that the spectrum looks pretty clean from 101-102 Hz, so somewhere in there would be probably be okay for a new location of the PCAL line.

Since it looks like the additional noise is at 102.12833 Hz, I did a quick check in Fscan data from Aug 5 for channels where there is high coherence with DELTAL_EXTERNAL at 102.12833 but *not* at 102.13000 Hz. This narrows down to just a few channels:

• H1:PEM-EX_MAG_EBAY_SEIRACK_{Z,Y}_DQ .
• H1:PEM-EX_ADC_0_09_OUT_DQ
• H1:ASC-OMC_A_YAW_OUT_DQ. Note that other ASC-OMC channels (Fscan tracks A,B and PIT,YAW) see high coherence at both frequencies.

(lines git issue opened as we work on this.)

As a result of Ansel's discovery, and conversation on the CAL call today -- I've moved the calibration line frequency from 102.13 to 104.23 Hz. See LHO:72108.

This line may have appeared in the previous lock the day before (Aug 4). The daily spectrogram for Aug 4 shows a line near 100 Hz starting at 21:00 UTC. 

Looking at alogs leading up to the time Derek notes above, I noticed that Gabriele retuned and tested new LSC FF. This change may be related to this new peak. Remembering some issues we had recently where DHARD filter impulses were ringing up violin modes, I checked the new LSC FF filters and how they are engaged in the guardian. Some of them have no ramp time, and the filter bank is turned on immediately along with the filters in the guardian. I have no idea why that would cause a peak at 102 Hz, but I updated those filters to have a 3 second ramp.

Reloaded the H1LSC model to load in Elenna's filter changes

Now that the calibration line has been moved, the comb-like structure at the calibration line frequency is no longer present (checked in the CLEAN channel).

We can also see the shape of the 102.12833 Hz line much more clearly without the overlapping calibration line. I have attached a plot for reference on the width and shape.

As discussed in todays commissioning meeting, I checked TMSX and ETMX movement for a kick during locking and couldn't see anything suspicious. I did find some increase motion/noise every 8Hz in TMSX 1s into ENGAGE_SOFT_LOOPS when ISC_LOCK isn't explicitly doing anything, plot attached. However this noise was present prior to Aug 4th, (July 30th attached).

TMS is suspicious as Betsy found that TMS's have violin modes ~103-104Hz.

Jeff draws attendtion to 38295, showing modes of quad blade springs above 110Hz, and 24917 showing quad top wire modes above 300Hz.

Elenna's notes with calibration lines off (as we are experimenting with for current lock) we can see this 102Hz peak at ISC_LOCK state ENGAGE_ASC_FOR_FULL_IFO. We were mistaken.

To preserve documentation, this problem has now been solved, with more details in 72537, 72319, and 72262.

The cause of this peak was a spurious, narrow, 102 Hz feature in the SRCL feedforward that we didn't catch when the filter was made. This has been been fixed, and the cause of the mistake has been documented in the first alog listed above so we hopefully don't repeat this error.

I've made this alert condition a bit smarter. IFO_NOTIFY will now wait for 20 minutes after reaching NOMINAL_LOW_NOISE for the camera servos to turn on before moving to 'ALERT_ACTIVE.' If ADS converges and the camera servos turn on before then (as we'd expect), the timer is stopped and a new 3 minute timer starts to indicate we've actually reached the point where nominally we would move to OBSERVE. If that timer expires, IFO_NOTIFY moves to 'ALERT_ACTIVE' to indicate something is preventing the move to OBSERVE.

These changes are loaded into the guardian and committed to svn, revision 26132.

New stock arrived and has been added to ICS. Will be stored in the totes in the TCS LVEA cabinet. 

• ITMX: Superluminescent Diode QSDM-790-5 
  • S/N 11.21.380
  • S/N 11.21.382
  • S/N 05.21.346 (note the data sheet is labeled 04.21.346 but QPhotonics noted this is a typo) 
• ITMY: Superluminescent Diode QSDM-840-5
  • S/N 11.21.303

• ITMX: Superluminescent Diode QSDM-790-5 
  • S/N 06.18.002 - used spare
  • S/N 11.21.380
  • S/N 11.21.382
  • S/N 05.21.346 - Installed July 2023 71476
  • S/N 07.14.255 Old sled, removed 71476
• ITMY: Superluminescent Diode QSDM-840-5
  • S/N 03.20.479 
  • S/N 06.16.005 
  • S/N 11.21.303 - Installed July 2023 71476
  • S/N 11.17.127 Old sled, removed 71476

ISC has been updated. As of August 2023, have 2 spare SLEDs for each ITM HWS.  

• ITMX Superluminescent Diode QSDM-790-5 
  • S/N 06.18.002 - used spare
  • S/N 11.21.380 - Installed Oct 2023 73371
  • S/N 11.21.382
  • S/N 05.21.346 - Old sled, removed  73371
  • S/N 07.14.255 Old sled, removed 71476
• ITMY: Superluminescent Diode QSDM-840-5
  • S/N 03.20.479 - Installed Oct 2023 73371
  • S/N 06.16.005 
  • S/N 11.21.303 - Old sled, removed  73371
  • S/N 11.17.127 Old sled, removed 71476

ISC has been updated. As of October 2023, have 1 spare SLEDs for each ITM HWS, with more ordered.  

Spare 8240nm SLEDs QSDM-840-5 09.23.313 and QSDM-840-5 09.23.314 arrived and will be placed in the TCS cabinets on Tuesday. We are expecting qty 2 790nm SLEDs too. 

Spare 790nm SLEDs QSDM-790-5--00-01.24.077 and QSDM-790-5--00-01.24.079 arrived and will be placed in the TCS cabinets on Tuesday. 

In 84417, we swapped:

• HWSX from QSDM-790-5 11.21.380 to 11.21.382
• HWSX from QSDM-840-5 03.20.479 to 06.18.005

The removed SLEDs have been dispositioned, DEFECT-TCS-7839.

In 87353 we removed 11.21.382 and installed 01.24.077.