Lockloss due to another of the same type of Glitch as yesterday's Lockloss Alog 72852 (and less than 20 minutes apart too). Attempting to re-lock automatically now while investigating the cause of the glitch.

IFO is in NLN as of 7:24 UTC (19:37 hr lock) and OBSERVING as of 23:12 UTC

Other:

IY violin modes 5 (and 6) have been going down steadily for the last 4 hrs since comissioning rang them up and a gain was applied to mode 5 (Screenshot). All violins are now under 10^-17 m/Hz^1/2 as of about 2:00 UTC.

Based on TJ's log 72857 about recent FC alignment drifts, I wanted to check out the situation with possible scattered light from the filter cavity since alignments can drift around (worth investigating more), and I wanted to check the CLF fiber polarization (this was basically fine).

Summary screenshot shows some measurements from today. I'm wondering if we might be closer than I realized to filter cavity backscatter. With an excitation that is ~5-10x greater than the ambient lsc control to FC2, it created scatter about 2x above darm at 100 Hz. Based on previous (higher-freq) measurements (LHO:68022) and estimates (LHO:67586), I had thought ambient LSC control was about 10-fold below DARM; this suggests we are within a factor of ~5? Though, there is a lot of uncertainty (on my end) of how hard we are driving FC2 given the suspension/loop roll-offs/etc, so I need to think more about the scaling between measured scatter and the excitation.

Strange line in FC-LSC error signal -- it seems to wander; I've seen it (or things like it) sometimes in other SQZ-related signals; but I haven't figured out where it comes from yet. It can be easily seen on SQZ summary pages (sqz > subsystems tab) since Derek and Iara helped us it up (thank you Detchar!!). I don't see it in CLF_ISS but sometimes in other error signals. Not clear to me that this is an issue if the peak is >100 Hz, but if it drifts to lower frequencies and this line is real/physical (not some random artifact), it could be problematic. The peak amplitude seems large enough that if it were in-band of the FC2 suspension and controls, it could plausibly get injected as real FC length noise and drive some measurable backscatter.

For the excitation -- I used the template from LHO:68022 but ran it to lower frequencies, in-band of FC-LSC. Compared to the FC error signal (specifcally H1:SQZ-FC_WFS_A_I_SUM_OUTPUT which is the input sensor to FC_LSC_DOF2_IN1), this DTT injection of 30,000 counts increased the integrated RMS of the in-loop error signal at 10 Hz by about 9-fold (= 3375 (w/excitation) / 387 (ambient), measured from dtt rms cursors). I injected into the fc-lsc loop at H1:SQZ-FC_LSC_DOF2_EXC, with various amplitudes (like 30k), and filter = butter("BandPass",4,10,300); this should then go to FC2_L for suspension feedback. I'm not sure that I'm using the best witness sensors for actual length noise driven in this excitation, but wasn't able to totally figure it out in time.

With this excitation going, I tried to walk the alignment to see if there was an alignment that minimizes backscatter, but I didn't figure this out in time. I tried to walk ZM2 with beam spot control off, and then set the QPD offsets where it landed. This was probably the wrong approach, since I wasn't able to then set the QPD offsets in time; maybe I should have walked the FC-QPD offsets with full ASC running at higher gain, since this loop is so slow. Might be worth trying this again for a bit; with the injection running, I wasn't sure if I was able to minimize scatter by walking ZM2 (p/y/ maybe psams?), but there were a couple directions in both pitch and yaw that looked promising. 

In the end, SDF diffs for ZM2 were accepted (ZM2 is not under asc-control), and I accepted the beam spot position change in the FC QPD Pitch offset (from 0.07 --> 0.08) while reverting the yaw change that I didn't figure out in time. I don't anticipate much overall change in the squeezer after these tests.

[Jenne, Gabriele, with thoughts and ideas from Elenna and Sheila]

The last few days, our sensitivity has been degrading a small amount, and Gabriele noted from a bruco that we're seeing increased MICH and SRCL coherence.  It hasn't even been a full 2 weeks since Gabriele and Elenna last tuned the MICH FF, so this is disappointing. Elenna has made the point in alog 72598 that the effectiveness of the MICH FF seems to be related to the actuation strength of the ETMX ESD.  We certainly see that the Veff of ETMX has been marching in a monatonic line for the last few months in Ibrahim's alog 72849. After roughly confirming that this makes sense, Gabriele and I took measurements in preparation for soon switching the LSC FF to use the ITMY PUM, just like LLO does, in hopes that makes us more immune to these gain changes.

Today, at Sheila's suggestion, I tried modifying the ETMX L3 DriveAlign L gain to counteract this actuation strength change.  (Fear not, I reverted the gain before commissioning ended, so our Observe segments do not have any change to any calibration.)  To check the effect of changing that drivealign vlaue, I looked both at the DARM open loop gain, as well as the coupling between MICH and DARM. 

• As-found, DARM OLG at 60 Hz was gain of 1.14 and 30.1 deg, when EX L3 drivealign L was at the nominal value of 184.65.  Corresponds to blue trace in first attachment.
• I adjusted the EX L3 drivealign L gain to get the DARM OLG back closer to its nominal value of 60 Hz.  After adjustment, the DARM OLG at 60 Hz is gain of 1.02, 29.8 deg, with EX drivealign 166.8. The MICHFF is quite bad though.  Ran a bb calib meas to check the calibration, although this is before Jeff reminded me that it would be useful to have the CAL-CDS_DARM_FE mimic of that drivealign gain match the actual gain: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20230913T195421Z.xml, but H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_GAIN had not been fixed to match my updated drivealign gain.  Corresponds to green trace in first attachment.
• I modified the calibration mimic of the ETMX L3 drivealign gain, ran another broadband measurement, and indeed saw that this made the calibration better.  This indicated that much of the loss in sensitivity was due to the poor LSC FF.  The bb measurement for this is at /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20230913T200136Z.xml . 
• I realized that probably we didn't want the DARM OLG to be the 60 Hz value it was when we started O4, but rather we wanted it to match what it was on the day the LSC FF was last tuned (which is closer to 65 Hz or so, from this plot from alog 72422 from Louis).  So, I switched to using drivealign to minimize MICH-> DARM coupling, and got a drivealign gain of 182.5.  This puts DARM 60 Hz values at mag 1.14 and phase of 30.3 deg.  (So, basically the same as earlier today....Hmmm. Likely some measurement uncertainty there.)  Had a small amount of quiet time when H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_GAIN (the Calib matching value) was also 182.5, so that's the channel to check for a quiet time.  Corresponds to red in first attachment. 
• Reverted ETMX L3 drivealign L gain back to 184.65 (original value, so we haven't altered our calibration).

This all seemed to jive with the MICH FF effectiveness being related to ETMX ESD actuation strength.  So, rather than try to track that, we decided to work on changing over to use the ITMY PUM like LLO does.  I note that our Transition_from_ETMX guardian state uses ITMX, not ITMY, so it should be safe to have made changes to the ITMY settings.

• Used -1 on ITMY in LSCFF output matrix, for both MICH and SRCL.  I'm not sure why LLO uses a -2 there.  Made sure signal goes through L3 filter banks but not to the ITMY ESD, and that signal does not go up to L1 or M0 (so, we'll only be actuating on L2).
• Gabriele and I copied over the f^2-ish filter that is in the LLO ITMY L2 LOCK filter bank, which is used to increase the drive at high frequencies since we're going to be driving up one stage.
• Our first try at measuring coupling functions rang up some violin modes, so Gabriele copied over a violin bandstop from the ITMY angular L2 dofs.  Our nasty doublet of 508.35 Hz violin modes is on ITMY, so perhaps that's why they got rung up, even though LLO does not use a bandstop and we didn't have one for ETMX.
• Gabriele took measurements for updating both MICHFF and SRCLFF.
• After this, we reverted everything back to using our nominal ETMX LSC FF actuation, since we didn't have time for fitting and testing the new filters today.
• Gabriele is working on fitting the new filters for the ITM, and we will come back soon to trying them.  I am hopeful that this will alleviate much of our need to retune these regularly!
• We did save the new ITMY settings in the Observe SDF (There were 2 LOCK_L gains, and then two filters engaged in ITMY L2 LOCK L).  I did not think to save these in the safe.snap file, so they will likely get reverted, and cause an SDF diff for our next lock :/ .

TITLE: 09/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 8mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.14 μm/s
QUICK SUMMARY:

IFO is in NLN as of 7:24 UTC (16 hr lock) and OBSERVING (after Wednesday commissioning) as of 23:12 UTC

TITLE: 09/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Locked for 15.5 hours, we just finished up 4.5 hours of commissioning and calibration time.
LOG:

• 1830 Out of observing for planned commissioning
• 2312 Observing

J. Kissel (after consult with J. Driggers)

While poking around the front-end computed response function systematic error yesterday and today (LHO:72848 and LHO:72863), I was reminded that one of the calibration lines used to measure the response function systematic error has been showing completely bogus large numbers for "a long time."

It's the 102.13 or 104.23 Hz PCALX calibration line.

I realize the systematic error measured at this frequency is wrong because we never updated the frequency in the DARM loop model parameter file. 
(See details below as to why this matters for both GDS and CALCS computation of the response function systematic error at this frequency.)

For reasons I outline below, it's easier to fix the problem by changing the calibration line frequency *back* to 102.13 Hz, so I've done so as of 2023-09-13 20:17:45 UTC, and the error estimate has returned to functional values around unity magnitude and zero phase by 2023-09-13 20:19:45 UTC (i.e. after the 2-minute gating, medianing, averaging algorithm catches up.)

All of this was done while we're were *out* of observing during today's commissioning window, so the *next* observation ready segment will pick up this change around 2023-09-13 ~23:00 UTC.

%%%%%%%%%%
Details & Motivation
%%%%%%%%%%

2023-06-21 Calibration updates to reflect power down to 60W PSL input, 20230621T211522Z pydarm_H1.ini file correctly has 102.13 Hz in the list of cal_line_sys_pcalx_frequencies. LHO:70693

2023-08-04 Friday afternoon, SRCL Feed-forward filter re-tuned, informed by a measurement that mistakenly has all calibration lines still ON, including 102.13 Hz line. So, FF measurement fitter installs a super high Q zero:pole pair to "compensate" for it. The resulting filter is installed in the SRCL FF path, starting a period of huge interference between this high-Q zero:pole pair in the SRCL FF and the 102.13 Hz calibration line itself. LHO:71961

2023-08-08 Next week Tuesday, DetChar Team finds "some feature" at 102.12833 Hz, and folks blame the calibration line itself since it's originally identified as appearing Saturday morning. LHO:72064

2023-08-09 That same week Wednesday, out of panic and as well as starting a "grasping at straws" test, we moved the 102.13 Hz PCALX calibration line to 104.23 Hz LHO:72108 -- but in our rush, forgot/didn't realize we needed to update down-stream computation that is impacted by this change:
    (a) The front-end demodulator that turns this line into something useful in CAL-CS needs its local oscillator frequency to change to match,
    (b) The cal_line_sys_pcalx_frequencies list within pydarm_H1.ini needs updating as well -- or else the "DARM loop model transfer function values at calibration line frequencies" EPICs records are wrong for the new frequency, which means both GDS and CAL-CS are interpreting this line incorrectly.

2023-08-29 Problem with SRCL FF filters is finally identified and removed, and the story about what happened on 2023-08-04 finally becomes clear. LHO:72537

2023-08-31 Calibration updated in order to account for the past two month's worth of commissioning and ETMX ESD actuation strength drift, but the 20230830T213653Z pydarm_H1.ini file incorrectly still has 102.13 Hz in its cal_line_sys_pcalx_frequencies list. LHO:72594
    >> I didn't aLOG it explicitly, but on this day, I noticed the (a) mistake above, and changed the CAL-CS DEMOD local oscillator frequency to 104.23 Hz for the front-end demodulation and estimation of systematic error. Because (b) was still going on, it didn't fix the problem.

2023-09-13 TODAY Realized the issue with front-end systematic error calculation was still wrong because of (b); the cal_line_sys_pcalx_frequencies list in the pydarm_H1.ini file, and reasoned it would be way easier to change the PCAL OSC and DEMOD OSC frequency back to 102.13 Hz than to go through the entire rigamarole of updating the calibration and "pushing new EPICs records." So, I moved calibration line *back* to 102.13 Hz changing the PCALX OSC7 frequency, and remembered to update the LINE8 DEMOD LO frequency to match.

First attachment: The graphical MEDM interface of where I changed the EPICs records that define the PCAL OSC7 calibration line frequency, and the CALCS LINE8 DEMOD LO frequency.
Second attachment: The PCAL OSC7 frequency change has been saved in the H1CALEX SDF system. (Only the OBSERVE.snap is show, because the h1calex model's safe and OBSERVE.snaps are the same file soft-linked together.)
Third attachment: The CALCS LINE8 DEMOD LO frequency change to match has been saved in the H1CALCS SDF system. (Only the OBSERVE.snap is show, because the h1calcs model's safe and OBSERVE.snaps are the same file soft-linked together.)
Fourth attachment: A graphical representation of the above mentioned timeline

%%%%%%%%%%%%%%
Calibration Line List Update
%%%%%%%%%%%%%%

Freq (Hz)   Actuator                   Purpose                      Channel that defines Freq             Changes Since Last Update (LHO:72108)     
15.6        ETMX UIM (L1) SUS          \kappa_UIM excitation        H1:SUS-ETMY_L1_CAL_LINE_FREQ          No change
16.4        ETMX PUM (L2) SUS          \kappa_PUM excitation        H1:SUS-ETMY_L2_CAL_LINE_FREQ          No change
17.1        PCALY                      actuator kappa reference     H1:CAL-PCALY_PCALOSC1_OSC_FREQ        No change
17.6        ETMX TST (L3) SUS          \kappa_TST excitation        H1:SUS-ETMY_L3_CAL_LINE_FREQ          No change
33.43       PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC4_OSC_FREQ        No change
53.67         |                            |                        H1:CAL-PCALX_PCALOSC5_OSC_FREQ        No change
77.73         |                            |                        H1:CAL-PCALX_PCALOSC6_OSC_FREQ        No change
102.13        |                            |                        H1:CAL-PCALX_PCALOSC7_OSC_FREQ        FREQUENCY CHANGE; THIS ALOG
283.91        V                            V                        H1:CAL-PCALX_PCALOSC8_OSC_FREQ        No change
284.01      PCALY                      PCALXY comparison            H1:CAL-PCALY_PCALOSC4_OSC_FREQ        No change
410.3       PCALY                      f_cc and kappa_C             H1:CAL-PCALY_PCALOSC2_OSC_FREQ        No change
1083.7      PCALY                      f_cc and kappa_C monitor     H1:CAL-PCALY_PCALOSC3_OSC_FREQ        No change
1153.2      PCALY                      Old "cancelling" line        H1:CAL-PCALY_PCALOSC9_OSC_FREQ        Added to the list, but has been on for all of O4.
n*500+1.3   PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC1_OSC_FREQ        No change (n=[2,3,4,5,6,7,8])

Simulines start:

PDT: 2023-09-13 11:36:48.648354 PDT
UTC: 2023-09-13 18:36:48.648354 UTC
GPS: 1378665426.648354
 

Simulines end:
PDT: 2023-09-13 11:58:49.591980 PDT
UTC: 2023-09-13 18:58:49.591980 UTC
GPS: 1378666747.591980
 

Files:

2023-09-13 18:58:49,243 | INFO | File written out to: /ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20230913
T183650Z.hdf5
2023-09-13 18:58:49,260 | INFO | File written out to: /ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20
230913T183650Z.hdf5
2023-09-13 18:58:49,269 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20
230913T183650Z.hdf5
2023-09-13 18:58:49,279 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20
230913T183650Z.hdf5
2023-09-13 18:58:49,289 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20
230913T183650Z.hdf5

Starting with a calibration measurement.

Last night we had some alignment issues after a lock loss (alog72854) and Jenne noticed that our jitter noise is higher (alog72853). Comparing two DARM spectra from the summary pages in the first attachment from two days ago to now, the ~120Hz noise is worse, though our higher freqency noise seems better now.

In terms of alignment, Ryan C's Misaligned GUI shows ITMY P being further off compared to near the end of the Sept 11 lock before maintenance, but I'm not seeing that in top mass OSEM values as seen in the 3rd attachment. Interestingly, L1 P seems to be slightly shifted from the reference time, but this isn't seen in M0 or the oplevs.

This GUI also points to SR2 being slightly off in P and Y, which it does seem to be compared to the reference time, but overall doesn't seem that far off from where it normally moves (6th attachment). PRM is another contender with larger alignment moves, but nothing out of the normal between locks (7th attachment).

The ETMs see similar movement as well (ETMX ETMY). Perhaps obvious, the most movement we see is during our move spots [508] and max power [520] states.

FC2 has been moving much more in the last 5 days. Not sure if this is a symtom or a cause, but perhaps this might point to some of our SQZ issues. FC1 shows a similar story, though doens't seem to be as drastic.

All of this to say that I don't notice any major alignment differences, aside from FCs, from before maintenance to now, but there are a few interesting bits to spend some more time looking at.

Wed Sep 13 10:06:26 2023 INFO: Fill completed in 6min 22secs

Jordan confirmed a good fill curbside.

TITLE: 09/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.14 μm/s
QUICK SUMMARY: Locked for 7.5 hours, calm environment.

CDS Overview OK, no alarms.

There are two issues to address in this entry regarding the configuration of the corner station AHU1.

1: persistent flooding of the condensate tray's was occurring which at times caused water to reach the floor of the AHU requiring that it be vacuumed up. The cause of improper draining was suspected to be a clogged drain line. The line was flushed both with compressed air and with pressurized water. Following that, a vacuum was pulled on each of the two drain lines with a wet/dry shop-vac type vacuum. Once the trap at the drain is primed, the drain line should now function without issue.

2: the likely cause of moisture to be present in the first place is in the way that the AHU was moving air. Unlike the neighboring AHU (2), The face bypass dampers in AHU1 above the cooling coil were OPEN and the dampers directly below which allow flow across the coil were CLOSED. The attached screen grab illustrates the desired configuration. The cause for the bypass dampers to have flipped was found in the supply temperature settings which were changed from a temp control mode of "auto reset" at a 52F setpoint to a temp control mode of "manual" with a 45F setpoint which now mirrors AHU2. 

This change has been in effect since midday Monday 9/11.

B. Gateley T. Guidry 

J. Kissel, D. Barker
WP #11423

Dave has graciously compiled, installed and restarted the h1calcs model. In doing so, that brings in the bug fix from LHO:72820, which fixes the issue that the front-end, CAL-CS KAPPA_UIM library block was receiving the KAPPA_TST uncertainty identified in LHO:72819.

Thus h1calcs is now using rev 26218 of the library part /opt/rtcds/userapps/release/cal/common/models/CAL_CS_MASTER.mdl.

I'll confirm that the UIM uncertainty is the *right* uncertainty during the next nominal low noise stretch later today (2023-09-12 ~20:00 UTC).

Elenna, Gabriele, Camilla 

This afternoon we updated the MICH Feedforward, it is now back to around the level it was last Friday, comparison attached. Last done in 72430. Maybe need to be done so soon because of the 72497 alignment changes on Friday.

The code for excitations and analysis has been moved to /opt/rtcds/userapps/release/lsc/h1/scripts/feedforward/

Elenna updated in guardian to engage FM1 rather than FM9 and sdf accepted. New filter attached. I forgot the accept this in h1lsc safe.snap and will ask the operators to accept MICHFF FM1 when we loose lock or come out of observe(72431), tagging OpsInfo.

Attached is a README file with instructions.

This morning between 9:20am and 10:00am I injected some noise to retune the MICH and SRCL feedforward. New filters have been uploaded with name '8-4-23'. Unfortunately the IFO lost lock before I was ready to test them.

We shoudl wait for the IFO to thermalize after relock, and then test the two new filters

First observed as a persistent mis-calibration in systematic error monitoring Pcal lines which measure PCAL / GDS-CALIB_STRAIN affecting both LLO and LHO, [LLO Link] [LHO Link], characterised by these measurements consistently disagreeing with the uncertainty envelope.
It us presently understood that this arises from bugs in the code producing the GDS FIR filters there exists a sizeable discrepancy, which Joseph Betzwieser is spear-heading a thorough investigation to correct,

I make a direct measurement of this systematic error by dividing CAL-DARM_ERR_DBL_DQ / GDS-CALIB_STRAIN , where the numerator is further corrected for kappa values of the sensing, cavity pole, and the 3 actuation stages (GDS does the same corrections internally). This gives a transfer function of the difference induced from errors in the GDS filters.

Attached in this aLog, and its sibling aLog in LLO, is this measurement in blue, the PCAL / GDS-CALIB_STRAIN measurement in orange, and the smoothed uncertainty correction vector in red. Attached also is a text file of this uncertainty correction for application in pyDARM to produce the final uncertainty, in the format of [Frequency, Real, Imaginary].