I'm still not sure what happened, but the SUS_CHARGE Guardian had a connection error for the H1:SUS-ITMX_L3_DRIVEALIGN_L2L filter bank, but it looks like it should have been a GAIN. Looks like this happened last week as well - alog71899. Before Camilla came in and pointed that out to me, I first tried stopping the node and then re-exec'ing it to create new epics connections, but this didn't work. I took the node to DOWN and then tried the injections again, but then stopped it a bit too early. I also fixed some tabs to spaces that might also have been creating some guardian confusion.

Fixed now and we ran it past the error point then brought it to DOWN for maintenance.

TITLE: 08/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY: SUS_CHARGE ran into a connection error (see separate alog), maintenance has began with minor activities on site.

TITLE: 08/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Came in to the detector in the process of relocking, and during my shift we had two locklosses, one unknown, one known, but the detector locked itself back up quickly each time without a need for initial alignment or any help. We are currently Observing and have been Locked for 2hrs 25mins.

23:00 Relocking from lockloss that happened before I arrived (72201)

23:46:40 NOMINAL_LOW_NOISE
00:00:26 Observing

02:09 Lockloss(72209), I am taking us to LOWNOISE_LENGTH_CONTROL for 102Hz peak test(72214)
03:13 Reached NOMINAL_LOW_NOISE

03:57 Lockloss caused by me and Jenne, we are each taking 50% of the blame :( (72211)
04:41 Reached NOMINAL_LOW_NOISE
05:11 Observing

LOG:

no log

In investigating the cause of the peak at 102Hz (72064, 72108), the cause had been narrowed down in between a calibration lines issue or an LSC FF issue. Since the change to the LSCFF ramp times did not fix the peak (72188), Jeff and Ryan S added a new ISC_LOCK state today(well 8/14 22:45UTC) (72205), TURN_ON_CALIBRATION_LINES, so that the calibration lines aren't turned on until late in the locking process.

In the following locking sequence (72201), the peak still appeared around the LOWNOISE_LENGTH_CONTROL and TURN_ON_CALIBRATION_LINES states, so it was decided that the next time we lost lock, we would pause on the way back up at LOWNOISE_LENGTH_CONTROL to determine which of the two states the peak was linked to.

We lost lock at 2:09UTC (72209), so I set the detector to only go through to the state right before LOWNOISE_LENGTH_CONTROL (I selected LOWNOISE_ESD_ETMY when it should've been LOWNOISE_ESD_ETMX but that presumeably wouldn't which of the two states the peak turned on at), and then selected LOWNOISE_LENGTH_CONTROL and the 102Hz peak appeared quickly after (full spectrum, zoomed in). Once we had been in that state for a bit I moved to TURN_ON_CALIBRATION_LINES to see if that would cause the peak to change in any way, but it didn't change(zoomed in - not the best ss sorry). So the peak at 102Hz is caused by one of the filter gains in LOWNOISE_LENGTH_CONTROL.

Lockloss @ 3:57 due to me having selected the wrong state to stop at before going into LOWNOISE_LENGTH_CONTROL for our 102Hz peak test :(. Currently relocking and everything is going well!!

After losing lock at 2:09UTC, I started relocking the detector but had it stop right before it got to LOWNOISE_LENGTH_CONTROL so we could see whether the 102Hz peak (72064) is related to the LSC filter gains turning on or the calibration lines turning on. We got our answer, so I continued and we are currently in NOMINAL_LOW_NOISE waiting for ADS to converge so we can go into Observing.

Lockloss at 2:09UTC. Got an EX saturation callout immediately before.

In relocking, I will be setting ISC_LOCK so we go to LOWNOISE_LENGTH_CONTROL instead of straight to NOMINAL_LOW_NOISE, to see if the 102Hz line is caused by LOWNOISE_LENGTH_CONTROL or by the calibration lines engaging at TURN_ON_CALIBRATION_LINES (72205).

R. Short, J. Kissel

Following investigations into the 102Hz feature that has been showing up for the past few weeks (see alogs 72064 and 72108 for context), it was decided to try a lock acquisition with the calibrations lines off until much later in the locking sequence. What this means in more specific terms is as follows:

• ISC_LOCK has a new state called TURN_ON_CALIBRATION_LINES (index=562) which will happen in the nominal locking sequence between LOWNOISE_LENGTH_CONTROL and INCREASE_DARM_OFFSET. This state sets the gains for all three ETMX stages' calibration lines according to lscparams (fixed the varying line heights in alog 71970).
• The PREP_FOR_LOCKING state now only turns off the lines in the main method and does not turn them back on.
• The TRANSITION_FROM_ETMX state used to turn the lines off. This is now no longer needed.
• The NOMINAL_LOW_NOISE state no longer makes sure the lines are on or turns them on at all.
• The NLN_CAL_MEAS state no longer saves the current line gains before turning the lines off to reapply them when they're turned back on. The state simply sets the gains to zero, then sets them back to their defined value from lscparams when moving back to NOMINAL_LOW_NOISE.

For each of these points where the calibration lines are turned off or on, the code structure is now consistent. Each time "the lines are turned on/off," the cal lines for ETMX stages L1, L2, and L3 (CLK, SIN, and COS), PCal X, PCal Y, and DARMOSC are all toggled.

The OMC and SUSETMX SAFE SDF tables have been updated appropriately (screenshots attached).

These changes were implemented on August 14th around 22:45 UTC, are committed to svn, and ISC_LOCK has been loaded.

TITLE: 08/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Commissioning time this morning while L1 was down and observing this afternoon until a lockloss shortly before the end of shift.

• 1519 Seismic back to CALM following Guam earthquake
• 1638 Dropped observing for opportunistic commissioning
• 1800 Started running fiber from CUR to MSR, lifting floor tiles (tagging DetChar, according to Marissa W. this has been visible in the past)
• 2018 Resumed observing
• 2044 Lockloss - alog

H1 is relocking automatically, currently locking PRMI.

LOG:

TITLE: 08/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 4mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.16 μm/s
QUICK SUMMARY:

Detector went down a bit before I arrived (72201) for unknown causes. Currently working its way back up and is doing well.

Lockloss @ 22:44 UTC - cause currently unknown, seemed fast

Today I took "unbiased" OLGs of INP1 P and Y (67187). I have plot the measurements with error shading.

INP1 P has a UGF of about 0.036 Hz and phase margin of 87 deg. This UGF seems very low for the target Gabriele and I had when we redesigned this loop (69108). I think should be closer to 0.1 Hz. INP1 Y has a UGF of about 0.25 Hz with phase margin 35 deg, which is higher than I would have expected for our target. Time permitting, I will look into the design of both of these loops and see if there are any adjustments worth making.

You can find the measurement templates, exported data, and processing code in '/ligo/home/elenna.capote/DRMI_ASC/INP1'.

The templates for the these measurements are also saved in [userapps]/asc/h1/templates/INP1 as 'INP1_{P,Y}_olg_broadband_shaped.xml'.

As a reminder, INP1 controls IM4 and is sensed on a combination of REFL RF45 WFS.

Follow up on previous tests (72106)

First I injected noise on SR2_M1_DAMP_P and SR2_M1_DAMP_L to measure the transfer function to SRCL. The result shows that the shape is different and the ratio is not constant in frequency. Therefore we probably can't cancel the coupling of SR2_DAMP_P to SRCL by rebalancing the driving matrix. Although I haven't thought carefully if there is some loop correction I need to do for those transfer functions. I measured and plotted the DAMP_*_OUT to SRCL_OUT. transfer functions. It might still be worth trying to change the P driving matrix while monitoring a P line to minimize the coupling to SRCL.

Then I reduced the damping gains for SR2 and SR3 even further. We are now running with SR2_M1_DAMP_*_GAIN = -0.1 (was -0.5 for all but P that was -0.2 since I reduced it yesterday). Also SR3_M1_DAMP_*_GAIN = -0.2 (was -1). This has improved a lot the SRCL motion and also improved DARM RMS. It looks like it also improved the range.

Tony has accepted this new configuration in SDF.

Detailed log below for future reference.

Time with SR2 P gain at -0.2 (but before that too)
from    PDT: 2023-08-10 08:52:40.466492 PDT
        UTC: 2023-08-10 15:52:40.466492 UTC
        GPS: 1375717978.466492
to      PDT: 2023-08-10 09:00:06.986101 PDT
        UTC: 2023-08-10 16:00:06.986101 UTC
        GPS: 1375718424.986101

H1:SUS-SR2_M1_DAMP_P_EXC  butter("BandPass",4,1,10) ampl 2
from    PDT: 2023-08-10 09:07:18.701326 PDT
        UTC: 2023-08-10 16:07:18.701326 UTC
        GPS: 1375718856.701326
to      PDT: 2023-08-10 09:10:48.310499 PDT
        UTC: 2023-08-10 16:10:48.310499 UTC
        GPS: 1375719066.310499

H1:SUS-SR2_M1_DAMP_L_EXC  butter("BandPass",4,1,10) ampl 0.2
from    PDT: 2023-08-10 09:13:48.039178 PDT
        UTC: 2023-08-10 16:13:48.039178 UTC
        GPS: 1375719246.039178
to      PDT: 2023-08-10 09:17:08.657970 PDT
        UTC: 2023-08-10 16:17:08.657970 UTC
        GPS: 1375719446.657970

All SR2 damping at -0.2, all SR3 damping at -0.5
start   PDT: 2023-08-10 09:31:47.701973 PDT
        UTC: 2023-08-10 16:31:47.701973 UTC
        GPS: 1375720325.701973
to      PDT: 2023-08-10 09:37:34.801318 PDT
        UTC: 2023-08-10 16:37:34.801318 UTC
        GPS: 1375720672.801318

All SR2 damping at -0.2, all SR3 damping at -0.2
start   PDT: 2023-08-10 09:38:42.830657 PDT
        UTC: 2023-08-10 16:38:42.830657 UTC
        GPS: 1375720740.830657
to      PDT: 2023-08-10 09:43:58.578103 PDT
        UTC: 2023-08-10 16:43:58.578103 UTC
        GPS: 1375721056.578103

All SR2 damping at -0.1, all SR3 damping at -0.2
start   PDT: 2023-08-10 09:45:38.009515 PDT
        UTC: 2023-08-10 16:45:38.009515 UTC
        GPS: 1375721156.009515

J. Kissel, A. Neunzert, E. Goetz, V. Bossilkov

As we continue the investigation in understanding why the noise the region around 102.13 Hz gets SUPER loud at the beginning of nominal low noise segments, and the calibration line seems to be reporting a huge amount of systematic error (see investigations in LHO:72064), Ansel has found that some new electronics noise has appeared in the end station as of Saturday Aug 5 2023 around 05:30a PT at frequency extremely, and unluckily close to the 102.13000 Hz calibration line -- something at 102.12833 Hz; see LHO:72105.

While we haven't yet ID'd the cause, and thus no have a solution -- we can still change the calibration frequency to move it away from this feature in hopes that there're not beating together terribly like that are now.

I've changed the calibration frequency line frequency to 104.23 Hz as of 21:13 UTC on Aug 09 2023.

This avoids 
    (a) LLO's similar frequency at 101.63 Hz, and 
    (b) because the former frequency, 102.13 Hz was near the upper edge of the 9.33 Hz wide [92.88, 102.21) Hz pulsar spin down, "non-vetoed" band, this new frequency 104.23 Hz skips up to the next 18.55 Hz wide "non-veto" band between [104.22, 122.77) Hz according to LHO:68139.

Stay tuned 
   - to see if this band-aid fix actually helps, or just spreads out the spacing between the comb, and
   - as we continue to investigate the issue of from where this thing came.

Other things of note: 

Since
   - this feature is *not* related to the calibration line itself, 
   - this calibration line is NOT used to generate any time-dependent correction factors and thus the calibration pipeline itself, nor the data it produces is affected
   - this calibration line is used only to *monitor* the calibration systematic error, 
   - this feature is clearly identified in an auxiliary PEM channel -- and that same channel *doesn't* see the calibration line
we conclude that there *isn't* some large systematic error that is occuring, it's just the calculation that's getting spoiled and misreporting large systematic error.
Thus, we make NO plan to do anything on further with the calibration or systematic error estimate side of things from this.

We anticipate that this now falls squarely into the noise subtraction pipeline's shoulders. Given that this 102.12833 Hz noise has a clear witness channel, and the noise creates non-linear nastiness, I expect this will be an excellent candidate for offline non-linear / NONSENS cleaning.


Here's the latest list of calibration lines:
Freq (Hz)   Actuator                   Purpose                      Channel that defines Freq             Changes Since Last Update (LHO:69736)     
15.6        ETMX UIM (L1) SUS          \kappa_UIM excitation        H1:SUS-ETMY_L1_CAL_LINE_FREQ          No change
16.4        ETMX PUM (L2) SUS          \kappa_PUM excitation        H1:SUS-ETMY_L2_CAL_LINE_FREQ          No change
17.1        PCALY                      actuator kappa reference     H1:CAL-PCALY_PCALOSC1_OSC_FREQ        No change
17.6        ETMX TST (L3) SUS          \kappa_TST excitation        H1:SUS-ETMY_L3_CAL_LINE_FREQ          No change
33.43       PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC4_OSC_FREQ        No change
53.67         |                            |                        H1:CAL-PCALX_PCALOSC5_OSC_FREQ        No change
77.73         |                            |                        H1:CAL-PCALX_PCALOSC6_OSC_FREQ        No change
104.23        |                            |                        H1:CAL-PCALX_PCALOSC7_OSC_FREQ        FREQUENCY CHANGE; THIS ALOG
283.91        V                            V                        H1:CAL-PCALX_PCALOSC8_OSC_FREQ        No change
284.01      PCALY                      PCALXY comparison            H1:CAL-PCALY_PCALOSC4_OSC_FREQ        No change
410.3       PCALY                      f_cc and kappa_C             H1:CAL-PCALY_PCALOSC2_OSC_FREQ        No change
1083.7      PCALY                      f_cc and kappa_C monitor     H1:CAL-PCALY_PCALOSC3_OSC_FREQ        No change
n*500+1.3   PCALX                      Systematic error lines       H1:CAL-PCALX_PCALOSC1_OSC_FREQ        No change (n=[2,3,4,5,6,7,8])

Benoit, Ansel, Derek

Benoit noticed that for recent locks, the 102.13 Hz calibration line is much louder than typical for the first few hours of the lock. An example of this behavior is shown in the attached spectrogram of H1 strain data on August 5 - this is the first day this behavior appeared. Ansel noted that this feature includes a comb-like structure around the line that is only present in the H1:GDS-CALIB_STRAIN_NOLINES channel and not H1:GDS-CALIB_STRAIN (see spectra for CALIB_STRAIN and CALIB_STRAIN_NOLINES on Aug 5). This issue also visible in the PCAL trends for the 102.13 Hz line. 

We are not sure if the excess noise near 102.13 Hz is from the calibration line itself or another noise source that is near the line. However, the behavior has been present for every lock since 12:30 UTC on August 5 2023.