Looking into the Violins this morning I noticed that Violin ITMY Modes 5 and 8 were still high, and had been rising for several hours.
Please see Attached screen shots of ndsopes and Gains. I have not interfered yet.

TITLE: 05/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 126Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.17 μm/s 
QUICK SUMMARY:
Current IFO Status: Locked in NOMINAL_LOW_NOISE

The fire water tank is low so I have started the well pump to replenish the tank. The pump should run ~4 hours. 

I've clicke the Observation Intent bit.  Since all the SDFs are clear, the IFO should go to Observing as soon as the Violin damping guardian returns True.

I've reverted EX cal line amplitudes, as RyanS and Ibrahim have done several times over the last few days. Calibrators, please have a look and change guardian to set these to the values you want.

I've also accepted FM4 of ITMY Mode 9 being off, since that's what guardian seems to have done.  Rahul, please have a look.

After deciding to finish PEM injections for the day due to a lockloss from nominal low noise maybe 20 min in (cause not obvious), I waited to relock hoping to set to observe for the night. However, we lost lock again in MAX_POWER state. I leave it to try to relock itself, already past DRMI, but will not be here to set it to observe. 
No intentional disturbances are running.

Because the ALS beatnotes were low earlier in the day, I had moved PR3 a hair: -118.4 to -118.0 P and -151.6 to -151.4 Y. This should be reset when the ALS table is re-aligned this coming Tuesday, it's only a bandaid until then.

Anamaria, Adrian, Robert

In Sept. of 2020 I checked all four cryobaffles and recommended that all but the ITMX cryobaffle be damped (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=56857).  We damped 2 of the 3 recommended ones and we even damped the one I didn’t recommend, but we were unable to damp the ETMX  cryobaffle because there was no incursion at that station.  Damping worked well on the cryobaffles that were damped (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=65621).

The undamped ETMX cryobaffle appears to be the source of the 4.1 Hz noise comb in DARM. The figure shows a spectrogram comparison of a 3-5 Hz swept sine injection using the architectural shaker attached to the manifold reduction flange at EX in 2020 and now. The 4.1 Hz frequency and the Q of the decaying scatter shelf  are very similar in 2020 and now, but the SNR of the scattered light noise is much higher now, at least ten times worse.  The figure also shows, on the second page, DTT spectra as the 4.1 Hz harmonics become a shelf as the shaking amplitude increases.  While the mechanical shaker produced the 4.1 Hz noise in DARM, the HEPI shaking that I did at EX a few weeks ago did not, further supporting the contention that it is the cryobaffle and not the ACB (which may also have a 4 Hz resonance), or anything else isolated by HEPI.

One possibility is that the increase in arm power resulted in more scattered light at EX, but the first appearance that Gabriele found of the 4 Hz comb, Jan. 31 of this year (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=69406), doesn’t seem to coincide with a power increase.  Scattered light might also have increased from alignment changes.

I think that I could damp the EX cryobaffle through a viewport in a short vent like I did with the Swiss Cheese baffle (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=36147 ).

[Jenne, Anamaria]

Anamaria ran an initial alignment this afternoon, but noted that OM1 wasn't getting cleared after the Input_Align step. I had thought I fixed this a few weeks ago, but we should go back and double check that it gets cleared so that we can finish initial alignment automatically.

Also, I trended the ITM green camera error signals, and it looks like they've been drifting away from 0 throughout the day (which is why initial alignment was needed).  They are uncontrolled except for initial alignment, but they should still usually be roughly stationary.  I assume it's the heat, but all of the VEA temperatures on the control room screenshots look fine at first glance.  Anyhow, maybe things are all fine, and it was just time for an alignment, but noting in case it's helpful to look at Monday morning.

TITLE: 05/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
SHIFT SUMMARY:

Locks:

Locklosses at 16:33 UTC (DRMI) and 16:34 UTC (Locking arms green).

Lock/NLN acquired at 18:12 - in comissioning for PEM Injections.

Lockloss at 22:37 UTC (DRMI) - the cause of this is unknown and it doesn't seem to be the wind (no gusts when trending for 1 hr before lockloss). Lockloss at 23:00. Shift ended while guardian was relocking.

Manual Touching:

• BS - was badly misaligned
• PRM - to help align BS
• SRM - just requested misaligned then realigned automatically to help align BS
• Manually paused guardian while realigning BS by hand since it was failing and bouncing between PRMI/DRMI states.

Other:

Before brief (sub 1 minute due to PEM starting injections) observing this morning, I REVERTED 7 SDF diffs. These 7 were the exact same (channels and change types/magnitudes) as the ones Ryan S reverted in alog 69565 during his Friday EVE shift.

• The first 6 relate to EX Cal line changes (Screenshot 1) - all originally changed within the same second right before guardian's auto lock early Sunday Morning (pre-shift) - Screenshot 1
• The last 1 relates to a nonexistent (?) IY L2 Mode 9 damping filter (F4) bank change - Screenshot 2
• Trending shows that the wind is what caused the lockloss yesterday - that particular gust of wind was the highest in 5 days with the last 5 equally (or more) powerful gusts over the last month also causing locklosses. However, trending the wind with sensitivity didn't starkly show that the wind was what only caused the issues relocking yesterday since there are many instances where we have re-locked with equally high wind speed periods (following an equally heavy gust). The sensitivity didn't always go down with lesser (but still prominent) gusts of wind. Overall inconclusive as to what happened yesterday other than the wind and something else (perhaps the diff beatnotes per Sheila's suggestion). It seems that Elenna's initial alignment and relocking late last night fixed it since we were able to relock (and keep lock) relatively easily today, albeit with slighlty lower windspeeds (only slightly though).
• PEM Injection crew was told to put the IFO into observing after they were done injecting.

LOG:                                                                                                                                                                                                                          

Sun May 14 10:11:09 2023 INFO: Fill completed in 11min 8secs

TITLE: 05/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Wind
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 18mph Gusts, 14mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.13 μm/s
QUICK SUMMARY: IFO has been locked for an hour. Working to get it to observing now.

Once the wind quieted down, I logged back in to relock. The DIFF beatnote was nonexistent, so I took us through initial alignment. It worked all the way through, no errors. Once that completed, I took us through locking. I was suprised that the IFO couldn't lock DRMI on the first try and had to go through PRMI after the timer ran out. But once PRMI locked DRMI was no problem. We are headed all the way back up to NLN for the night.

TITLE: 05/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Wind
SHIFT SUMMARY:
Locks:

The first lockloss was at 18:26 following a 16:22 hour lock with observing until 15:35 UTC.

Locklosses at: 22:12, 22:04, 21:59, 21:53, 21:50, 21:36, 21:32, 21:25, 21:12,21:09, 21:03, 19:36, 19:28, 19:21, 18:51 (all UTC) all assumed due to the same issue. Troubleshooting the issue is described below.

Manual Touching:

• BS P
• SRM P/Y
• SR2 (to align SRM)
• ALS P/Y X/Y

Troubleshooting:

1. Failed initial alignment: there was a failed initial alignment (assumed) due to an ALS X issue. The issue is tricky to trend due to Beckhoff error trending problems (I am told) - but the ALS X guardian was bouncing to fault every so often and unable to offload to get past initial alignment - this seems to be fixed by an IX alignment during the last initial alignment.
2. SRM misalignment (see below).
3. OM misalignment: This was TJs suggestion since we had issues trending DRMI. Trending the OMs and checking the OSEM values - it seems that this is not the issue.
4. Wind: Wind speeds are showing as 20-40 MPH for the entirety of the locking difficulty sometimes flashing yellow above the nominal (green) indicator - this is from the PEM weather overview
5. SR3 was moved (back) yesterday due to it being moved the day before because of temp excursions. Since IFO was stuck at the same spot, this was the initial assumption for the issue but upon trending the same channels, we found it was not and that the OPLEVs were in the right spot. (it was not touched either).

SRM Misalignment:

The problem is that the SRC1 Pitch was heavily misaligned (either due to the failed initial alignment or prior). Upon using the usual troubleshooting page, the "misaligned?" checker showed that nothing was super misaligned since the last 24 hrs. However, when we trended the SRM P/Y channels, we saw that the SRC1 P/Y were not converging at all and was heavily misaligned. Figuring that this was an issue with the failed initial alignment, we tried another initial alignment. This time, guardian got stuck at align SRC so we took it into our own hands and did a manual SRC alignment per the initial alignment checklist. We had to move SRM by a lot (see screenshot) due to it being kicked badly right after the failed initial alignment (one could only assume this is what caused it to terribly misalign though this is unknown). After this was tackled, initial alignment went fine and then we went onto normal locking. Now we are losing lock at ALS every single time due to what can only be assumed to be the wind or an actual fault/issue with ALSX (ALSY is fine and catches great every time). On the rare occasion that ALS offloads, we lose lock for no apparent reason when IR is found just as we are "supposed" to go continue (either at Check MICH or at DRMI 1F). SRM was badly railed and not converging properly. Now that it has been realigned, in theory everything should work when the wind dies down. Overall, might just be a natural issue (wind) messing with a very bad (now fixed) alignment issue. Since I'm operating tomorrow, I'll see if this is the case (given low wind).

Other:

Called TJ - helped troubleshooting.

Got Called by Sheila who also helped with troubleshooting and suggesting going to down (along with Jenne later) until the wind dies down.

Huge thank you to Elenna who came in and continued troubleshooting with me. She also informed/taught me which confusions were valid and which were NOS (new operator syndrome). This was all on her own time during the weekend after giving a tour - thanks!

LOG:

Sat May 13 10:09:16 2023 INFO: Fill completed in 9min 15secs

Continuing my work from last night on getting the magnetic injection suite ready for O4, I've finished tuning all frequency bands on all seven coils we'd like to run injections through. The exception to this is the EY VEA coil, as I could not see a response in the nearby magnetometers when trying to drive it. We do not yet have remote control over the AC Pulizzi unit for this coil, so someone may need to manually turn it on until we have that ability.

Following a discussion with Robert this evening, I tuned an additional high frequency band (1kHz-4kHz) on each coil in hopes to reach the Nyquist frequency of the magnetometers, although we're not sure if this will prove useful as it's challenging to drive the coils enough in this frequency band to get a response above background. I populated the appropriate filter banks in the PEMCS, PEMEX, and PEMEY front-ends with the 6th-order elliptical bandpass filter and 40db attenuation we use for the other injections and loaded them.

After tuning the coils I could, I added the injection states into the PEM_MAG_INJ guardian for these new high frequency injections and ran the magnetic injection suite with 128 second injection duration. All 18 injections (plus time for a background taken beforehand) took a total of 47 minutes and ran from 05:44 to 06:31 UTC. The injection log for this run is:

/ligo/www/www/exports/pem/WeeklyMagneticInjection/logs/1367991892.txt

The injection suite is not currently set to run automatically on Tuesday morning, but one could set it by simply requesting INJECTIONS_COMPLETE in the PEM_MAG_INJ guardian. The start time is read in from the associated injparams.py file, so one should look there if they'd like to change the start time of the injections.

Two stages of whitening were enabled for the OMC-REFL_A photodetector.

The slow controls DC readout is very close to +10V and sometimes saturates. The fast DAQ has more headroom and should be fine. If not the analog PD gain should be reduced to 10dB.

PS. OMC_REFL_A is using the old DCPD whitening chassis: Only the 24dB gain step works, all others will short the output. Filter stage 2 is not a whitening stage. 

Unfotunately turning off the cosmic ray power supply did not remove the intermittent 4.05 Hz bumps in DARM, even though all the CS signals that showed a large peak at 4.06 Hz are now clean.

Looking again at all the signals that contain a peak around 4.05 Hz, we now have

          H1:PEM-CS_MAG_LVEA_VERTEX_Z_DQ 4.050 Hz
               H1:PEM-EX_ADC_0_08_OUT_DQ 4.040 Hz
               H1:PEM-EX_ADC_0_09_OUT_DQ 4.040 Hz
          H1:PEM-EX_LOWFMIC_VEA_FLOOR_DQ 4.040 Hz
         H1:PEM-EX_MAG_EBAY_SUSRACK_Y_DQ 4.040 Hz
         H1:PEM-EX_MAG_EBAY_SUSRACK_Z_DQ 4.040 Hz
             H1:PEM-EX_MIC_EBAY_RACKS_DQ 4.040 Hz
      H1:PEM-EX_TEMPERATURE_BSC9_ETMX_DQ 4.040 Hz
       H1:PEM-EX_VMON_ETMX_ESDPOWER18_DQ 4.040 Hz
               H1:PEM-EY_ADC_0_14_OUT_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SEIRACK_X_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SEIRACK_Y_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SEIRACK_Z_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SUSRACK_X_DQ 4.040 Hz
         H1:PEM-EY_MAG_EBAY_SUSRACK_Y_DQ 4.040 Hz

Attached plots shows spectra and time series. Although the frequency of all peaks is very close, EX signals are only coherent among themselves, and not with EY signals, and the other way around too. So it looks like at both end stations there is a source of 4.04 Hz that is local. Maybe a clue? Still, we don't have evidence that those sources are related to the DARM noise.

Retuned ITMY angle to length gains, looking at reducing the coupling of CHARD pitch and yaw lines at 8.125 Hz.

Old settings:
P2L = -0.05
Y2L = -1.7

New settings:
P2L = 0.04
Y2L = -0.17

Coupling from CHARD_P to DARM is reduced by 12 db
Coupling from CHRAD_Y to DARM is reduced by 35 db