After finding that the new lines and a broad peak at around 1020Hz are correlated with the RF AM monitor channels and RF45 out of loop sensors but not in-loop (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23552), I went to the floor and started to move the Tenma power supply from the wall AC outlet to the power strip in the PSL rack, and move the +-17V from the ISC rack to the PSL rack.

We powered off the RF AM monitor by first disabling the +-30V output of Tenma, then disconnecting the +-17V connector from the ISC rack, then disconnecting the AC power cable of Tenma from the AC outlet. One of these actions kicked the IFO out of lock.

When the IFO locked again, the lines and the peak were still there. At this point, RF AM monitor was still connected to the EOM, but the +-17V cable was disconnected from the supply, and the +-30V supply AC power cable was disconnected. In the first attachment, the red trace ("RFAM monitor power disconnected outside of the PSL room") corresponds to this state. The second attachemnt shows the connection cartoon.

The blue trace is from this morning when RFAM monitor was connected and working (third attachment for the cartoon). Black is from a week ago, i.e. no RFAM monitor unit.

Then we totally isolated RF AM monitor from everything else by terminating the coupled output of the couplers, and disconnecting the DAQ cable from the AA chassis. Green trace ("RF AM monitor totally disconnected") corresponds to this state as well as the fourth attachment. The lines and the 1020Hz peak definitely got smaller but they're still there.

At this point, we could remove the couplers and pull everything out to completely revert back to the old configuration to see if the noise goes away.

However, doing so means breaking the lock again, and considering the relocking difficulty we experienced today, I talked with Michael Landry and we agreed to run the IFO as is till tomorrow.

As of now, LSC-MOD_RF9 channels are not measuring anything.

TITLE: 11/19 [DAY Shift]: 16:00-24:00 UTC (08:00-16:00 PDT), all times posted in UTC
STATE Of H1: Observing at ~ 77 MPc.
SHIFT SUMMARY: Most of the shift was devoted to troubleshooting RF 45 AM monitor noise.
Back to observing at 00:13 UTC. ITMX, ITMY and BS ISI blends are on 45mHz. EMTX X and ETMY Y ISI blends are on 45mHz. ETMX Y and ETMY X ISI blends are on Quite_90. Accepted SDF differences for these (see attached). Ran a2l. Winds are around 10 mph. Earthquake seismic band is between ~ 0.01 and 0.05 um/s. Microseism is between 0.1 and 0.4 um/s. From the cameras the lights are off in the LVEA, PSL enclosure, end X, end Y and mid X. I can not tell if they are off at mid Y.
INCOMING OPERATOR: Ed
ACTIVITY LOG:

17:10 Out of observing for Keita to test grounding configurations for RF AM monitor. Not going into PSL enclosure. (WP 5616)
17:39 Kyle to X28
17:48 Lock loss likely from powering down of RF AM monitor
19:08 Evan and Keita to PSL enclosure to disconnect RF AM monitor
19:25 Lock loss likely from Evan and Keita in PSL enclosure
19:37 Evan and Keita done
20:07 Kyle back from X28 for lunch
20:15 Evan to LVEA (WP 5617)
20:40 Evan done
21:26 Kyle to mid Y then X28
23:46 Evan and Keita to LVEA to disconnect RF AM monitor DAQ cable
00:08 Kyle back

TITLE: Nov 19 EVE Shift 00:00-08:00UTC (08:00-04:00 PDT), all times posted in UTC

STATE Of H1: Locking

OUTGOING OPERATOR: Patrick

QUICK SUMMARY: IFO is locked at NLN but ongoing 45MHzRFAM work has us not Observing yet. µSei is ≈.3µm/s. EQ bands ≈.03µm/s. Winds are below 20MPH.

1005-1205 and 1400-1545 hrs local -> Kyle back and forth between CS and BT port X2-8 

Ran diesel generator at BT port X2-8 between 1030-1540 hrs local

The ALS Comm and Diff "shuttered" states are now GoTo states, since we've had two or more locklosses today when shuttering ALS.  This is somewhat similar to what we saw back in August (aLog 20286) where if the ALS can't get to the shuttered state, it goes to lockloss, going through the ALS Down state, which clears things like the DARM gain.  This isn't good, since at this point we're not using ALS for DARM anymore, so it doesn't matter if the green laser loses lock.  Today, the green Xarm was losing lock before we asked for the shuttered state, which was causing the guardian to take ALS_DIFF through its down state.  Anyhow, "shuttered" is now a GoTo state so that even if the greens lose lock, we don't lose the whole IFO needlessly.

I restarted it. Screenshot of error attached.

Keita, Evan

We installed a mixer-based 45 MHz RFAM monitor using two spare 45 MHz spigots on the distribution amplifier on the ISC rack. This provides an unobtrusive way to look for 45 MHz amplitude glitches on the ISC rack (i.e., it does not require inserting anything into the main 45 MHz drive that goes to the EOM driver and the EOM). On the other hand, if the glitches are only on the main 45 MHz drive, this monitor will not tell us anything.

Set-up is as follows:

• Spigot #7 is attenuated by 5 dB and goes to mixer LO. Spigot #8 is attenuated by 12 dB and goes to mixer RF. The spigots are nominally 13 dBm, so the mixer (ZAD-1, level 7) should get 8 dBm in its LO port and 1 dBm in its RF port.
• IF of mixer is sent through an 11 MHz minicircuits LPF, and then is terminated with 50 Ω. The dc component of this output is 175 mV.
• The output is then sent into an ac-coupled SR560 with a gain of 1000. This produces a signal whose voltage ASD is 18 µV/Hz^1/2 at 100 Hz, as measured with an SR785. If the SR560 is terminated at the input with 50 Ω, the voltage ASD is 5 µV/Hz^1/2 at 100 Hz.
• The output of the SR560 is sent into LSC-EXTRA_AI_1. This should add an additional 4 µV/Hz^1/2 of noise.

I have calibrated LSC-EXTRA_AI_1_OUT so that it is in volts. This channel is not dequeued, but if we have another hours-long episode of glitching we can watch the test points in real time.

Evan and Keita are disconnecting the RF AM monitor in the PSL enclosure.
Running a2l.

The ISC_LOCK guardian was stuck at DC Readout Transition because the OMC wasn't ready, although the guardian log only said "not ready", and there was no notification that popped up.  Patrick re-requested the OMC guardian to go to Ready For Handoff (which worked), but I also added a notify to the ISC_LOCK guardian to say "OMC not ready" if this happens again.

Lost lock trying to engage ISS second loop by hand.

SUS SR2 saturating 17:48:35 UTC

Did this yesterday afternoon (11/18) but forgot to log it then.

17:10 UTC Out of observing for Keita to test grounding configurations for RF AM monitor. Not going into PSL enclosure. (WP 5616)

I've been noticing all night but not sure where from. Those nasty little lines seem to have come with a broad peak at 1040Hz. It's not the violin mode second harmonics (which ends at ~1010Hz).