Looks like zotvac0 has gone offline and the EDC is now disconnected from all 27 channels. We'll leave it like this overnight and work on it first thing tomorrow.

TJ reverted this threshold earlier today. Around the same time (11:22pst) I made a change to the filter used for the peakmon calibration, that should reduce the confusion between microseism and earthquakes.

First picture show 3 ground spectra, red is a more normal frequency higher microseism time on the 16th, blue is a spectra take when both the primary and secondary microseism were elevated from a large mass of wave activity right off the coast and pink is a largish eq from a couple days ago. On the blue spectra  the .1-.3hz peak has spread down to 80 or 90mhz, and was bleeding into the band peakmon uses.

Second image compares the bandpass peakmon was using to the one I installed today. Blue is the new filter, a cheby1 lowpass with a zpk([0],[.015],1) high pass, green is the old total filter. The problem this week with the old filter was the higher primary microseism peak at 50-60 mhz and the broader peak around .1-.3hz extending down to  80mhz. The blue filter should greatly reduce the chance of that secondary peak confusing the SEI guardians.Rise time for this filter is important, and the blue filter has some lower frequency poles than the green, but the foton step response for the filters isn't dramatically different. The new filter takes a couple seconds longer to respond, but that should be easy to tweak.

 We've already had one successful earthquake transition about an hour after I changed the filter. Wave forecasts for next week suggest more high microseism starting this next Tues-Wed.

As part of thinking about why a little bit of excess motion around a few Hz can cause nonstationarity in the GW band, I had a cursory look at the filters in place for the ETM R0 tracking. 

I have not yet re-looked to see why we made different choices for the angular vs length loops, but the top frame of the attached plot shows the magnitude of the control filters for ETMY's R0 tracking for pitch and length.  The bottom frame shows the rough magnitude of the open loop gain of the length loop, using a filter that was in the filter bank called 'plant', and including the gain of 20 that is in the filter bank. 

So, it looks like the angular loops (I'm assuming the yaw filters, which are named the same, are actually the same as the pitch filters) emphasize this region where we've got a bit of excess motion in DARM due to the LSC FF, but the length loop does not.  We should give this some thought, but consider adding some emphasis to the length tracking filter to have the length tracking work up to ~5 Hz or so. 

A parallel path forward is that we should also consider is changing the highpass in the SRCL FF filter bank to be more severe (thus sacrificing some phase and efficacy around 10 Hz, but hopefully an overall win above 20 Hz), however this implies doing the full remeasurement and refitting of the SRCL FF. We can work on this later this week, or whenever we next have some commissioning time.

Here's a design for a more aggressive high pass filter, at the price of 20 degrees of phase rotation at 10 Hz. I think we might need a retuned of the SRCLFF after implementing it. The new high pass is saved into FM10 of SRCLFF1 (not yet loaded).

The noise reinjection between 2 and 4 Hz should be about 4 times lower if we manage to retuned the SRCLFF filter without increaasing the low frequency gain.

I looked at the quad L2 & L3 MASTER_OUT channels as well as the ETMX L2 NOISEMON and FASTIMON channels that you looked at for the 10/18 01:53UTC lockloss in 73552, and noticed a couple of things:

• this glitch was also seen by all of the other quad L2 MASTER_OUT channels as well as ASC-AS_A(attachment1)
• to me it seems very clear that the first channel to see this pre-lockloss glitch is ETMX_L3_MASTER_OUT(attachment2-zoomedin, attachment3-zoomedout)

Comparing this to the two recent locklosses that had pre-lockloss glitches, 10/15 08:53UTC and 10/18 01:53UTC(73552), both had not been caught by other quads or ASC-AS_A, and in regards to which channel had seen the glitch first, for the 10/15 LL I could not tell who saw it first (I had said that it had hit ETMX L3 first in 73552 but I think that was wrong), and for the 10/18 01:53 LL we previously discovered that either DARM or the DCPDs saw it first.

Just to clarify, the Counts term in the Power_scatter is the difference in the Oplev counts between locked and unlocked ifo.

We checked with Renate Hartog (the PNSN network manager) through a series of calls and emails. She's gotten everything going now, thanks Renate!

Turns out that they had a system re-boot at about that time and people forgot to restart a (virtual) network interface. As of 17:13 pacific, the issue has been resolved and we can now connect to the PNSN.

Hopefully things will be stable and running again soon. Thanks for the notification about this Dave.

Edgard

DARM low frequency (< 4 Hz) is highly coherent with ETMX M0 and R0 L damping signals. This might just be recoil from the LSC drive, but it might be worth trying to reduce the L damping gain and see if DARM RMS improves

Bicoherence is also showing that the noise between 15 and 30 Hz is modulated according to the main peaks visible in DARM at low frequency.

We might be circling back to the point where we need to reconsider/remeasure our DAC noise. Linking two different (and disagreeing) projections from the last time we thought about this, it has the correct slope. However, Craig's projection and the noisemon measurement did not agree, something we never resolved.

Projection from Craig: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=68489

Measurement from noisemons: https://alog.ligo-wa.caltech.edu/aLOG/uploads/68382_20230403203223_lho_pum_dac_noisebudget.pdf

I updated the noisemon projections for PUM DAC noise, and fixed an error in their calibration for the noise budget. They now agree reasonably well with the estimates Craig made by switching coil driver states. From this we can conclude that PUM DAC noise is not close to being a limiting noise in DARM at present.

To Chris' point above -- we note that the PUMs are using 20-bit DACs, and we are NOT using and "DAC Dither" (see aLOGs motivating why we do *not* use them in LHO:68428, and LHO:65807, namely that [in the little testing that we've done] we've seen no improvement, so we decided they weren't worth the extra complexity and maintenance.)

If at some point there’s a need to test DAC dithers again, please look at either (1) noisemon coherence with the DAC request signal, or (2) noisemon spectra with a bandstop in the DAC request to reveal the DAC noise floor.  Without one of those measures, the noisemons are usually not informative, because the DAC noise is buried under the DAC request.

Attached is a revised PUM DAC noisemon projection, with one more calibration fix that increases the noise estimate below 20 Hz (although it remains below DARM).

DARM_IN1(attachment1) saw some sort of glitch(left data tip) ~170ms before DARM registered the lockloss starting(right data tip).

ETMX L2 and L3 also saw this glitch(attachment2). Seems like the glitch hit DARM_IN1, then ETMX L3, then ETMX L2?

I noticed something similar with the 10/15 08:53UTC LL(attachment3), but in that case it looked like the movement was first seen in ETMX L3, then DARM_IN1 and ETMX L2. In that instance the glitch also happened a full 0.5s before the lockloss.

03:25 Back Observing

Very interesting find Oli! I've attached some more plots to hopefully help us narrow down where this is coming from. The L2 ETMX fastimon and noisemon channels see this same glitch, but the L2 OSEM sensor inputs on all of the quads don't. This makes me think that it could possible be coming from the OMC DCPD signals, the error signals forLSC-DARM. Tough to say based on data rates, but the second attachement also maybe shows the DCPD signal moving first, but take that with a grain of salt.

I made the second step down from 1.1 to 1.0 at 22:41UTC for H1:TCS-ETM{X,Y}_RH_SET{UPPER,LOWER}POWER.

Unsure if this made any difference to the range. Range looks improved after the step down but the wind also dropped during this change, ndscopes attached. No large change seen in DARM.

I checked H1:OMC-DCPD_SUM_OUT_DQ and don't see the comb in any of the three listed intervals (neither state A nor B). Tested with a couple of SFT lengths (900s and 1800s) in each case.

Since it seems that the coupling is NOT a direct electronics coupling from Beckhoff -> OM2 -> DCPD, we fully connected the Beckhoff cable to the OM2 heater driver chassis and locked the OMC to the shoulder with an X single bounce beam (~20mA DCPD_SUM, not 40mA like in the usual nominal low noise state). That way, if the Beckhoff is somehow coupling to OMC PZT that might cause visible combs in the DCPD.

We didn't see the comb in this configuration. See the 1st attachment, red is the shoulder lock and green is when 1.66Hz comb was visible with the full IFO (the same time reported by Ansel in alog 73000), showing just two largest peaks of 1.66Hz harmonics visible in the green trace. (It seems that the 277.41Hz and 279.07 Hz peak are 167th and 168th harmonics of 1.66Hz.) Anyway, because of the higher noise floor, even if the combs are there we couldn't have seen these peaks. We've had a different comb spacing since then (alog 73028) but anyway I don't see anything at around 280Hz. FYI I used 2048 FFTs for both, red is a single FFT and the green is an average of 6. This is w/o any normalization (like RIN).

In the top panel of 2nd attachment, red is the RIN of OMC-DCPD_SUM_OUT_DQ of the shoulder lock, blue and dark green are RIN of 2nd loop in- and out-of-loop sensor array. Magenta, cyan and blue green are the same set of signals when H1 was in observing last night. Bottom panel shows coherence between DCPD_SUM during the shoulder lock and ISS sensors as well as IMC_F, which just means that there's no coherence except for high kHz.

If you look at Georgia's length noise spectrum from 2019 (alog 47286), you'll see that it's not totally dissimilar to our 2nd plot top panel even though Georgia's measurement used dither lock data. Daniel points out that a low-Q peak at around 1000Hz is a mechanical resonance of OMC structure causing the real length noise.

Configurations: H1:IMC-PWR_IN~25.2W. ISS 2nd loop is on. Single bounce X beam. DCPD_SUM peaked at about 38mW when the length offset was scanned, and the lock point was set to the middle (i.e. 19mA). DC pointing loops using AS WFS DC (DC3 and DC4) were on. OMC QPD loops were not ON (it was enabled at first but was disabled by the guardian at some point before we started the measurement). We were in this state from Oct/17/2023 18:12:00 - 19:17:20 UTC.

BTW Beckhoff cable is still fully connected to the OM2 heater driver chassis. This is the first observation data with such configuration after Fil worked on the grounding of Beckhoff chassis (alog 73233).

Detchar, please find the comb in the obs mode data starting Oct/17/2023 22:33:40 UTC.

The comb indeed re-appeared after 22:33 UTC on 10/17. I've attached one of the Fscan daily spectrograms (1st figure); you can see it appear in the upper right corner, around 280 Hz as usual at the start of the lock stretch.

Two other notes:

• The comb is now back to its original spacing of 1.6611 Hz.
• There are new strong lines visible at +/- 1.235 Hz from the comb teeth. If this structure has appeared before, I'm not aware of it. I've attached an image showing the lines (2nd figure, average of DMT-ANALYSIS_READY data between 22:00 10/17 and 02:00 10/18) and a comparison with older data from Sept 20th (3rd figure).

Just to see if anything changes, I used the switchable breakout board at the back of the OM2 heater driver chassis to break the thermistor connections but kept the heater driver input coming from the Beckhoff. The only two pins that are conducting are pins 6 and 19.

That happened at around Oct/18/2023 20:18:00 to 20:19-something UTC when others were doing the commissioning measurements.

Detchar, please look at the data once the commissioning activities are over for today.

Because there was an elevated noise floor in the data from Oct/17/2023 18:12:00 mentioned in Keita's previous comment, there was some doubt as to whether the comb would have been visible even if it were present. To check this, we did a direct comparison with a slightly later time when the comb was definitely present & visible. The first figure shows an hour of OMC-DCPD_SUM_OUT_DQ data starting at UTC 00:00 on 10/18 (comparison time with visible comb). Blue and yellow points indicate the comb and its +/-1.235 Hz sidebands. The second figure shows the time period of interest starting 18:12 on 10/17, with identical averaging/plotting parameters (1800s SFTs with 50% overlap, no normalization applied so that amplitudes can be compared) and identical frequencies marked. If it were present with equivalent strength, it looks like the comb ought to have been visible in the time period of interest despite the elevated noise floor. So this supports the conclusion that the comb *not* present in the 10/17 18:12 data.

Following up, here's about 4 hours of DELTAL_EXTERNAL after Oct 18 22:00. So this is after Keita left only the heater driver input connected to the Beckhoff on Oct/18/2023 20:18:00. The comb is gone in this configuration.