For those of us who haven't been on DetChar calls to have heard this latest DetChar nickname... "Scratchy glitches?"

Hi Jeff, 

Scotty's comment above refers to Andy's comment to the range drop alog 30797 (see attachment here and compare to Andy's spectrogram). We're trying to help figure out its cause. It's a good lead that they seem to be related to RM1 and RM2 motion. 

"Scratchy" is the name used in GravitySpy for these glitches. They are called that because they sound like scratches in audio https://wiki.ligo.org/DetChar/InstrumentSounds . In FFT they look like mountains, or if you look closer, like series of wavy lines. They were one of the most numerous types of H1 glitches in O1. In DetChar we also once called them "Blue mountains." Confusing, I know. But there is a DCC entry disambiguating (in this case equating) scratchy and blue mountain https://dcc.ligo.org/LIGO-G1601301 and a further entry listing all of the major glitch types https://dcc.ligo.org/G1500642 and the notes on the GravitySpy page. 

If FAMIS were allowed to digest this activity, it could expect to become more "regular" (I'm laughing at my own jokes!)

note that to permt sus-aux channels to be acquired at 4kHz, the models h1susauxh2, h1susauxh34 and h1susauxh56 will be modified from 2K models to 4K models as part of tomorrow's work.

Happened again but for a different channel H1:SUS-ITMX_L2_DAMP_MODE2_RMSLP_LOG10_OUTMON ( Sheila's post was for H1:LSC-PD_DOF_MTRX_7_4). I trended and found data for both of those channels at the connection error times, and during the second error I could also caget the channel while ISC_LOCK still could not connect. I'll keep trying to dig and see what I find.

Relevant ISC_LOCK log:

2016-10-25_00:25:57.034950Z ISC_LOCK [COIL_DRIVERS.enter]
2016-10-25_00:26:09.444680Z Traceback (most recent call last):
2016-10-25_00:26:09.444730Z   File "_ctypes/callbacks.c", line 314, in 'calling callback function'
2016-10-25_00:26:12.128960Z ISC_LOCK [COIL_DRIVERS.main] USERMSG 0: EZCA CONNECTION ERROR: Could not connect to channel (timeout=2s): H1:SUS-ITMX_L2_DAMP_MODE2_RMSLP_LOG10_OUTMON
2016-10-25_00:26:12.129190Z   File "/ligo/apps/linux-x86_64/epics-3.14.12.2_long-ubuntu12/pyext/pyepics/lib/python2.6/site-packages/epics/ca.py", line 465, in _onConnectionEvent
2016-10-25_00:26:12.131850Z     if int(ichid) == int(args.chid):
2016-10-25_00:26:12.132700Z TypeError: int() argument must be a string or a number, not 'NoneType'
2016-10-25_00:26:12.162700Z ISC_LOCK EZCA CONNECTION ERROR. attempting to reestablish...
2016-10-25_00:26:12.175240Z ISC_LOCK CERROR: State method raised an EzcaConnectionError exception.
2016-10-25_00:26:12.175450Z ISC_LOCK CERROR: Current state method will be rerun until the connection error clears.
2016-10-25_00:26:12.175630Z ISC_LOCK CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.

It happened again just now. 

Opened FRS on this, marked a high priority fault.

ticket 6558

We were able to increase power to 25W okay, but turning off the AC coupling made things go crazy and we lost lock.  The diffracted power went up, and we lost lock around the time it hit 10%. 

The 2nd loop output offset observed by the 1st loop was about 30mV (attached, CH8). With the 2nd ISS gain slider set at 13dB and a fixed gain stage of 30, this correspond to 0.2mV offset in the AC coupling point. This offset is relatively small.

One thing that has happened in the past two weeks or so is that the power the 1st loop sensor (PDA) receives was cut by about half (second attachment). This was caused by the move from the old position to the new position of the PD.

Since the sensing gain of the 1st loop was reduced by a factor of two, seen from the 2nd loop the 1st loop is twice as efficient an actuator. Apparently the second loop gain slider was not changed (the slider is still at 13dB), so even if the same offset was there before, the effect was a factor of two smaller before.

Another thing which is kind of far-fetched is that I switched off the DBB crate completely and we know that opening/closing the DBB and frontend/200W shutters caused some offset change in the 2nd loop board.

Note to self: check the front-end calculations of the uncertainty and coherence of these lines before and after this change after the IFO reverted back to 25 [W] input power. 
Example checks: 
   - do the calculations show the expected decrease in coherence / increase in uncertainty? 
   - how much was the uncertainty / coherence when the SNR was so high? 
   - do we like that level of uncertainty? did it reveal more real optical parameter changes instead of noise?

Delayed update, these changes were accepted in the SDF today (Oct. 26, 2016, ~10:20 PDT).

Thanks-

This shows that we have ASC noise below 30 Hz, and that perhaps the A2L for ITMY was not tuned well at the end of the lock: ITMY_L3_ISCINF_P

SRCL noise is high from 15-50 Hz, we will attempt to make a better feedforward filter soon for this (This is also the conculsion of some quick noise injections this morning.)

PRCL coherence is high both where the SRCL coherence is high and at the jitter peaks, which could be coupling through SRCL or frequency noise lock point errors.

PSL channels that have good whitening show coherence around our high frequency lump.  (OSC PD INT ISS PDB)

There are a few channels that I think could be added to the excluded list:

SUS-ETMY_L2_FASTIMON_LL_OUT_DQ

OMC-PI_DCPD_64KHZ_AHF_DQ

OMC-DCPD_NULL_OUT_DQ

Looking at the frequency range of interest (mainly 50 - 200 Hz), there isn't any coherence with significant channels. This is not unexpected if the noise is due to scattering, since it would be a highly non linear coupling, thus not seen with a coherence analysis.

From you request, I ran Bruco on those times. The results are as follows,

bad time (12:20 UTC) : https://ldas-jobs.ligo-wa.caltech.edu/~youngmin/BruCo/PRE-ER10/H1/Oct24/H1-1161346817-600/

good reference(11:08 UTC): https://ldas-jobs.ligo-wa.caltech.edu/~youngmin/BruCo/PRE-ER10/H1/Oct24/H1-1161342497-600/

These could give you a hint for range drop.

Here is a plot of thge auxiliary loops, again comparing good vs bad.

Note the two broad noise humps around 11.8Hz and 23.6Hz. They both increased at the bad time compared to the good time.

Interesingly, the peaks showing up in the DARM spectrum are the 4th 5th, etc. to 12th harmonic of that 11.8-ish Hz.

It all smells to me like some form of scatter in the input chain.

The IMs do not change their motion between Stefan's good time (11:08 UTC) and bad time (12:20 UTC).  But, the RMs, particularly RM2, see elevated motion, almost a factor of 2 more motion between 8Hz - 15Hz.

First screenshot is IMs, second is RMs.  In both, the references are the good 11:08 time, and the currents are the bad 12:20 time.

Stefan and TeamSEI are looking at the HEPI and ISI motion in the input HAMs right now.

EDIT:  As one would expect, the REFL diodes see an increase in jitter at these same frequencies, predominantly in pitch.  See 3rd attachment.

I quickly grabbed a time during O1 when this type of noise was happening, and it also corresponds to elevated motion around 6 Hz in RM1 and RM2. Attached are a spectrogram of DARM, and the pitch and yaw of RM2 at the time compared to a reference.

There is a vertical mode of the RMs at 6.1 Hz (that's the LLO value, couldn't find it for LHO). Maybe those are bouncing more, and twice that is what's showing up in PRCL?

There should not be any ~6 Hz mode from the RM suspensions (HSTS or HLTS), so I am puzzled what this is. 
For a list of expected resonant frequencies for HSTS and HLTS see links from this page

https://awiki.ligo-wa.caltech.edu/aLIGO/Resonances

@Norna: the RMs, for "REFL Mirrors" are HAM Tip-Tilt Suspensions, or HTTS (see, e.g. G1200071). These, indeed, have been modeled to have their highest (and only) vertical mode at 6.1 Hz (see HTTS Model on the aWiki).

I can confirm there is no data committed to the SUS repo on the measured vertical mode frequencies of these not-officially-SUS-group suspensions at H1. Apologies! Remember, these suspensions don't have transverse / vertical / roll sensors or actuators, so one have to rely on dirt coupling showing up in the ASDs of the longitudinal / pitch / yaw sensors. 

We'll grab some free-swinging ASDs during tomorrow's maintenance period.

Stefan has had Hugh and I looking SEI coupling to PRCL over this period, and so far I haven't found anything, but HAM1 HEPI is coherent with the RM damp channels and RM2 shows some coherence to CAL_DELTAL, around 10hz.  Attached plot shows coiherence from RM2_P to HEPI Z L4Cs (blue), RM2_P to CAL_PRCL (brown), and RM2_P to CAL_DELTAL (pink). The HAM1_Z to PRCL is similar to the RM2_P to CAL_PRCL, so I didn't include it. HAM1 X and RY showed less coherence, and X was at lower frequency. There are some things we can do to improve the HAM1 motion if it's deemed necessary, like increasing the gain on the Z isolation loops, but there's not a lot of extra margin there.

Here are ASDs of the HAM3 HEPI L4Cs (~in-line dofs: RY RZ & X) and the CAL-CS_PRCL_DQ.  The HAM2 and HAM1 HEPI channels would be assessed the same way:  The increase in motion seen on the HAM HEPIs is much broader than that seen on the PRC signal.  Also, none of these inertial sensor channels see any broadband coherence with the PRC, example also attached.

Freee swing PSD of RMs and OM are in alog 30852.