Josh, Andy, David, Jess

Conclusion: We suspect that the 50Hz glitches in DARM are caused by EX Mains glitches that happen 400+/-70ms earlier (reliably matched) and may(?) couple through EX Seismic. 

Throughout September there has been a persitent line of glitches at 50Hz in DARM with a pretty high rate. See the attached glitch-gram from today's summary pages. Based on the one hour plots on the summary pages, the rate of these 50Hz glitches in DARM is about once per minute. 

We found (see e.g. the first round here) that these DARM glitches are correlated in time with glitches in EX channels, particularly PEM EX SEIS/ACC*, ISI ETMX ST1 BLND*, and ASC-X_TR*. 

It looks like the cause of this may be a big glitch in EX Mains. Wew see the EX and DARM channels are all glitching with a 400+/-70ms delay (averaged five examples followed up by hand) after EX mainsmon. Attached are two examples where we lined up the glitch in EX mainsmon and showed that the glitches in EX seismic and DARM are 400ms later. 

Notes:

• The EX Mains glitches are broadband, but EX Seis, ISI, and DARM are all only localized glitches at 50Hz.
• We checked and the EX microphones look clean.
• The glitches seen by EX mainsmon are also seen (sometimes) very weakly by the LVEA mainsmons (at the same time, i.e. earlier than in other EX channels and DARM).

We would be happy to follow up further leads, but wanted to report what we found so far. 

PS. Should you wish to repeat this yourself, attached is a list of these glitches in DARM today (with nice GPS times) and a screenshot of the ldvw settings I used. 

Terramon told us that we had an incoming earthquake large enough to likely break lock, so I asked Jim to take us out of Observing mode so that I could quickly try to increase the SRM offloading, to see if that would save the lock.  In the end, we still lost the lock.

I changed the SRM M1 Lock L gain from the nominal -0.02 to -0.05 with a 10 second ramp.

I need to look at the data to know more, but the verbal alarm did not complain about SRM saturations.  It did however complain about ETMY saturations. 

If that really did help prevent SRM saturations from causing our lockloss, I wonder if we should think about increasing every optics' offloading gain by a factor of 2 or 3 when we have an earthquake coming.  Anyhow, I need to look at the data to see what really happened, and to see if I can figure out what really helped.

IM4 QPD, lower 2 quadrants were saturating - lowered their whitening gain by 9dB from 36dB to 27dB (gain value from 12 to 9) IM4 is not in any loop, so no impact on feed-back.


Before clicking undisturbed, we updated some ODC settings:
- default ODC filter mask were set for PRM, SRM, SR3, ETMX and ETMYY
- took out LSC and ASC party check from MASTER mask (the party from those models is not reported correctly, see alog 20936)
- all changes are in SDF

- additionally updated the ASC ODC transmon alarm thresholds form 0.2 (Xarm) and 0.3 (Yarm) to to 0.7 at 20:02 UTC - the intent bit was unset for 2 min,. but the data is unaffected.
Note that H1:ASC-Y_TR_B_YAW_OUTPUT had drifted all the way up to -0.55 - presumably due to TMS drift.
We are using -1.1*TRY_A_YAW + 0.5*TRY_B_YAW for servoing, because this independent of TMS drift. Of course, if we move enough that we fall off one diode, that's no longer true. We should keep an eye on that.
- all changes are in SDF

Things were quiet. Then Stefan arrived, and I realized I had set the observatory mode but forgot the Intent Bit, and which spiraled into a bunch SDF differences. The LSC guardian showed a difference on a t-ramp (annoying that this kind of difference would keep us from going into Observe) and ASC had two diffs that I guess Evan logged about last night but didn't accept. Then Stefan went and set a bunch of ODC masks (this also would have kept us from going into observe, so also annoying) as well as tuning the whitening the IM4 QPD (which was saturating, Stefan assures me is out of loop so shouldn't affect anything). Guardian recovered from the earlier lock loss in 20 minutes (not annoying!) back to 70 mpc and it has been otherwise quiet.

Given PeterF's recent calculations of maximum HW injection amplitude vs frequency,
it appears that the current injection amplitude for the 1991-Hz pulsar (#14) is too high and the
amplitude for the 1403-Hz pulsar (#4) is high enough to be worrisome, once we have in place an updated
inverse actuation filter for hardware injections (assuming we continue using the current injection
point, for now -- see Jeff's entry).

So I have reduced the #14 amplitude by a factor of 10 and the #4 amplitude by a factor of 2 in
the config files used by the injection machine. The next time pulsar injections are restarted,
the new amplitudes should become active. Note that I've redefined the symbolic link RELEASE
to point to a new pulsar subdirectory ER8B with the revised amplitudes.

For reference, the new maximum pulsar injection amplitudes (+ and x source polarization values,
which are summed together with sidereal-time-dependent antenna-pattern functions) are plotted in
the attached figure, along with eyeballed curves of Peter's max allowed ETM displacement amplitudes
(ESD and PCAL actuation) for frequencies higher than 100 Hz. 

The new pulsar injection parameters are given in full here.

ER8 Day 26. No unexpected restarts. tw0 recovered from raid problem.

model restarts logged for Fri 11/Sep/2015
2015_09_11 08:45 h1tw0
2015_09_11 09:16 h1tw0

ER8 Day 25. no unexpected restarts, working on recovering tw0.

model restarts logged for Thu 10/Sep/2015
2015_09_10 11:51 h1tw0
2015_09_10 12:35 h1tw0
2015_09_10 12:40 h1tw0
2015_09_10 12:45 h1tw0
2015_09_10 12:49 h1tw0

IFO locked all shift, 4 ETMY glitches, nothing else to report.

A few thought as a follow on to Jeff's entry:

1. Injecting signals at DARM control (see G1501146) seems like a bad idea
   • while this worked in iLIGO, which had a relatively simple actuation TF, the DARM ctrl to DARM displacement TF is much more complicated now that we have a multi-stage suspension
   • the inverse TF looks like 1 / A * (B + C + D), which, by virtue of containing a sum in the denominator, can be very messy
   • not only is it messy, it is not robust against small changes in the DARM actuation path.  Think 1 / A * (g * B + C + D) and then let g vary... this will cause the poles and zeros to move.
   • Injecting directly into an actuator with a simple TF to DARM displacement (or force) would be much better (e.g., at the ESD or with PCAL).
2. I'll see what I can do to make an inverse filter, but I will be happy if a better way is found and this filter is never used.

Since turning up the bandwidth of the AS_C → SR2 loops, we had not engaged cutoff filters. There are now 18 Hz elliptic LPFs installed for both pitch and yaw. They are turned on the ISC LOCK guardian just after the final gain increase of the yaw loop.

TITLE:  9/11 EVE Shift:  23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC

SUPPORT:  Evan helped me out with a question.  Others were here working throughout the night.

SHIFT SUMMARY:  

H1 at LOW_NOMINAL_NOISE for most of the shift (~43min of locking after a lockloss), and was in Commissioning for much of that time.  Range hovered around 75-80Mpc.  A couple of Mexico aftershocks barely registered here.  Other than that a fairly quiet night.  

OPERATOR OPEN ITEMS:  

• DIAG_MAIN has two BRS messages ("reboot needed" & rung up).  TJ will head out to EX to address this on Maintenance.  
• O1 CDS Overview has (1) RED H1CALCS (this is ok, it's injection-related), & (2) WHITE fw2 (Dave plans to remove fw2 from this medm).

INCOMING OPERATOR:  Cheryl

SHIFT ACTIVITIES:

• 0:46 Dick out of Optics Lab
• 3:22 Lockloss.  
• 4:06 H1 back up to NOMINAL_LOW_NOISE.
• 4:07 Evan going to start Noise Budget Injections until L1 is back in Observing
• 6:08 Double Coincidence begins:  H1 Back to Observation Mode and joining L1.

J. Kissel, D. Tuyenbayev

Although the jury is still out as to what accuracy we need for the start of O1, I've pushed forward with the plan to design an inverse actuation filter that removes the notches from the test mass stage of actuation, as discussed yesterday (LHO aLOG 21379). In doing so, I've plotted the total DARM actuation function for the first time since (a) we've an updated knowledge each stages' the actuation strength (LHO aLOG 21280), and (b) since the hierarchy filters were changed to reduce the impact of the wide-band impact of PUM violin mode resonances (LHO aLOG 20143). 

Regrettably, just like in ER7 (see pg 7 of G1500750), I've discovered that the PUM/L2 interaction with the TST/L3 stage is still very ugly, if not worse than it was. This means that even after taking out the notches in the TST/L3 stage and softening the Q of the TST/L3, 950 [Hz] L3 elliptic filter, there remains very complicated, 30%, 15 [deg] wiggles in the 100 [Hz] to 1000 [Hz] band that would require a great deal of poles and zeros to get correct.

I've found poles and zeros that could be used for a first cut filter, but it really does not do the full actuation function any justice, and is not worth installing. Matt Evans has suggested that he will give it a go tomorrow, but I really think this accelerates the need to inject hardware injections in a different place (the two options being PCAL and further upstream in the SUS actuation chain -- both of which have no notches or elliptic filters or anything complicated to invert). Either that, or we do an honest job cleaning up and rolling off the PUM with a sensible design, which means a significant change to the calibration and impact of the robustness of the IFO operation. Yes, it's just updating digital filters in the front end, but it would be a day or so of model parameter set development, coupled with verification against an open loop gain measurement, and testing whether the IFO was still stable. Kiwamu, Peter, Rick, and I are leaning towards the PCAL option, but we don't yet know what impact this will have on the hardware injection software infrastructure. 

As usual, Stay tuned. 
#NoSleepTilO1

The full story
--------------

Each page of the first pdf attachment walks you through the story of how I've attempted to go about this.

Just like before the H1 Mini-run and ER7, I've gone into the creation of this filter assuming that below the ~30 [Hz] PUM / TST cross-over frequency the PUM is the PUM stage, and above the TST is the TST stage. As such, I think "I should be able to create a simple set of poles and zeroes that roughly rise the shape of f^2, that I can roll off at high frequency, that may have a few poles and zeroes worth of complication. Certaintly something I can fit into one filter module of 10 second-order-sections. This way I don't have to deal with complicated, often buggy, fitting functions like vectfit or liso, nor with matlab-to-foton filter loading functions that are prone to errors / mistakes like autoquack. Plus I'll truely understand what I'm installing, and I can keep track of the systematic error all along the way." 

As such, I started the project as planned: take out the notches from the L3 stage, and reduce the Q of the elliptic filter. 
Pg 1 shows the progression as I reduce the complexity of the TST / L3 stage actuation authority. Blue is the full L3 actuation function, as taken directly from the canonical DARM model LHO aLOG 21386. Green is that same model, with the notches removed. Red is the same model, notches removed, and with the Q of the elliptic reduced from essentially infinity to 30.

Pg 2 shows the detail of the tailoring of the elliptic filter.

Great! This'll be nice and easy to invert.

So, I shoved the newly modified L3 / TST stage into the entire actuation function -- and I see a whole bunch of wiggles between 100 and 1000 [Hz] that are just not a part of the L3 / TST actuation (as seen on pg 1).

After quintuple checking for bugs, I plotted the decomposition of the total actuation function into all stages. This is pg 4. One can immediately see from this that there are interactions between the PUM and TST stage well-above the PUM / TST cross-over frequency. YUCK.

After digging around a little bit, I take a look at the filters that have been installed when the control authority was "reduced" in LHO aLOG 20143. I see that not only was the PUM *not* rolled off better, but the notches and elliptic filters that were put in place *instead* of just softly rolling off the stage better are just rediculous. Bode plots of the PUM notch filters and elliptic filters are shown in the second attachment for proof. 

Finally, just so I could demonstrate how a simple, 10-pole, 10-zero fit just would in no way be able to reproduce this actuation function, I put together such a function and caluculated the residuals. Pg 5 and 6 (back to the first attachment) show these residuals.

----------
The analysis script that generated these plots and the filter are here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/InvActuation/generate_invactuation_filter_ER8.m

The model and parameter file to generate the full actuation function live here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/
H1DARMOLGTFmodel_ER8.m
H1DARMparams_1125963332.m

The filter file that contains the digital filters in the SUS chain lives here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Common/H1CalFilterArchive/h1susetmy/H1SUSETMY_1124818267.txt

I attach the filter file and screenshots of all relavant filter banks here, for Matt's convenience, if he wants to waste a perfectly good Saturday.

9/11 EVE Shift:  23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC

3:22 H1 lockloss.  "ITMY Saturation" on VerbalAlarm.  Terramon said a 4.5 Japan quake within a minute of this, but we don't see any seismic evidence.

• Lock Attempt#1:  Slight tweaks on ETMx for ALS & then H1 went on its way.  DRMI took about 6min.  Dropped out at RF DARM.
• Attempt#2:  During this attempt, I noticed I had accidentally tweaked TMSx instead of the ETMx(!) in Attempt#1.  So with Evan's suggestion, I paused at DC_READOUT, and then carefully tweaked the TMSx back (accidentally took it to 40.085, and returned it to its nominal 41.485).  I returned TMSx with no issues.
• 4:06 H1 back up to NOMINAL_LOW_NOISE.
• 4:07 Evan going to start Noise Budget Injections (so this is why the range will look wonky from 4:15 & on....

I also talked with LLO (William on shift), and he said they were fighting roll modes.  When they get it back, Anamaria/Robert would like to do PEM Injections (they said it was late, so they might only wait it out a little longer).  So Evan will be opportunistic (WP5467) with H1 in the meantime.