TJ investigated why the LOCKLOSS_SHUTTER_CHECK guardian was sometimes mistakenly identifying locklosses when there had not been any lockloss while ISC_LOCK was in the DRMI_ON_POP state.

As a reminder, the only purpose of LOCKLOSS_SHUTTER_CHECK is to check that the shutter triggers after locklosses in which we had more than 25kW circulating power in the arms.  The lockloss checking for this guardian is independent of any other guardian.  

The problem TJ found was a consequence of the work described in 31980 .  Since that work when we switch the whitening gain on the transmon QPDs, there is a large spike in the arm transmission channels which the LOCKLOSS_SHUTTER_CHECK guardian recognizes as a lockloss (TJ will attach a plot).  

We edited the ISC_LOCK guardian to hold the output of the TR_A,B_NSUM filter before making the switch, and turn the hold off after the switch is done.  We loaded this when we were knocked out of observe by TCS.  This is a simple change but if operators have any trouble with DRMI_ON_POP tonight you can call TJ or I.  

TITLE: 12/01 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

Much_Of_Morning Main Issues:

1. ETMy 4735Hz Violin Mode Harmonic damping!  Upgraded Filter Banks for ETMy & damped out the mode(!)
2. HAM3 ISI Saturation & Trips.  Hugh noted, addressed, & appear to be better so far.

LOG:

• 16:25(?) HAM3 ISI tripped while in PRMI; this is unusual.
• 16:35-16:55 Down for Guardian Memory Upgrade (WP6366)
• 19:03 Mid-station visit (Karen)
• 21:57 OBSERVING!
  • NOTE:  Have been getting DIAG_MAIN message about the FAST Shutter
  • 22:08 Lockloss, nothing obvious from Seismic or StripTools.  It was NOT HAM3ISI.
  • Ran Lockloss select which brought up mainly ASC Channels, but I did not see anything obvious.
• 22:39 OBSERVING
  • 23:22 Incoming Peruvian EQ, Transitioned EX & EY ISIs to "250_SC_EQ" state
  • 23:25 Lockloss
• 0:15 OBSERVING
  • Jim ran another A2L with his new medm/script interface before going to Observe.

Locking Notes:

After hand-off this morning, held at VIOLIN_MODE_DAMPING.  Kiwamu came in to take a look at H1 & wanted to note a few items he checked.

• ETMy:  SUS L3 outputs saturated when trying LOWNOISE_ESD_ETMY:  The Suspect was:  higher order violin mode harmonic at ~4753Hz.
• For next lockloss attempt, Enabled FM3 for LSC DARM2 & Made it to NLN, then made first damping attempt of 4753 Mode.
• A2L run, but later received "OMC DCPD Saturdations"
  • Jenne:  OMC_Lock:  REMOVE_WHITENING state & then back to READY_TO_HANDOFF
• Lockloss when trying to take out phase of MODE9....I should have ramped gain down, changed phase, & ramped gain back up. 
• HAM3 trip while going up to REDUCE CARM OFFSET
• Spent a few hours damping 4753Hz in DC READOUT (see other's alogs)

USGS Link

• Magnitude of EQ (according to Terramon):  6.3
• Location of EQ (according to Terramon):  43km NE of Huarichancara, Peru
• Starting time of Event:  22:49UTC (started seeing 0.03-0.1 BLRMS increase
• Lock Status:  H1 Lockloss at 23:25UTC

J. Kissel, J. Driggers, C. Gray 

We've successfully damped the 4735.09 [Hz] violin mode harmonic. The key: the +/- 60deg phase filters in the ETMY_L2_DAMP_MODE9 bank, which we thought were moving the damping control signal phase around the unit circle, was actually not adjusting the phase at this 4.7 [kHz] since they were tuned for the 2nd harmonics at 1 [kHz] (thanks to Jenne who dug into foton to check that these filter made sense). This left us with essentially only 0 [deg] (+ gain) or 180 [deg] (- gain) as options, and neither worked well. 

After rearranging some filters between MODE9 and MODE10 filter banks, Jenne was able to create new phase adjustment filters with 30 [deg] increments for 5 [kHz]. +60 [deg] with a positive gain worked well for the first two orders of magnitude, but we eventually needed to nudge by -30 [deg] once other modes around these frequencies began to be resolved, confusing the error signal. Thus, the final settings that we think will work well:
MODE10: 
gain = +0.02   (we were able to use up to +0.1 while we were trying hard)
FM4       ("100dB")        gain(100,"dB")
FM9       ("+30deg5k")     zpk([0],[28.5238+i*4744.41;28.5238-i*4744.41],1,"n")gain(2.07488e-06)
FM10      ("4735")         gain(100,"dB")*butter("BandPass",4,4734.5,4735.5)gain(120,"dB")

I think all the success I claimed yesterday () was merely by turning on the notch filter in the DARM path and waiting.

I've updated the LHO Violin Mode Wiki, and I've also updated the ISC_LOCK guardian code to ensure this continues to get turned on (specifically I edited the gen_VIOLIN_MODE_DAMPING functional state in the ISC_GEN_STATES.py subfunction that creates VIOLIN_MODE_DAMPING_1 and VIOLIN_MODE_DAMPING_2 in the acquisition sequence).

Unclear if related to Cheryl's reported problems with the fundamentals over night (LHO aLOG 32059). Note the Violin Mode Wiki or the ISC_LOCK guardian have been updated with her changes.

See the first attached, showing the ISC lock state (Ch16) dropping as the HAM3 ISI WD (Ch1) trips to State3 (Damping only.)

All the CPSs show a twitch and shift as the DC coupled ISO loops disengage but note the scale of the glitch on H2 and V2 CPSs.  What is curious to me is how the C2 sensors clearly exceed saturation level (20000) but the SAT_COUNT (Ch2) does not count them.  Ahhh, I think that any 'trip' of the watchdog, even if only to damping only, will clear the saturations.  I think we should change this in the code although I think someone coded it explicitly this way (BTL/HRAP.)  I think this hinders us though and maybe we can rework it.

I decided to Power Cycle the CPS and while off reseated the corner2 Gauge Boards in the satellite rack.  Did this several times.  ISI reisolated without difficulty.

Attached as well is another view a few hours ago of this glitching dropping the IFO out of some state better than zero.  Prior to the ISI trip is another glitch on the Corner2 CPS that was not enough to rile things up.

Finally, a 7 hour trend capturing all the recent occurrances of this.  Prior to this, 22 Nov saw a CPS saturation and trip but that was a Tuesday.

All BSC spectra looks pretty normal. Noise below 20 Hz is due to different platforms behaving differently.

HAM spectra looks fine above 20Hz except for some non-consequential bumps between 20 and 40Hz in HAMs 2&5.

In order to go to Observe, JeffK noticed that some of the violin mode gains change lock-to-lock.  We looked in the guardian, and the violin damping mode generator state uses a function called adjustDamperGain() which lowers the damping gain of selected modes if the peak is too high. 

For all of the modes which are currently under this adjuster's control, I have not-monitored the gain channel for that mode.  The attached screenshot is of the guardian code that is adjusting the gain, and serves as a list of all the modes whos gain I've not-monitored. 

If we comment some of these adjusters out, or add more, we'll need to re-visit the list of not-monitored channels.

This morning we continued to see issues with a rung up 4735Hz Violin Mode Harmonic.  Managed to squash it out after a few hours of upgrading filter banks (Jenne), getting schooled on technique (Jenne, Terra, Jeff), and the pesky mode finally went away and made it to NLN.....

....But then HAM3 ISI tripped as we were about ready to join L1 in OBSERVING.

With H1 down, Hugh went to address HAM3.

On our way back up to NLN & hopefully OBSERVING soon!

J. Kissel

Since we've spent so much time over the past few days exploring for the correct damping phases on violin modes, I figured it was time we upgraded our unit-circle-sticky-note system. As such, I've create a one-sheet scratch pad for damping mode phases that one can use to keep oneself sane while searching for just the right phase on up to 4 resonant modes.

The official place for this diagram, in case it ever needs changing is G1602364, but the source also lives in the SUS SVN repo,
/ligo/svncommon/SusSVN/sus/trunk/Common/Documents/G1602364, and I attach it here.

This has been printed out, laminated, and put next to the operator's work station with an appropriate dry erase marker.

Happy hunting!

Don't know the reason yet as I don't see the control loops responding before the Actuators.  See SEI log 1083 for some details.

The first attached plot shows the glitches on H2 and V2 and the Saturations counts totalizing.  The other CPS look fine.

This has not caused us to lose lock everytime.  The second plot is 6 hours showing ISI tripping twice--lots of things happen here so you can't say much from such a quick coarse look.  However, at the two places where the CPS SAT COUNT totals and we don't trip, saturations are really only seen on H2 & V2.

Looking back 30 days suggests to me this might have happened a couple times but the platform tripped so without diving deeper, I can't say for sure.

I'd say this warrants a satellite rack power cycle and a few iterations of board reseating to clean up contacts at the first opportune time.

Report on DetChar DQ glitch shift for Monday Nov 28 - Wednesday Nov 30

• Duty cycle was low for 2 days leading up to O2 (29.2% Monday, 6.4% Tuesday), but recovered for first day of O1 (42.9% Wednesday)

• Inspiral range in the 70 Mpc ballpark, drooping to 60 Mpc occasionally

Looked at 3 major sources of glitches during this shift:

• Loud 20-200 Hz glitches believed to be caused by Optical Lever laser glitching, and seem to be improved by adjusting OpLev laser power

• Strong line of 1080 Hz glitches, reasonable suspicion they're coming from the PSL (often saw definite coincidence with ~270 Hz glitches in IMC-WFS_A_DC_SUM_OUT_DQ as well as what I thought was quite telling, PSL-ISS_PDA_REL_OUT_DQ - way upstream of DARM if I recall correctly), but replacing a piezo mount didn't seem to help and turning off PCal injection line didn't help

• Relatively new lines of glitches at 2 kHz and 3 kHz that were quite evil, but we're pretty sure are caused by rung-up violin modes (and can thus be gotten rid of by spending a little time damping).

Full report can be found at https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20161128

WP6366 Increase memory in Guardian machine

Dave, TJ, Carlos:

at 08:47PDT h1guardian0 was powered back up after having its RAM size increased from 12GB to 48GB. TJ reports all nodes are operational.

J. Kissel, E. Merilh, J. Warner, T. Hardwick, J. Driggers, S. Dwyer

Prompted by DetChar worries about glitching around the harmonics of violin modes, Ed, Jim, and I went on an epic campaign to damp the ~1kHz, 2nd harmonic violin modes. These are tricky because not all modes had been successfully damped before, and one has to alternate filters in two filter banks to hit all 8 modes for a given suspension. 

We've either used, or updated Nutsinee's violin mode table, with the notable newly damped entries being 
994.8973    ITMY     -60deg, +gain      MODE9: FM2 (-60deg), FM4 (100dB), FM9 (994.87) 
997.7169    ITMY       0deg, -400gain   MODE9: FM4 (100dB), FM6(997.717)                 VERY Slow
997.8868    ITMY       0deg, -200gain   MODE10: FM4 (100dB), FM6(997.89) 

Also, we inadvertently rung up modes around 4735 Hz, so we spent a LONG time trying to fight that. We eventually won by temporarily turning on the 4735Hz notch in FM3 of the LSC-DARM2 filter bank and waiting a few hours. I had successfully damped the ETMY mode at 4735.09 Hz by moving the band-pass filter in H1:SUS-ETMY_L2_DAMP_MODE9 's FM10 from centered around 4735.5 to centered around 4735 Hz exactly, and using positive gain with zero phase. However, there still remains a mode rung up at 4735.4 Hz but it's from an as-of-yet unidentified test mass, and we didn't want to spend the time exploring. These 4.7 kHz lines have only appeared once before in late October (LHO aLOG 31020).

Attched is a before vs. after ASD of DELTAL_EXTERNAL. I question the calibration, but what's important is the difference between the two traces. Pretty much all modes in this frequency band have been reduced by 2 or 3 orders of magnitude -- better than O1 levels. Hopefully these stick through the next few lock losses and acquisitions.

Thanks again to the above mentioned authors for all their help!

The free memory size on the guardian machine is about 4GB. At the current rate of usage we predict a reboot is needed before next Tuesday. At the next opportune time, we will increase the memory size from 12GB to 48GB and perhaps schedule regular reboots on Tuesdays. 

Plot of available memory for the month of November is attached (Y-axis mis-labelled, actually MB).

I hit load and cleared the error. But I have no idea about the state of the injection since it didn't happen the first time. Looks like it will try again at GPS 1163938117?

J. Kissel for S. Karki

I've moved the roaming calibration line to its highest frequency we intend to go, and it's also the last super-long duration we need. We may run through the lower frequency points again, given that (a) they need much less data, and (b) those data points were taken at various input powers that will likely confuse/complicate the analysis. Below is the current schedule status.

Current Schedule Status:
Frequency    Planned Amplitude        Planned Duration      Actual Amplitude    Start Time                 Stop Time                    Achieved Duration
(Hz)         (ct)                     (hh:mm)                   (ct)               (UTC)                    (UTC)                         (hh:mm)
---------------------------------------------------------------------------------------------------------------------------------------------------------
1001.3       35k                      02:00                   39322.0           Nov 11 2016 21:37:50 UTC    Nov 12 2016 03:28:21 UTC      ~several hours @ 25 W
1501.3       35k                      02:00                   39322.0           Oct 24 2016 15:26:57 UTC    Oct 31 2016 15:44:29 UTC      ~week @ 25 W
2001.3       35k                      02:00                   39322.0           Oct 17 2016 21:22:03 UTC    Oct 24 2016 15:26:57 UTC      several days (at both 50W and 25 W)
2501.3       35k                      05:00                   39322.0           Oct 12 2016 03:20:41 UTC    Oct 17 2016 21:22:03 UTC      days     @ 50 W
3001.3       35k                      05:00                   39322.0           Oct 06 2016 18:39:26 UTC    Oct 12 2016 03:20:41 UTC      days     @ 50 W
3501.3       35k                      05:00                   39322.0           Jul 06 2016 18:56:13 UTC    Oct 06 2016 18:39:26 UTC      months   @ 50 W
4001.3       40k                      10:00                   39322.0           Nov 12 2016 03:28:21 UTC    Nov 16 2016 22:17:29 UTC      days     @ 30 W (see LHO aLOG 31546 for caveats)
4301.3       40k                      10:00                   39322.0           Nov 16 2016 22:17:29 UTC    Nov 18 2016 17:08:49 UTC      days     @ 30 W          
4501.3       40k                      10:00                   39322.0           Nov 18 2016 17:08:49 UTC    Nov 20 2016 16:54:32 UTC      days     @ 30 W (see LHO aLOG 31610 for caveats)   
4801.3       40k                      10:00                   39222.0           Nov 20 2016 16:54:32 UTC    Nov 22 2016 23:56:06 UTC      days     @ 30 W
5001.3       40k                      10:00                   39222.0           Nov 22 2016 23:56:06 UTC