Locked at NLN for about 5 hours. Violin modes continue ringing down. DARM noise remains elevated and range is holding around 35Mpc. Turning over to the day shift. 

   Relocked at NLN. ETMX violin modes are still rung up, but slowly coming down. The DARM spectrum is elevated above 80Hz and slowly improving. A2Ls are both below the reference. Damped PI Modes 26 and 27 by a Phase sign flip. Continuing to limp along. Working on damping as limited capabilities allow. 

NOTE: Due to apparent profile problems on laptop cannot open either Firefox or Thunderbird. This is limiting capabilities somewhat.   

J. Kissel, J. Bartlett

For funzies, on my way out, I taught Jeff how to turn on and off DCPD whitening so I could check whether we had done enough damping today to get us back into the nominal configuration of 1 stage of DCPD whitening. 
Nope. 
We heard verbal indicate that we were saturating the DCPDs with one stage of whitening. 
We lost lock while trying to remove the stage of whitening we added in.

So -- back at it tomorrow!

J. Kissel, C. Vorvick

We spent the remainder of the shift trying to damp violin modes as best we could with the IFO in DC readout, but with no stages of DCPD whitening. We were able to reduce the RMS counts of ADC input on the DCPDs by about a factor of 5, by performing all the already known violin mode damping tricks for the fundamentals (~500 Hz) and 2nd harmonics (1000 Hz), as per Nutsinee's glorious work on the sacred LHO violin mode table. However, this earthquake has rung up *every* harmonic.

So, I begun the task of developing filters to damp the previously-undamped 3rd harmonics. I took a few hours of play, but I was able to known down the 6 highest RMS contributors, but there's still plenty more to go. I've added new rows to the violin mode table that detail the configuration and filters I created, but I summarize here:

EX      1478.17      MODE 1     FM 4 6 7, G = -1000
EX      1475.25      MODE 2     FM 4 6, G = -1000
EX      1474.08      MODE 3     FM 4 6 8, G = -2000
EX      1471.93      MODE 5     FM 4 6 7, G = -2000
EY      1484.53      MODE 2     FM 4 6 8, G = +2000
IX      1467.96      MODE 1     FM 4 6 7, G = -2000

These details may change over time as we consolidate filter banks, so always treat the violin mode table as cannon, not this aLOG.
Note that all of these settings are for a 30W, DC readout IFO, with PUM coil drivers set to State 3, and driving in Pitch.

The filter design process at 1.5k has become pretty straight-forward:
- Have a "live" (3 avgs, exponential) DTT spectra of H1:OMC-DCPD_A_IN1 and H1:OMC-DCPD_B_IN1 going with RMSs of each and a reference plotted for all (like I have attached). 0.005 Hz BW, so you can nail down which mode you're attacking. (Pick the one that's contributing the most to the RMS in the 1.5 kHz band). 
- Also have a live DTT spectra of H1:DARM_IN1_DQ going, at faster pace (I chose 0.02 Hz BW) for use while tuning the gain and phase.
- Copy the 1.5k set to a new empty filter bank (thanks for that now-essential feature Jim Batch!)
- Adjust the frequency of the bandpass to be centered around the mode you're trying to attack. I've been successful with ~0.3-0.5 Hz bandwidths. Note, once I have a bandwidth I like (where my figure of merit is "how fast is the phase wrapping?" I shoot for less that 3-4 phase wraps in the bandpass frequencies), I shift the bandpass up or down such that the phase at the exact frequency of the mode is 0 +/- 5-10 [deg].
- Adjust the "-60 deg" filter by highlighting the zpk portion, hitting the ZPK button, switching to Mag/Phase, then adjusting the frequency of the complex pole (i.e. add new, and remove old) by 10-50 Hz until I get -60+/-5 [deg] at the mode frequency. I make sure to then normalize the filter to 1 at the mode frequency when I get the right poles.
- Adjust the "+60 deg" filter in the same way, just shooting for +60+/-5[deg]. Remember to normalize to 1 a the the desired mode frequency.
- Save the foton file, load it into the bank.
- Turn on the bandpass, and set a gain, watching the output of the filter bank on a StripTool. 
- Start with gains of 10-100, but you won't see action until about a gain of 1000.
- Change the sign and phase as normal to find which combination creates damping.

Design at 2k and 2.5k should be pretty darn similar, once you make a starter band-pass and +/-60 deg phase filters.

The attached shows the result of our work at 1k and 1.5k this evening.

I left all of the damping filters I've designed on and running. These should get blown away if we lose lock, so now harm done. Plus there's always the SDF system.

TITLE: 07/07 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: locked - damping violins
LOG:

• JeffK to EX to reset HEPI pump, and here all shift to work on damping violins
• Bubba on site at LSB 2:45-3:30UTC
• H1 currently locked in Violin Mode Damping 2
• I've increased the bounce mode damping gains to 0.5 (0.1 nominal) and the roll mode damping gains to 30 (20 nominal)

J. Kissel, C. Vorvick, H. Radkins (remotely)

While struggling to damping 2nd harmonic violin modes having just transitioned to DC readout for the first time after yesterday's Montana earthquake, the X-END seismic system suddenly tripped all three stages of its watchdog (ISI ETMX ST1 ST2, and HEPI). Totally unrelated random other problem to solve.

The ISI and HEPI overview screen quickly revealed that the HEPI pump station pressure was NOT OK. A call in to Hugh helped us realize this was a re-occurrence of an over voltage error of that pump station's Variable Frequency Drive (VFD) system (see previous occurences -- seemingly once a year in the summer -- documented in LHO aLOGs 29113, 28653, 12337, 7536).

To fix, I followed the recipes described in the above mentioned aLOGs, repeated here for clarity and a refresher:
- DO NOT UNTRIP HEPI Watchdog! Untripping the ISI watchdogs are fine, but if you haven't changed the SEI manager state, as soon as you untrip all watchdogs, the manager will trying to bring everything back up, and you can't without pump pressure.
- On HEPI Pump Controller MEDM screen, switch pump controller's PID loop over to Manual, ramp down "Manual State Output Tweak" by selecting any large "Count Size Tweak" (say, 10s or 100s) and hitting the "-" until to output (H1:HPI-PUMP_EX_CONTROL_VOUT) reads zero (it was frozen at the max output of 2048).
- Drive to end station (bring CDS laptop!!), turn on wireless network once there, by plugging the dangling red cable into the bottom right most ethernet switch port.
- Head into pump controller room with big flathead screw driver. 
- Confirm that HEPI pump controller box shows a green heartbeat blinking light, and a red "status OK" light.
- Confirm that the translucent fluid in the pump fluid indicator is around 8.5 inches (there's a ruler built into the side of the indicator). Don't be fooled by the little red level indicator thing, those aren't the droids for which you're looking.
- Open up big gray VFD control box (Hugh calls it a "DD box" ...), but confirm on your way that the big 60's NASA light is on (it's RED when ON and OK)
- The VFD controller shows a red "OU3" on its top digital LED display.
- Hit RESET button in bottom, left corner, hit FWD in upper right corner of keypad. "OU3" should change to "0.00", and the green LED screen should read RUN. Nothing else will happen until you start increasing the State Output Tweak again.
- From you CDS laptop, back on the HEPI pump controller overview, set "Count Size Tweak" to 10s, and slowly (one click every 5-10 [s]) click "+" above "Manual State Output Tweak" to increase differential pressure readout (H1:HPI-PUMP_EX_DIFF_PRESS) to 70 [PSI] (a red trace will begin to appear on the long black window once you get it within range ~1-2 [PSI] of the nominal 70)
- Once you get above ~50 ish [PSI] you can click a little faster, we're just looking to prevent large turn-on transients in the servo which'll move the platform.
- Once you're close (within 10 [PSI]) you can flip the PID back over to manual.
- Once the pressure is with 1-2 [PSI] of nominal, untrip the HEPI and ISI watchdogs.

Pictures of all of these things can be found in the above referenced aLOGs.

TITLE: 07/07 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    Wind: 12mph Gusts, 10mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

• H1 was locked and JeffB and I worked on violin modes
• EX HEPI pump low pressure allarm, he pump tripped off, JeffB going to EX to fix (called Hugh, following his instructions)

TITLE: 07/06 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Bad earthquake recovery
LOG:

 See my log earlier. In addition to that recovery, we've spent the last four hours damping suspensions. Jeff got bounce and roll modes under control, we were working on violins at the last lock loss. No show stoppers, just annoying.

Also, Hanford Emergency called and the robot said not to use the regular emergency line. I didn't catch the office number it said to use, but for cell phone it said to use regular 9-1-1.

FAMIS 6530

Added 50 mL H2O to the crystal chiller. The fault light on the diode chiller is not lit and the water level indicator displays 'OK'. Both canister filters appear clear.

Still recovering from the earthquake, currently waiting for Jeff to damp down bounce modes in ALS diff. Corey had started, but there was no light visible on a lot of cameras (notably ALSX/Y had no spots), the AS spot looked bad.

What we've done so far:

1. Use the drift mon page looking back to the last lock to get most of the suspensions close. Corey had done some IMC alignment and the modecleaner was locked, so the MCs and IMs are not in exactly the same spots, but MC2 trans and IM4 trans were close.

2. ETM oplevs were way off, and the green WFS had no light on them, so we used the oplevs as a new reference and moved the ETMs until the green WFS were happy. ALS spots were still not visible.

3. We tried using the ditherAlign.py script, but it seemed to have only worked for TMSX & the script changes the TEST offsets (used for misaligning the quads) which caused problems for us later. Kiwamu & Keita think the TMSY alignment was too bad for even the ditheralign script to work. At the end of the dither of TMSY, we saw a spot move across the ITMY camera, so Kiwamu used that camera to find the ALSY spot and move TMSY to a rough alignment.

4. At about the same time as 3. we moved PR3 to recover the ALSX spot on the camera. 

5. Similarly, we moved SR3 to recover that AS camera spot position.

6. Hand alignment of ALSX and ALSY (the usual ETM/ITM/TMS tweaking to get ALS TRX/Y power up to some level high enough to engage the loops). Then finish usual ALS initial alignment.

7. During input align, we restored PRM and SRM to their alignment from the previous lock. We then noticed fringey bad crap on the AS camera. This is when we realized/remembered that the ditheralign script changes the TEST OFFSETS on the ITMs. This needs to be changed because it made input alignment impossible, because ITMY was getting misaligned enough with the new offsets. We also had to temporarily increase the LSC Xarm loop gain. I think Kiwamu said by a factor of 2.

8. During PRM align we restored PR2 alignment from the previous lock.

9. Getting Mich dark locked required taking the ALIGN_IFO guardian down and doing a rough alignment of the BS, making the AS spot and smooth and round as possible before requesting Mich dark.

10. SRC align required a large realignment of SRM. The ASC wouldn't engage until the SRM was close enough, but we didn't have to do anything special. Request SRC align and pushed SRM around until the ASC grabbed it. It was difficult to see any improvements on the AS camera and SRM has to move a lot.

11. When we got back to locking, ALSY was being difficult and the camera loops was pulling the arm out of alignment. Jeff cleared the history with the script on the ALS overview, that seemed to fix it. 

12. Jeff expected bounce modes to be high, so we spent ~1.5 hrs sitting on ALS diff while he gingerly damped them down.

After that, we had problems moving past ALS diff, so we re-did initial alignment. We're now at DRMI, moving to RF_DARM to see what other beasts are waiting for us.

DQ Shifter: Alan Weinstein, Email: ajw@ligo.caltech.edu
LSC Fellow: Paul Marsh, Email: paul.mecheng@gmail.com

• High duty cycles: 93.8% on Monday, 93.2% on Tuesday, 73.5% on Wednesday (maintenance day).
• BNS range between 60-65 Mpc throughout.
• Many range drops appear coincident with ETMY ESD overflows. Dunno if it's cause or effect. Might this be fixable with more frequent ESD sign flips?
• Tuesday maintenance delayed till Wednesday due to July 4 holiday. H1 brought out of observing mode on Wednesday 7/5, shortly after 15:00 UTC, back in observing by 21:14 UTC.
• Fire 15-20 mi NW of site forces closure of rte 240, lifted on Monday evening (local). Lots of smoke & smell, but no danger to the site or interruption of lock.
• Rode through a 5.9 Mag EQ in Chile, on Tuesday morning (local). Terramon screenshot, USGS.
• Running a2l on 7/3, 19:43 UTC reduced noise curve below 20 Hz, but greatly increased glitching from 20-40 Hz. Running a2l and aligning TMSX and TMSY on 7/3, 23:56 UTC brought it back down (alog 37299 ). Similar pathologies after Wednesday maintenance, 7/5, 21:14 UTC.
• H1:CAL-PINJX_CW_GAIN change at 7/4/17, 03:52 threw H1 out of Observe ( alog 37304 ). How did that gain change??
• Lock loss on 7/4/17, 22:13-22:57 UTC - alog 37318 - looks like it was caused by a PSL FSS DAC overflow ( plot1 ,  plot2 ) .
• Hveto: H1:SUS-ETMY_L1_NOISEMON_xx_OUT_DQ was round 1 winner, all three days. (Also UPV).
• Three GRB alerts on 7/5. LLO is not getting GRB alerts, so operator has been talking to them each time ( alog 37324 )
• No significant GraceDB triggers. Most significant is a cWB trigger with FAR of 1/(0.5 months).

Full summary is here.

at 08:01:00 PDT h1iopsusex reported a very long ADC holdtime of 77uS at 08:01:00 PDT. This caused the IOP to lose synchronization of its 18bit DAC channels and it reverted to the safe state of zeroing all DAC outputs. I restarted the IOP model, which of course required restart of all the SUS models. Jim first put the EX seismic into a safe state to allow this.

I verified the 18bit -DAC AUTOCAL reported success for all DACs, I reset the IPC errors and cleared the DAQ-CRC counters.

Here is the adc hold time information from /proc/h1iopsusex/status:

adcHoldTimeEverMax=77

adcHoldTimeEverMaxWhen=1183388478

here is the AUTOCAL data

[5393357.982766] h1iopsusex: DAC AUTOCAL SUCCESS in 5384 milliseconds 

[5393363.341051] h1iopsusex: DAC AUTOCAL SUCCESS in 5346 milliseconds 

[5393369.127216] h1iopsusex: DAC AUTOCAL SUCCESS in 5331 milliseconds 

[5393374.485325] h1iopsusex: DAC AUTOCAL SUCCESS in 5346 milliseconds 

[5393379.843006] h1iopsusex: DAC AUTOCAL SUCCESS in 5398 milliseconds

TITLE: 07/06 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: EARTHQUAKE  (recovery)
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

Issue #1: IMC

A bit of a rough shift with recovery from a rare large earthquake in Montana keeping us down.  Able to untrip Watchdogs within the first 90min or so, but a large part of the night was focused on restoring the basics from the ground up.  

Getting the IMC to relock was not trivial.  The first time I finally got it to lock I steered mainly the MC2, but after chatting with Cheryl she suggested I back out the change I made to MC2 and then focus instead on the PZT mirror (and then MC1, MC2 next).  After restoring MC1-3 to their WITNESS values before the EQ, tweaked the PZT as much as I could for locking, but I then resorted to mostly using MC2 to get the IMC locked.  

NOTE:  With IMC locked, I do not see an MC Trans video spot (but AS AIR showed a 00 mode).

Issue #2:  ALS....but probably alignment

Both ALSX & Y have low REFL A power.

Issue #3:  SUSETMX Computer

Jim noticed that the ETMx had a computer issue.  He is now on the phone with Dave addressing it.

Have left the OBSERVATORY MODE in Earthquake.

Input Mirrors Restored

Went though the recovery procedure for the IM mirrors after HAM2 SEI trips & they are now at their nominal values.  (remember:  These are downstream of the IMC...I always need to remind myself.)

IMC Status:  Not Locking

• MC Not locking.  MC REFL video spot looks off in both pitch & yaw.
• Since IMC was not re-locking, looked at MC1-3 (they were all within 5um).  I restored them to their values when the IMC was locked.  Still no IMC locking.
• Cleared Histories for IMC WFS.  No luck here.  ----This changes the value of the IMC PZT pit/yaw values.

As noted above, the IMC Refl video spot looked to be off in pitch & a little off in yaw with regards to the MC REFL camera (attached).  I have to be honest that I don't know what the location of this spot looks like for a nominal state because the IMC usually quickly locks up. 

After Clearing Histories (noted above), the IMC PZT values stepped to new values.  (6-hr trend attached).

Approaching limit of what I can do without the help of an expert.  My guess is that the PZT mirrors would be a remaining knob I have not tried, but I would prefer to NOT touch those mirrors without an expert.  (Will go through alogs to see if I can find more help.)

TITLE: 07/06 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.31 μm/s
    Secondary useism: 0.64 μm/s

Large & local EQ hit us an hour ago as Cheryl notes. 
QUICK SUMMARY:

Most SUS & SEI systems are tripped.  Also see RED DACKILL on CDS Overview for SUS.  Observatory Mode was switched to EARTHQUAKE at about 7:20utc (but we've been in this state since 6:32utc).

TITLE: 07/06 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 0Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: locked, and then not - 5.8M in Montana
LOG:

• locked when I got here
• needed a2l and ran that
• a2l made H1 glitch
• I adjusted TMSX/Y glitching stopped range went down, swarm of glitches, seemed to get better, then EQ