TITLE:  9/12 EVE Shift:  23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC

STATE OF H1:  Has been fairly steady at 70Mpc with a few glitches & locked for all but 40min of the shift.

SUPPORT:  Evan & Jenne were here fairlyl late

DAY'S ACTIVITIES:

• 23:00 H1 still recovering from noisy New Zealand quake.  Below are attempts during my shift
  • Dropped out at various points (Jenne tried a few tricks to help us ride the seismic:  offloading SRM Length signal from M3 to M1 & Boost for DHard Yaw)
  • Had one lockloss due to engaging the 2nd Loop for ISS (& Commissioners grumbled)
  • On about the 3rd attempt H1 finally made it up (and this was without any of Jenne's tricks).
• 23:23 NOMINAL_LOW_NOISE
• 23:25 LSC SDF diff was changed to Not Monitored (H1:LSC-PRCL_TRAMP [5.0setpoint, & 2.0epicsvalue])
• 0:23 Stefan took ifo to Undisturbed
• 0:27 Back to Commissioning for Evan Noise Budget Injections
• 2:25 Evan noticed IMC-F oscillations (~30sec period) on the Tidal striptool.  We didn't see any other oscillations/seismic noise.  He mentioned this occurred during Jeff's shift recently, too.
• 3:08 Observation Mode for H1*  (this segment has OMC REFL Beam Diverter Open; this was not noticed by SDF...is it Beckhoff?  This was for OMC jitter test.)
• 3:53 GRB Verbal Alarm
• 4:55 L1 at Low_Noise, but not Observation so, Dropped Out of Observation Mode to:
  1. Close OMC Refl Beam Diverter (ETMy saturation around this time)
  2. Allow Peter Shawhan to restart Continuous Wave Injections, but he was unsuccessful
• 5:00 Back to Observation Mode
• 5:26 Critical Alarm for H0:FMC-EX_CY_H2O_SUP_DEGF
• 6:58 Handing off to Cheryl

Restart CW postponed till tomorrow due to GRB.  Will restart Burst Injection "schedule", but it will not be active during GRB stand down time.

I still don't know where the glitches that are correlated with ETMY saturations come from, but I do know why ETMY saturates. 

We have many very deep notches in the L3 LSC LOCK filter bank in ETMY.  Some of these are as deep as -180 dB!  (There are probably others in the LSC-DARM filter bank, but I didn't look there to plot filters).  The top half of the first attachment shows all of these notches.  (Since I zoomed in and captured references of each notch with many points to see the actual depth, I had to split the filter over 2 plots - sorry).

Later, we have 2 digital filters (for each quadrant) in the ESDOUTF filter banks.  These are compensating for the analog "Acquisition" filter, and the analog low pass.  Together, they have magnitudes of more than 70 dB above 1 kHz.  The bottom half of the first attachment shows the magnitude of these filters together with the nearly-negligible-at-high-frequency filters in the DriveAlign filter bank. 

So, we are multiplying our signals by very small numbers, then multiplying them up by very large numbers.  This seems like a recipe for badness.  I don't actually know if numerical precision is the cause of these problems, but perhaps it is?

The effect of these filters is shown in the second attachment.  Note here that brown and green traces on the top and bottom plots are the same data, but with different y-axes for clarity.  The pink trace is the MASTER_OUT signal for one quadrant of the ESD, right before it goes to the DAC.  The glitch is much larger than the 132k count DAC range, so we are certainly saturating the DAC.  The blue trace is the output of the L3 LOCK L filter bank, after the filters shown in the top half of the first attachment.  The green trace is that signal, filtered by FMs 3 and 4 of the DRIVEALIGN_L2L filter bank.  There is still no obvious glitch.  The brown trace is again the output of the Lock L filter bank, filtered by the same DriveAlign filters, and then also filtered by the AntiLP in the ESD output filter bank.  Notice that due to the high gain at high frequency of that filter, we clearly see the glitch.  DTT is somehow unable to filter a signal by the AntiAcq filter (zpk([152],[3250],1,"n")), so I don't have that included in the brown trace.  But, the pink MasterOut trace is the actual signal, after all of the filters. 

The third attachment shows the time just before the lockloss around 21:00 12Sept2015, and we can clearly see that this filtering effect is causing the signal out to the DAC to be quite weird.  However, since the Lowpass and Acq filters are engaged, the signal actually going to the ESD may be mostly okay.  In this plot, the blue trace is the length signal as soon as it gets to the suspension, the red trace is after all of the notch filters, black is the MasterOut after all of the digital filtering, and green is a readback of the analog signal going to the ESD.  The glitches are basically impossible to see in any trace other than the MasterOut. 

To conclude, maybe we don't really care if the ETMY ESD saturates occasionally, but the reason it does is the huge high gain at high frequency of the digital filters that compensate for the analog acquisition and low pass filters.  Note that in ER7 we used the acquisition filter, and so some amoung of high gain at high frequency, but now in ER8 (and O1) we are also using the lowpass, so we need much more high freq high gain in our compensation filters.  As Evan pointed out in alog 21315 this is why we're seeing more of these saturations now than we used to. 

_{(Now that I've written all of this out, it all seems very obvious.  But, when I was looking at the 3rd attachment (which started this all....), it was not at all obvious to me why the black trace looked so different from the others.)}

At 5:26 UTC (10:26pmPST), received a LOW/MAJOR alarm for H0:FMC-EX_CY_H2O_SUP_DEGF with a value of 34.99degF.  Looking at trends, it appears this signal approaches this lower limit value every 24hrs (so perhaps it should be lowered more, or something needs to be addressed at EX).  It stayed in alarm for 7minutes and then the signal started turning up.

Attached is a 3-day trend of the channel.  Will send an email to Bubba about this, so he can check it out Monday.

(Peter Shawhan, Corey Gray)

After the evening's GRB alert stand-down period expired, I asked Corey to turn off the Intent bit briefly so that I could start the CW hardware injections with the new ER8B configuration.  At about 21:55 PDT I did "bin/x_start_psinject Peter Start with new ER8B config", at the injections were supposed to start at GPS 1126155410.  However, it didn't work.  The file log/psinject_H1.15_09_12_21:55:33 says: "../../bin/psinject: error while loading shared libraries: libCore.so: cannot open shared object file: No such file or directory".  I won't try to debug that tonight.

I added 17 burst hardware injections to the future schedule, at the following GPS times:
1126160500, 1126180500, 1126190500, 1126210500, 1126220500, 1126240500, 1126270500, 1126280500, 1126300500, 1126310500, 1126330500, 1126340500, 1126360500, 1126390500, 1126400500, 1126420500, 1126450500

As described the last time we started a sequence of burst injections (alog), an injection will automatically be skipped if the instrument is not in observing mode, if there is a GRB alert one-hour stand-down, if the operator disables or pauses injections, etc.

Stefan, Evan

We spent some time trying to see if some of the unexplained noise in DARM is caused by jitter (or other nonlinear pointing noise) in HAM6.

First, some things that didn't work:

• I tried shaking the table directly from 9 to 300 Hz by exciting the ISI isolation loops in X, Y, and Z (times to look at are 00:52:30–00:57:30 for Z, 01:01:00–01:06:00 for X, 01:11:10–01:21:50 for Y, all 2015-09-12 Z). I drove until the GS13s registered more than 10 times the quiescent table motion, but I did not see any effect in DARM.
• I tried shaking the table from 700 to 900 Hz by exciting the ISI isolation loops in all six dofs. [Robert had previously shaken HAM6 and found that motion in this frequency band was showing up in DARM from 50 to 200 Hz. Additionally, his assessment was that this was probably coupling in through the table.] I drove until the GS13s registered 100× the quiescent table motion, but I did not see any effect in DARM. Of course, since May we've installed an OMC shroud and moved the OMC angular dither line frequencies, so it's possible that one of these things could have eliminated that coupling. (Or is there some subtlely about using an external shaker versus shaking the ISI?)

We were able to make jitter noise by driving OM3 (I guess it would be surprising if we weren't able to do this). The first attachment shows some example bilinear coupling made by driving OM3 in yaw at 94.3 Hz.

Then we opened the OMC REFL beam diverter and tried some bandlimited injection between 35 and 45 Hz. The result is attached. We saw no coherence between OMCR and DARM (consistent with the notion that the coupling is all bilinear). Also, pitch and yaw of OMC REFL are highly cross-coupled. [Sorry for choosing a polluted band—I wanted a place where the DARM noise was low but where the OMC REFL QPDs are still plausibly seeing displacement noise.]

Finally, we made a broadband injection from 10 Hz to 300 Hz, in pitch and yaw. The times were 01:44:40–01:51:40 for yaw and 01:54:30–01:59:30 for pitch, both 2015-09-13 Z. The result for yaw is attached.

At 3:52 (8:52pmPST), Verbal Alarm notified us of a GRB.  Event info is here.  Verbal Alarm was quick with the notification (TJ and others mentioned it had been getting notices minutes/hours after an event).

Following GRB Checklist (p.2 of L1500117), we have the following:

• Confirmed it's not a test (event page says signal from sattelite).
• Interferomter is locked* (variance is that the OMC Refl Beam Diverter is open) & will stayed untouched until 4:52 (9:52pmPST).
• Interferometer is NOT unlocked.
• This does not occur near a Maintenace Day

9/12 EVE Shift:  23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC

At 3:08UTC H1 was taken to Observation Mode since Evan was done with his injections.  

Prior to this we saw some oscillations on IMC-F on the Tidal StripTool of nuc1, but we saw no other oscillations or seismic noise.  

Support:  Jenne & Evan are also in the Control Room, but sounding like they are finishing up for the night.  

Attached are plots of the time-dependent calibration correction factors as computed by the GDS calibration pipeline for GPS times 1126003432-1126011584 (Fri Sep 11 03:43:35 PDT 2015 - Fri Sep 11 05:59:27 PDT 2015).  The derivation of these factors and their meaning is described in DCC-T1500377.  

The channel names correspond to the factors as such:

GDS-CALIB_F_CC: frequency of coupled cavity pole
  - average value ~ 350 Hz
GDS-CALIB_KAPPA_C: time-dependent scale factor for the sensing 
  - average value ~ 1.07
  - expected average value = 1
GDS-CALIB_KAPPA_A_IMAGINARY: imaginary part of time-dependent scale factor for the total actuation
  - average value ~ -0.01
  - expected average value = 0
GDS-CALIB_KAPPA_A_REAL: real part of time-dependent scale factor for the total actuation
  - average value ~ 1.01
  - expected average value = 1
GDS-CALIB_KAPPA_PU_IMAGINARY: imaginary part of time-dependent scale factor for the PUM/UIM stages of actuation
  - average value ~ 0.03
  - expected average value = 0
GDS-CALIB_KAPPA_PU_REAL: real part of time-dependent scale factor for the PUM/UIM stages of actuation
  - average value ~ 1.07
  - expected average value = 1
GDS-CALIB_KAPPA_TST_IMAGINARY: imaginary part of time-dependent scale factor for the TST stage of actuation
  - average value ~ 0.05
  - expected average value = 0
GDS-CALIB_KAPPA_TST_REAL: real part of time-dependent scale factor for the TST stage of actuation
  - average value ~ 1.0
  - expected average value = 1.0

These values indicate some (O(5%)) differences in gain between the current calibration models and the IFO.  We expect this level of difference for kappa_tst and kappa_c.  However, we anticipated kappa_pu to be closer to the expected average values.  This discrepancy is still being investigated.

'tinj', the transient injection driver program, has not been running since Sept. 8 -- which is fine since there were no signals scheduled to be injected.  This evening I restarted it by logging into the h1hwinj1 machine (as user 'hinj'), cd'ing to /data/scirun/O1/HardwareInjection/Details/tinj/, and doing "nohup run_tinj >& nohup.out &".  So tinj is now running, but we have not YET put any new injections into the schedule.  We'll do that after checking with people that they won't interfere with other activities.

TITLE:  9/12 EVE Shift:  23:00-7:00UTC (16:00-0:00PDT), all times posted in UTC

STATE OF H1:  Relocking, but also riding out remnants of New Zealand EQ from 2hrs earlier.  

OUTGOING OPERATOR:  Jim

QUICK SUMMARY:  Walked in to H1 in Lock Acquisition, and saw several attempts during high seismic noise.  

• Dropped out at various points (Jenne tried a few tricks to help us ride the seismic:  offloading SRM Length signal from M3 to M1 & Boost for DHard Yaw)
• Had one lockloss due to engaging the 2nd Loop for ISS (& Commissioners grumbled).
• On about the 3rd attempt H1 finally made it up (and this was without any of Jenne's tricks).

Once in NOMINAL_LOW_NOISE, Evan took over for commissioning (I talked with Joe at LLO and they were having Bounce Woes.  When they get it locked, Anamaria/Robert would probably take L1.  So they will be out of Observation for a while.  In the meantime, Evan will conduct Noise Budget injections.  

Seismic coming down & winds inching up.

We started a sequence of scheduled transient hardware injections with burst waveforms on September 3 (LLO aLOG 20246, LHO aLOG 21186) with the intention of testing the basic operation of the low-latency burst analysis and the EM follow-up event selection logic.  It was done with the inverse actuation filters left over from ER7, so we didn't expect accurate injection of the desired strain waveforms, but that was OK for the purpose at the time.  However, we woke up the next morning to find that one of the injected coherent signals had been unexpectedly loud.  (Another injected coherent signal, though, was fine.)  Investigations of that anomalous event have been collected at https://wiki.ligo.org/LSC/JRPComm/G181472.

No further transient hardware injections have been done since then (LHO aLOG 21202, LLO aLOG 20254).  However, we are planning to resume burst hardware injections, restricted to fairly low frequencies, soon.

• Title: 9/12  Day Shift: 15:00-23:00UTC 

• State of H1: Relocking

• Support: Jenne, Stefan, Kiwamu, Evan on site

• Shift Summary: It started out quiet...

• Incoming Operator: Corey

• Activity Log:

• 15:00 arrived on site, IFO locked intent bit set
  18:44 Lockloss, no clear reason
  19:00-ish IFO back up
  19:30 ish Stefan comes on site, messes with ODC, I find LSC and ASC SDF diffs
  20:00 Stefan sets more ODC with drops us out of Observe, bit reset when he's done
  21:00 Switched to commissioning as an earthquake arrives. Jenne is trying to increase SRM offloading (prev alog) to try to ride it out
  21:04 Lockloss
  15:00 arrived on site, IFO locked intent bit set

  18:44 Lockloss, no clear reason

  19:00-ish IFO back up

  19:30 ish Stefan comes on site, messes with ODC, I find LSC and ASC SDF diffs

  20:00 Stefan sets more ODC with drops us out of Observe, bit reset when he's done

  21:00 Switched to commissioning as an earthquake arrives. Jenne is trying to increase SRM offloading (prev alog) to try to ride it out

  21:04 Lockloss

  22:something Rick came into the control room with a tour group

   

  We've not been able to recover H1 since the earthquake at 21:00, but the ground is still settling and Terramon claims an earthquake in Snoqualmie at 15:22, but I dont believe it.

   

Josh, Andy, David, Jess

Conclusion: We suspect that the 50Hz glitches in DARM are caused by EX Mains glitches that happen 400+/-70ms earlier (reliably matched) and may(?) couple through EX Seismic. 

Throughout September there has been a persitent line of glitches at 50Hz in DARM with a pretty high rate. See the attached glitch-gram from today's summary pages. Based on the one hour plots on the summary pages, the rate of these 50Hz glitches in DARM is about once per minute. 

We found (see e.g. the first round here) that these DARM glitches are correlated in time with glitches in EX channels, particularly PEM EX SEIS/ACC*, ISI ETMX ST1 BLND*, and ASC-X_TR*. 

It looks like the cause of this may be a big glitch in EX Mains. Wew see the EX and DARM channels are all glitching with a 400+/-70ms delay (averaged five examples followed up by hand) after EX mainsmon. Attached are two examples where we lined up the glitch in EX mainsmon and showed that the glitches in EX seismic and DARM are 400ms later. 

Notes:

• The EX Mains glitches are broadband, but EX Seis, ISI, and DARM are all only localized glitches at 50Hz.
• We checked and the EX microphones look clean.
• The glitches seen by EX mainsmon are also seen (sometimes) very weakly by the LVEA mainsmons (at the same time, i.e. earlier than in other EX channels and DARM).

We would be happy to follow up further leads, but wanted to report what we found so far. 

PS. Should you wish to repeat this yourself, attached is a list of these glitches in DARM today (with nice GPS times) and a screenshot of the ldvw settings I used. 

Terramon told us that we had an incoming earthquake large enough to likely break lock, so I asked Jim to take us out of Observing mode so that I could quickly try to increase the SRM offloading, to see if that would save the lock.  In the end, we still lost the lock.

I changed the SRM M1 Lock L gain from the nominal -0.02 to -0.05 with a 10 second ramp.

I need to look at the data to know more, but the verbal alarm did not complain about SRM saturations.  It did however complain about ETMY saturations. 

If that really did help prevent SRM saturations from causing our lockloss, I wonder if we should think about increasing every optics' offloading gain by a factor of 2 or 3 when we have an earthquake coming.  Anyhow, I need to look at the data to see what really happened, and to see if I can figure out what really helped.

IM4 QPD, lower 2 quadrants were saturating - lowered their whitening gain by 9dB from 36dB to 27dB (gain value from 12 to 9) IM4 is not in any loop, so no impact on feed-back.


Before clicking undisturbed, we updated some ODC settings:
- default ODC filter mask were set for PRM, SRM, SR3, ETMX and ETMYY
- took out LSC and ASC party check from MASTER mask (the party from those models is not reported correctly, see alog 20936)
- all changes are in SDF

- additionally updated the ASC ODC transmon alarm thresholds form 0.2 (Xarm) and 0.3 (Yarm) to to 0.7 at 20:02 UTC - the intent bit was unset for 2 min,. but the data is unaffected.
Note that H1:ASC-Y_TR_B_YAW_OUTPUT had drifted all the way up to -0.55 - presumably due to TMS drift.
We are using -1.1*TRY_A_YAW + 0.5*TRY_B_YAW for servoing, because this independent of TMS drift. Of course, if we move enough that we fall off one diode, that's no longer true. We should keep an eye on that.
- all changes are in SDF

A few thought as a follow on to Jeff's entry:

1. Injecting signals at DARM control (see G1501146) seems like a bad idea
   • while this worked in iLIGO, which had a relatively simple actuation TF, the DARM ctrl to DARM displacement TF is much more complicated now that we have a multi-stage suspension
   • the inverse TF looks like 1 / A * (B + C + D), which, by virtue of containing a sum in the denominator, can be very messy
   • not only is it messy, it is not robust against small changes in the DARM actuation path.  Think 1 / A * (g * B + C + D) and then let g vary... this will cause the poles and zeros to move.
   • Injecting directly into an actuator with a simple TF to DARM displacement (or force) would be much better (e.g., at the ESD or with PCAL).
2. I'll see what I can do to make an inverse filter, but I will be happy if a better way is found and this filter is never used.