Lockloss after 60 hours locking at NOMINAL LOW NOISE (beat the old record of 46 hours from three days ago). There's a 5.4M earthquake in Argentina at a depth of 198.5km happened right around the time. Terramon suggested this earthquake shouldn't drop us out of lock but LLO has also dropped lock at the same time. USGS reported no bigger earthquake.

TITLE: 09/28 [DAY Shift]: 15:00-23:00 UTC (08:00-16:00 PDT), all times posted in UTC

STATE Of H1: Observing at ~64Mpc. The ifo has been locked for 58 HOURS! 

OUTGOING OPERATOR: TJ

QUICK SUMMARY: Nominal seismic activity in earthquake band. Rising in seismic activity in anthropogenic band. RF45 behaving okay. Wind below 5 mph. No broadband loud glitches in the past half an hour.

TITLE: 09/28 [Owl Shift]: 07:00-15:00 UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Observing, ~70 MPc
SHIFT SUMMARY: Really quiet first half, then the RF45 Noise would intermittently appear. I would have to adjust H1:LSC-MOD_RF45_AM_RFSET down to fix it. Definitely seems to be getting worse.
INCOMING OPERATOR: Nutsinee

LOG:

• 07:43 - ETMY sat.
• 07:53 - H!SUSETMX Diag reset
• 08:39 - GRB alert
• 09:09 - ETMY sat.
• 09:58 - ETMY sat.
• 10:28 - ETMY sat.
• 10:52 - ETMY sat.
• 10:57-11:17 ETMY sat.
• 10:57 - RF45 Noise start
• 11:17 - Out of Observation to try to fix Noise
• 11:27 - Observing
• 12:57 - RF Noise back
• 13:07 - Out of Observing
• 13:16 - Observing
• 14:17 - ETMY sat.
• 14:19 - ETMY sat.
• 14:21 - ETMY sat.
• 14:25 - Handfull of ETMY sat.
• 14:31 - Out of Observing
• 14:43 - Observing

The RF Noise isn't quite as bad this time, for the past 10min or so it will breifly get noisy, relax for min, then get noisy againa and saturate ETMY.

LLO is down right now so I will take that opportunity to try and fine tune the RF45Set again.

The RF Noise is back again, I will try to tweak H1:LSC-MOD_RF45_AM_RFSET again.

I followed alog21716 and brought H1:LSC-MOD_RF45_AM_RFSET down to 22.2 and checked that H1:ASC-ODC_AS_A_DC_LT_TH was set to -11000. This dropped the noise down to a level much closer to what it originally was (at least according to the dtt H1_45MHz_Demod_FOM.xml). I tried tweaking the RFSET slightly to see it I could improve it further but it is too hard to tell if it gets better or worse.

Range is back up to ~70Mpc. Screenshot attached of dtt session.

I will hopefully only take it out a few minutes to follow Stephans alog21716 to adjust the modulation index on H1:LSC-MOD_RF45_AM_RFSET .

If this does not fix it I will make some calls.

Just as I starting writing about how uneventful it has been, the RF Noise started. Patrick reported that the noise fixed itself last shift, so I will wait a little before taking any action.

TITLE: 09/28 [Owl Shift]: 07:00-15:00 UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing at ~70 MPc
OUTGOING OPERATOR: Patrick
QUICK SUMMARY: H1SUSETMX Timing error being reported on the CDS Overview, I'll reset it when LLO or us drops. Environment is calm, wind speeds are less than 10 mph, and we've been locked for almost 50 hours!

TITLE: 09/27 [EVE Shift]: 23:00-07:00 UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing, ~72 MPc
SHIFT SUMMARY: Rode through four earthquakes in the Northern Mid-Atlantic Ridge. 45 Hz RF noise came and went a couple of times. A timing error has appeared on H1SUSETMX. Had a number of SUS ETMY saturations. Had a couple of OMC DCPD saturations. Wind is below 5 mph.
INCOMING OPERATOR: TJ

SUS E_T_M_Y saturating (Sun Sep 28 1:16:40 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 1:16:42 UTC)
OMC DCPD Saturation (Sun Sep 28 1:16:42 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 1:16:48 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 2:30:11 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 3:22:20 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 3:23:31 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 3:56:10 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 3:56:12 UTC)
OMC DCPD Saturation (Sun Sep 28 3:58:30 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 3:58:33 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 5:35:33 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:35 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:41 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:45 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:47 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:49 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:51 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:54 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:9:56 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:2 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:26 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:28 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:30 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:32 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:48 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:53 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:55 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:57 UTC)
SUS E_T_M_Y saturating (Sun Sep 28 6:10:59 UTC)

Throughout ER8 to present we have seen many examples of the 45 MHz EOM driver acting up. I did a study of this noise between 11th Sept - 23rd Sept using a BLRMS of H1:LSC-MOD_RF45_AM_CTRL_OUT_DQ and from this study, found a threshold on the BLRMS on which to create a data quality flag (details of exactly what I did can be found on this page). This is the flag the analyses use to completely remove this bad data from their searches (CAT1 veto)

From 12th Sept 00:00 UTC - 26th Sept 03:00 UTC the flag marked 23789 seconds (6.61 hours) of LHO data (2.93% of H1:DMT-ANALYSIS_READY). This corresponds to 17611 seconds (4.89 hours) of coincident data (3.14% of {H1,L1}:DMT_ANALYSIS_READY). 

From 26th Sept 03:00 UTC - 28th Sept 00:00 UTC the flag marked 25992 seconds (7.22 hours) of LHO data (17.5% of H1:DMT-ANALYSIS_READY). This corresponds to 10970 seconds (3.05 hours) of coincident data (12.6 % of  {H1,L1}:DMT_ANALYSIS_READY)

The EOM driver seems to have been acting up more in the last few days - the flag has marked more time as bad in the last few days as 2 weeks of data taking!

00:03 - 00:08 UTC Stepped out of control room
00:51 UTC Peter K. here eclipse hunting at LSB parking lot
~ 02:00 - 02:10 UTC Stepped out of control room
02:20 UTC heard buzzing from speaker in control room, turned it off
~02:20 UTC RF noise starting
02:39 - 02:43 UTC Stepped out of control room
RF noise stopped
SUS ETMX timing error on CDS overview

The FAR threshold used by Approval Processor to select GW event candidates for potential EM follow-up has been adjusted this evening.  It is now 1.9e-7 Hz for both the gstlal LowMass pipeline and the MBTAOnline pipeline; thus, we expect the union of them to be around 3.8e-7 Hz (one per month) of CBC triggers.  For Burst triggers, we currently are using only one pipeline, cWB, so the threshold for that is 3.8e-7 Hz.

One consequence of this is that a CBC trigger with FAR between 1.9e-7 and 3.8e-7 will cause the audible alarm to sound in the control room (because the script that does that just requires FAR < 3.8e-7) but Approval Processor will not select that trigger and the operator will not be presented with an Operator Sign-off box on the GraceDB page for that trigger.  So just don't be surprised if that happens.

TITLE: 09/27 [EVE Shift]: 23:00-07:00 UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing at ~70 MPc
OUTGOING OPERATOR: Nutsinee
QUICK SUMMARY: Lights appear off in the LVEA, PSL enclosure, end X, end Y and mid X. I can not tell from the camera if they are off at mid Y. Seismic in 0.03 - 0.1 Hz band is ~.02 um/s. Seismic in 0.1 - 0.3 Hz band is ~.06 um/s. Wind speeds are less than 20 mph.

TITLE: 09/27 [DAY Shift]: 15:00-23:00 UTC (08:00-16:00 PDT), all times posted in UTC

STATE Of H1: Observing at ~73 Mpc. The ifo has been locked 42 hours now.

SUPPORT: Mike

QUICK SUMMARY: The RF45 hasn't been acting up much (Laura suggested it probably was acting up between ~20:00-21:00 UTC, but the glitches was probably settle since it didn't set off the Verbal alarm) . Hardly any ETMY glitches alarm today. Wind speed ~10 mph. Seismic activity in 0.03-0.1 Hz band coming back to nominal.

INCOMING OPERATOR: Patrick

ACTIVITY LOG:

NO ACIVITY. It's been a boring shift (good for science!).

We tested a set of waveforms to see if they would saturation the ETMY ESD.

Test the 15Hz waveform

Previously we have tested a waveform that began at 30Hz, see aLog 21838. It was requested we try it from 15Hz instead of 30Hz.

Here are the commands I ran for testing the 15Hz waveform:
ezcawrite H1:CAL-INJ_TINJ_TYPE 1
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.1 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.25 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.5 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt 
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt 
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt

The start times of the injections from the log file (log3.txt):
SIStrOpen: Waveform starts at GPS=1127276638, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127276799, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127277026, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127277293, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127277517, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127277840, epoch=0, sample=0

I did not notice any ESD saturations.

John's waveforms

John V. has also provided 10 waveforms that I tested, see aLog 21964.

Cautiously testing the first waveform

Here are the command I entered as I slowly scaled up the first waveform's amplitude: 
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.2 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.5 -d -d >> log3.txt 
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.75 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.85 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt

The start times for these injections:
SIStrOpen: Waveform starts at GPS=1127278105, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127278209, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127278748, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127279093, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127279520, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127279644, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127280123, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127280433, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127281232, epoch=0, sample=0

The injection that began at 1127280433 was near an ETMY alert from the robot voice. The ETMY alert happened a bit after the injection happened. SO I waited, I repeated the same injection, and there was no alert.

I did one of these injections with a padding of 10 minutes so if the search groups are going to followup one of these injections in more detail this would be a good choice:
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt # gave this guy a bit of time

The start time of this injection is:
SIStrOpen: Waveform starts at GPS=1127281956, epoch=0, sample=0

Testing the other nine waveforms

Then I tested the remaining 9 waveforms by beginning with a scale factor of 0.3333 and incrementing to 1.0. I did these with little spacing between the injections so I do not think this set is worth the analysis groups following up. However, I think the injections with scale factor 1.0 should be looked at for ESD saturation (I believe Andy has a script for that):
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 1.0 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 0.3333 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 0.6666 -d -d >> log3.txt
awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 1.0 -d -d >> log3.txt

The start times of these injections:
SIStrOpen: Waveform starts at GPS=1127282013, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282066, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282120, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282173, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282226, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282278, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282326, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282377, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282422, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282476, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282517, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282577, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282631, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282681, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282727, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282770, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282819, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127282964, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283007, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283048, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283089, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283160, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283204, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283245, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283292, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283338, epoch=0, sample=0
SIStrOpen: Waveform starts at GPS=1127283440, epoch=0, sample=0

As I did these injections I had the SUS-ETMY_L3_ESDOUTF_LL_OUT channel open in dataviewer. I did not see any injection go above the 90000 counts. The highest was posterior0*.out which was near 80000 counts.