Distribution of hours at which scratchy glitches occurred according to the ML output from GravitySpy. In addition, a histogram of amount of O1 time spend in analysis ready mode is provided. I have uploaded omega scans and FFT spectrograms of what Scratchy glitch looked like in O1.

The results of cdsutils.avg() in guardian is sometimes giving us very weird values. 

We use this function to measure the offset value of the trans QPDs in Prep_TR_CARM.  At one point, the result of the average gave the same (wrong) value for both the X and Y QPDs, to within 9 decimal places (right side of screenshot, about halfway down).  Obviously this isn't right, but the fact that the values are identical will hopefully help track down what happened.

The next lock, it correctly got a value for the TransX (left side of screenshot, about halfway down), but didn't write a value for the TransY QPD, which indicates that it was trying to write the exact same value that was already there (epics writes aren't logged if they don't change the value). 

So, why did 3 different cdsutils averages all return a value of 751.242126465?

This isn't the first time that this has happened.  Stefan recalls at least one time from over the weekend, and I know Cheryl and I found this sometime last week. 

I drove a 222.3 Hz line in the 9 MHz RFAM stabilization error point (giving 3.4×10⁻⁴ RAN rms) and then watched the resulting lines in REFL LF and ASC NSUM as we powered up from 2 W to 25 W. [Note that the DC readback of the RFAM servo really does give us a RAN, not a RIN. This can be verified by noting that the dc value changes by a factor of 2 when the rf power is reduced by 6 dB.]

At 2 W, we have 0.013 W of 9 MHz power reflected from the PRM and 0.0007 W of 9 MHz power coming out of the AS port.

At 25 W, we have 0.11 W of 9 MHz power reflected from the PRM and 0.034 W of 9 MHz power coming out of the AS port.

The lock stretch happens around 2016-10-25 00:21:00 Z if anyone wants to look at it.

The values for the reflected PRM power still seem to imply that the 9 MHz sideband either is not strongly overcoupled to the PRC, or the modulation depth is smaller than the old PSL OSA measurement (0.22 rad). For 0.22 rad of modulation depth and strong overcoupling, we'd expect something like 0.045 W reflected off the PRM at 2 W of input power. Also, the amount of 9 MHz leaking out the AS port evidently does not scale linearly with the input power.

Summary:
The Pcal Y laser beam is likely clipping somewhere in the beam path. This will need to be addressed ASAP. In the future we need to keep a close eye on the Pcal summary spectra on the DetChar web pages.

Details:
Jeff K. and I noticed that the spectrum for the Y-end Pcal seemed particularly noisy. I plotted some TX and RX PD channels at different times since Oct. 11. Several days since Oct. 11, the Pcal team has been to EY to perform some Pcal maintenance. One of those times (I think Oct. 18, but we don't have an aLOG reference for this), we realigned the beams on the test mass. Potentially, this change caused some clipping.

Attached are the spectra for TX and RX. Notice that there are no dramatic changes in the TX spectra. In the RX spectra, there is structure becoming more apparent with time in the 15-30 Hz region and 90-140 Hz. Also, various other peaks are growing

Also attached is a minute trend of the TX and RX PD mean values. On Oct 18, after realignment (the step down), the RX PD starts to drift downward while the TX PD power holds steady. The decrease in RX PD is nearly 10% from the start of the realignment. 

The Pcal team should address this ASAP, hopefully during tomorrow's maintenance time.

Here is the sequence of events which happened in the 18:23 minute PDT Sat 22 October

Having both frame writers down meant we lost three full frames for the GPS times 1161220928, 1161220992, 1161221056

Clearly fixing the retransmission errors will become a higher priority if they are cascaded like this and not random as they have been in the past. Our third frame writer h1fw2 did not crash and could have been used to fill in the gap if it were to be connected to LDAS.

3:32 pm local

Overfilled CP3 by doubling LLCV to 34%. Took 18 min. to see vapor and tiny amounts of LN2 out the exhaust line. TCs responded but still did not readout high negative #s like I'd expect or saw with 1/2 turn on bypass fill valve. So I left LLCV at 35% for an additional 7 min. but did not see a drastic change in temps. Never saw much flow out of the bypass exhaust pipe. 

Bumped LLCV nominal from 17% to 18% open.

Removed and replaced battery packs for all vacuum rack UPSes (Ends/Mids/Corner station).  No glitches noted on racks.

Work done under WP#6270.

The front end diode box that was removed from service some months ago, S/N DB 12-07
had one of its Lumina power supplies replaced - the one on the right hand side as
you face the front panel and key switch.
  old: S/N 38226
  new: S/N 118533


Fil/Gerardo/Peter

J. Kissel
Integration Issue 6463
ECR: E1600316
WP: 6263

P. Fritschel, S. Aston, and I have revamped the SUS channel list that is stored in the frames in order to
(1) Reduce the overall channel frame rate in light of the new scheme of storing only one science frame (no commissioning frames), and
(2) because the list was an un-organized hodge podge of inconsistent ideas of what to store from over 6 years of ideas.
The new channel list (and the rationale for each channel and its rate) can be found in T1600432, and will not change throughout O2.

I've spend the day modifying front-end models such that they all conform to this new model. This impacts *every* SUS model, and we'll install the changes tomorrow (including the removal of ISIWIT channels, prepped on Friday; LHO aLOG 30728).

For the SUS models used in any control system, the channel list was changed in the respective suspension type's library part,
  Sending        BSFM_MASTER.mdl
  Sending        HLTS_MASTER.mdl
  Sending        HSSS_MASTER.mdl
  Sending        HSTS_MASTER.mdl
  Sending        MC_MASTER.mdl
  Sending        OMCS_MASTER.mdl
  Sending        QUAD_ITM_MASTER.mdl
  Sending        QUAD_MASTER.mdl
  Sending        RC_MASTER.mdl
  Sending        TMTS_MASTER.mdl
  Transmitting file data ..........
  Committed revision 14509.
and commited to the user apps repo.

For monitor models, the changes are done on the secondary level, but that layer is not made of library parts, so they have to be individually changed per suspension. These models, 
  Sending        models/h1susauxb123.mdl
  Sending        models/h1susauxex.mdl
  Sending        models/h1susauxey.mdl
  Sending        models/h1susauxh2.mdl
  Sending        models/h1susauxh34.mdl
  Sending        models/h1susauxh56.mdl
  Transmitting file data ......
  Committed revision 14508.
are also no committed to the userapps repo.

We're ready for a rowdy maintenance day tomorrow -- hold on to your butts!

Today I did some more cleanup work on the /opt/rtcds file system following yesterday's full-filesytem errors.

We perform hourly zfs snapshots on this file system, and zfs-sync them to the backup machine h1fs1 at the same rate. h1fs0 had hourly snapshots going back to May 2016.

Yesterday I had deleted all of May and thinned June down to one-per-day. Today we made the decision that since all the files are backed up to tape, we can delete all snapshots older than 30 days. This will ensure that disk allocaed to a deleted file will be recovered after the last snapshot which references it is destroyed after 30 days. I destroyed all snapshots up to 26th September 2016.

After the snapshot cleanup, the 928G file system is using 728G (split as 157G used by snapshots and 571G used by disk-system). This is a usage of 78% which is what DF reports.

• Some opportunistic maintenance went on today during the PSL incursion.
• Not much successfull locking

Ed, Sheila

Are ezca connection errors becoming more frequent?  Ed has had two in the last hour or so, one of which contributed to a lockloss (ISC_DRMI).

The first one was from ISC_LOCK, the screenshot is attached. 

[Jenne, Daniel, Stefan]

There seems to be an offset somewhere in the ISS second loop.  When the 2nd loop comes on, even though it is supposed to be AC coupled, the diffracted power decreases significantly.  This is very repeatable with on/off/on/off tests.  One bad thing about this (other than having electronics with unknown behavior) is that the diffracted power is very low, and can hit the bottom rail, causing lockloss - this happened just after we started trending the diffracted power to see why it was so low.

Daniel made it so the second loop doesn't change the DC level of diffracted power by changing the input offset for the AC coupling servo (H1:PSL-ISS_SECONDLOOP_AC_COUPLING_SERVO_OFFSET from 0.0 to -0.5), the output bias of the AC coupling servo (H1:PSL-ISS_SECONDLOOP_AC_COUPLING_INT_BIAS from 210 to 200), and the input offset of the 2nd loop (H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET from 24.0 to 23.5  - this is just summed in to the error point of the 2nd loop servo).  What we haven't checked yet is if we can increase the laser power with these settings.

Why is there some offset in the ISS 2nd loop that changes the diffracted power??  When did this start happening?

At 5:20am local time we saw a significant range drop (from about 70Mpc to 60Mpc) that seems to be due to a signicant increase of the line structure in the bucket that always lingers around the noise floor.

Attached are two spectra - 1h12min apart (from 11:18:00 and 12:20:00 UTC on 2016/10/24), showing that structure clearly.

Plot two shows the seismic BLRMS from the three LEAs - the corner shows the clearest increase. We are now chasing any particularly bad correlations around 12:20 thgis morning in the hope that it will give us a hint where this scatter is from.