Just had a lockloss. It started with a long string of continual EY saturations, some broad "low frequency" noise (roughly 10-50 hz) appeared in the live DARM spectra, then a lump appear around 50 hz. This eventually became a sharp peak at about 47 hz that would get large for a few seconds, then settle down a little. Attached spectra show a measurement just before the lock loss (red) with the 47 hz feature, a measurement with the low frequency stuff before the 47hz peak came up (brown), and a measurement 2 hours ago (green). 47 hz sounds familiar, but I know not from whence.

I've made my spot position plotting scripts much simpler to use, and put them in the svn folder that holds the A2L scripts.

The script to open, modify if needed, and run is /opt/rtcds/userapps/release/isc/common/scripts/decoup/BeamPosition/Run_A2L_plotter.m

This script calls 2 sub-scripts.  The first one (GetA2Lspots.m) will go through the data in ..../scripts/decoup/rec_LHO/ and pull out any measrements that have been taken since the last time this was run.  It will calculate the resulting beam spot positions for the test masses, and save the data.  The second sub-script (PlotTestMassSpotPositions.m) is just a plotter, for which you can change some parameters (start and stop time for the time axis, etc) in the Run_A2L_plotter script.  Or you can just run each script individually, using the Run_A2L_plotter as an example. 

I haven't yet added a few fancy options that I'd like, such as the option to plot the O1 or O2 average spot positions, but the basics are there.

I didn't check in our A2Ldata.mat file, but the scripts are all in the /opt/rtcds/userapps/release/isc/common/scripts/decoup/BeamPosition/ folder.

J. Kissel, E. Merilh, J. Warner, T. Hardwick, J. Driggers, S. Dwyer

Prompted by DetChar worries about glitching around the harmonics of violin modes, Ed, Jim, and I went on an epic campaign to damp the ~1kHz, 2nd harmonic violin modes. These are tricky because not all modes had been successfully damped before, and one has to alternate filters in two filter banks to hit all 8 modes for a given suspension. 

We've either used, or updated Nutsinee's violin mode table, with the notable newly damped entries being 
994.8973    ITMY     -60deg, +gain      MODE9: FM2 (-60deg), FM4 (100dB), FM9 (994.87) 
997.7169    ITMY       0deg, -400gain   MODE9: FM4 (100dB), FM6(997.717)                 VERY Slow
997.8868    ITMY       0deg, -200gain   MODE10: FM4 (100dB), FM6(997.89) 

Also, we inadvertently rung up modes around 4735 Hz, so we spent a LONG time trying to fight that. We eventually won by temporarily turning on the 4735Hz notch in FM3 of the LSC-DARM2 filter bank and waiting a few hours. I had successfully damped the ETMY mode at 4735.09 Hz by moving the band-pass filter in H1:SUS-ETMY_L2_DAMP_MODE9 's FM10 from centered around 4735.5 to centered around 4735 Hz exactly, and using positive gain with zero phase. However, there still remains a mode rung up at 4735.4 Hz but it's from an as-of-yet unidentified test mass, and we didn't want to spend the time exploring. These 4.7 kHz lines have only appeared once before in late October (LHO aLOG 31020).

Attched is a before vs. after ASD of DELTAL_EXTERNAL. I question the calibration, but what's important is the difference between the two traces. Pretty much all modes in this frequency band have been reduced by 2 or 3 orders of magnitude -- better than O1 levels. Hopefully these stick through the next few lock losses and acquisitions.

Thanks again to the above mentioned authors for all their help!

Sheila D., Evan G.

We investigated whether 1080 Hz glitching seen throughout ER10 and into the start of O2 is due to jitter noise. 

It has been seen in various runs on Hveto that a large number of the 1080 Hz glitches are vetoed with channels like H1:IMC-PZT_PIT_OUT_DQ (see, for example, here). Note that the auxiliary channel has glitches over a broad range of frequencies that are used to veto the 1080 Hz glitches.

To test if jitter really could be a culprit, we inject a broadband excitation into IMC PZT YAW and use IMC WFS A DC segment 1 and IMC WFS B DC segment 1 as witnesses for the jitter coupling. In the witness channels, the noise goes up by a factor of 2-4. At the same time, the broad noise feature in DELTAL_EXTERNAL at 1080 Hz basically remains unchanged (see attached figure). It is possible there is some underlying jitter noise at these frequencies (because the DARM spectrum goes up a little at nearby frequencies), but the primary cause of the 1080 Hz doesn't seem to be caused by jitter noise.

We tried excitations in IMC PZT PIT, but this also does not have any effect on the 1080 feature or the glitch level.

The free memory size on the guardian machine is about 4GB. At the current rate of usage we predict a reboot is needed before next Tuesday. At the next opportune time, we will increase the memory size from 12GB to 48GB and perhaps schedule regular reboots on Tuesdays. 

Plot of available memory for the month of November is attached (Y-axis mis-labelled, actually MB).

3:45pm local

Took 26 seconds to overfill CP3 from control room. Increased LLCV to 50% from 17%. Put it back to 17%.

Kiwamu Daniel

The attached spectrum shows the IMC-WFS_[AB]_[IQ][1234] signals. Electronics noise dominates above 2 kHz for WFS B and above 1 kHz for WFS A. IMC-WFS_A_Q4 is either exactly in the Q phase or broken. The DC signals seem to be even closer to the ADC noise, since they have no whitening filters.

Our Utility provider needed to do some switching of our power feed yesterday and returned it to normal today.  At approximately 1545 PST 29Nov2016 DOE contractor changed our feed from one buss to another at the substation for work that needed to be done on other circuits in the substation.  At approximately 1350 PST 30Nov2016 they restored our circuit to the normal feed.   Fortunately with the new switch in the system we did not see this work.  The IFO remain locked and everything seemed fine.  A cursory look at Line monitors and Magnetometers did not reveal any anomalies.  If any of our PEM monitors did see an event at this time please let us know as this is useful feedback.

Continuing the schedule for this roaming line with a move from 2001.3 to 2501.3 Hz.

Frequency    Planned Amplitude        Planned Duration      Actual Amplitude    Start Time                 Stop Time                    Achieved Duration
(Hz)         (ct)                     (hh:mm)                   (ct)               (UTC)                    (UTC)                         (hh:mm)
---------------------------------------------------------------------------------------------------------------------------------------------------------
1001.3       35k                      02:00                   39322.0           Nov 28 2016 17:20:44 UTC  Nov 30 2016 17:16:00 UTC         days    @ 30 W  
1501.3       35k                      02:00                   39322.0           Nov 30 2016 17:27:00 UTC  Nov 30 2016 19:36:00 UTC         02:09   @ 30 W         
2001.3       35k                      02:00                   39322.0           Nov 30 2016 19:36:00 UTC  Nov 30 2016 22:07:00 UTC         02:31   @ 30 W
2501.3       35k                      05:00                   39322.0           Nov 30 2016 22:08:00 UTC
3001.3       35k                      05:00                   39322.0           
3501.3       35k                      05:00                   39322.0           
4001.3       40k                      10:00                   39322.0           
4301.3       40k                      10:00                   39322.0                
4501.3       40k                      10:00                   39322.0           
4801.3       40k                      10:00                   39222.0           
5001.3       40k                      10:00                   39222.0           

Frequency    Planned Amplitude        Planned Duration      Actual Amplitude    Start Time                 Stop Time                    Achieved Duration
(Hz)         (ct)                     (hh:mm)                   (ct)               (UTC)                    (UTC)                         (hh:mm)
---------------------------------------------------------------------------------------------------------------------------------------------------------
1001.3       35k                      02:00                   39322.0           Nov 11 2016 21:37:50 UTC    Nov 12 2016 03:28:21 UTC      ~several hours @ 25 W
1501.3       35k                      02:00                   39322.0           Oct 24 2016 15:26:57 UTC    Oct 31 2016 15:44:29 UTC      ~week @ 25 W
2001.3       35k                      02:00                   39322.0           Oct 17 2016 21:22:03 UTC    Oct 24 2016 15:26:57 UTC      several days (at both 50W and 25 W)
2501.3       35k                      05:00                   39322.0           Oct 12 2016 03:20:41 UTC    Oct 17 2016 21:22:03 UTC      days     @ 50 W
3001.3       35k                      05:00                   39322.0           Oct 06 2016 18:39:26 UTC    Oct 12 2016 03:20:41 UTC      days     @ 50 W
3501.3       35k                      05:00                   39322.0           Jul 06 2016 18:56:13 UTC    Oct 06 2016 18:39:26 UTC      months   @ 50 W
4001.3       40k                      10:00                   39322.0           Nov 12 2016 03:28:21 UTC    Nov 16 2016 22:17:29 UTC      days     @ 30 W (see LHO aLOG 31546 for caveats)
4301.3       40k                      10:00                   39322.0           Nov 16 2016 22:17:29 UTC    Nov 18 2016 17:08:49 UTC      days     @ 30 W          
4501.3       40k                      10:00                   39322.0           Nov 18 2016 17:08:49 UTC    Nov 20 2016 16:54:32 UTC      days     @ 30 W (see LHO aLOG 31610 for caveats)   
4801.3       40k                      10:00                   39222.0           Nov 20 2016 16:54:32 UTC    Nov 22 2016 23:56:06 UTC      days     @ 30 W
5001.3       40k                      10:00                   39222.0           Nov 22 2016 23:56:06 UTC    Nov 28 2016 17:20:44 UTC      days     @ 30 W (line was OFF and ON for Hardware INJ)

We are having issues accessing nds2 data at the LHO nds2 server that is from the last few days.

I am looking at it now.  The data is available on the cluster at LHO, nds2 is just not seeing it.  I'm updating the frame source lists on the server and will restart it when that is done, hopefully that will clear up the problems.  ETA less than an hour.

A couple people asked today if Pcal calibration lines could be contributing to any of the noise humps. We have tested this previously, (recent LHO aLOGs 31101 and 31035). I looked at the time last night when we shuttered the Pcal Y laser (LHO aLOG 31991).

Attached are power spectra from a few minutes before shuttering, and just after shuttering for the four Pcal Y lines. In blue is the reference time just before shuttering, while red is the time just after shuttering. Observe that the blue Pcal line frequencies (7.9 Hz, 36.7 Hz, 331.9 Hz, and 1083.7 Hz) disappear, but the noise humps around 330 Hz and 1080 Hz remain.

In addition, as noted in LHO aLOG 31991, the glitch rate remained the same during this test.

Conclusions: Pcal does not contribute to the glitching observed in DELTAL_EXTERNAL, and Pcal does not contribute to noise humps at ~330 Hz and ~1080 Hz (laser on versus off comparisons again confirm these).

h1cam17 was reported down this morning by Whatsup Monitoring system, we will leave the camera down since we are in Observation until is needed it, if someone has an imperative need to use it let cdsadmin know and we will reboot the camera remotely.



From: cds-alerts@LIGO.ORG
Subject: Device is Down (h1cam17.cds.ligo-wa.caltech.edu).
Date: November 30, 2016 at 9:33:38 AM PST
To: carlos.perez@ligo.org

Ping is Down on Device: h1cam17.cds.ligo-wa.caltech.edu (10.106.0.37).
Details:
Monitors that are down include: Ping
Monitors that are up include: 
Notes on this device (from device property page):
This device was scanned by discovery on 10/6/2016 10:12:15 AM.
----------------------------------------
See More
Ping is Down on Device: h1cam17.cds.ligo-wa.caltech.edu (10.106.0.37).
Details:
Monitors that are down include: Ping
Monitors that are up include: 
Notes on this device (from device property page):
This device was scanned by discovery on 10/6/2016 10:12:15 AM.
----------------------------------------

This mail was sent on November 30, 2016 at 09:33:32 AM
Ipswitch WhatsUp Gold

Day Shift: 16:00-00:00 UTC (08:00-16:00 PST)

With pomp & circumstance, O2 started with H1 down (& quickly on its way up) & a livestream of the start O2 going on in the Control Room.  Cheryl took H1 to NLN & after a bit of shuffling, we are now in OBSERVING.

• 16:20 H1 at NOMINAL LOW NOISE
  • Completed LVEA Sweep, Turned off lights at EX
  • Ran 1st A2L
  • Initial restart of HWS computer
  • Usual tweaks of PI MODE28 & 27
• 16:43  UNDISTURBED
  • 1647 Knocked out due SDF Diffs related to HWS camera tweaks
  • Have removed these channels from being monitored
  • Ran a 2nd A2L since we had been at NLN for 23min
• 17:02  UNDISTURBED
  • 17:16 Dropped out intentionally to check if high freq Calib line caused glitching (Kissel)
• 17:27 UNDISTURBED
  • 17:28 Tested INJ_TRANS Guardian Node's INJECT_KILL state by going to it, and it DID NOT drop us from Observing.  Went back to INJECT_SUCCESS.
  • 18:50 Wifi Routers at EX & EY apparently are still ON.  They are going to be remotely powered down (Carlos), but next chance we have to go out and pull their power, we'll repower them ON & then have someone go out and unplug their cables.
  • 19:36:01 Dropped out intentionally for high freq Calib Line (kissel)
• 19:36:34 UNDISTURBED
  • 20:02 Handing off to Ed for the rest of the shift

Activities For NEXT LockLoss:

1)  Hit LOAD on IMC_LOCK

2) Notify Carlos, so he can run to EX & EY to turn OFF the wifi routers.

Looking at the current glitch rate, it is elevated compared to previous locks in the last days.

Figure 1: current glitch rates (Nov 30)
Figure 2: Nov 28 glitch rates

Note that the current glitch rates are all elevated. It's easier if you open these in windows you can swap back and forth, or just stare at them side-by-side.

Looking at the SNR distribution, there is a large population of SNR<30 glitches (note large hump instead of linear decay on the log-log plot)
Figure 3: current SNR distribution (Nov 30)
Figure 4: Nov 28 SNR distribution

Now looking at the histogram of glitch SNR versus frequency, it's clear there are more numerous higher SNR glitches at low frequencies, but the higher glitch rate for low SNR glitches seems to be coming mostly from the (new?) 2 kHz and 3 kHz glitches.
Figure 5: current SNR versus frequency (Nov 30)
Figure 6: Nov 28 SNR versus frequency

I made a few measurements tonight, and we did a little bit more work to be able to go to observe. 

Measurements:

First, I tried to look at why our yaw ASC loops move at 1.88 Hz, I tried to modify the MICH Y loop a few times which broke the lock but Jim relocked right away.  

Then I did a repeat of noise injections for jitter with the new PZT mount, and did repeats of MICH/PRCL/SRCL/ASC injections.  Since MICH Y was about 10 times larger in DARM than pit, (it was at about the level of CHARD in DARM) I adjusted MICH Y2L by hand using a 21 Hz line.  By chaning the gain from 2.54 to 1, the coupling of the line to DARM was reduced by a bit more than a factor of 10, and the MICH yaw noise is now a factor of 10 delow darm at 20Hz.  

Lastly, I quickly checked if I could change the noise by adjusting the bias on ETMX.  A few weeks ago I had changed the bias to -400V, which reduced the 60Hz line by a factor of 2, but the line has gotten larger over the last few weeks.  However, it is still true that the best bias is -400V.  We still see no difference in the broad level of noise when changing this bias. 

Going to observe:

I've added round(,3) to the SOFT input matrix elements that needed it, and to MCL_GAIN in ANALOG_CARM

DIAG main complained about IM2 y being out the nominal range, this is because of the move we made after the IMC PZT work (31951).  I changed the nominal value from -209 to -325 for DAMP Y IN1

A few minutes after Cheryl went to observe, we were kicked out of observe again because of fiber polarization, both an SDF difference becuase of the PLL autolocker and because of a warning in DIAG main.  This is something that shouldn't kick us out of observation mode because it doesn't matter at all.  We should change DAIG_MAIN to only make this test when we are acquiring lock, and perhaps not monitor some channels in SDF observes. We decided the easiest solution for tonight was to fix the fiber polarization, so Cheryl did that. 

Lastly, Cheryl suggested that we orgainze the gaurdian state for ISC_LOCK so that states which are not normally used are above NOMINAL_LOW NOISE, I've renumbered the states but not yet loaded the guardian because I think that would knock us out of observation mode and we want to let the hardware injections happen. 

REDUCE_RF9 modulation depth guardian problem:

It seems like the reduce RF9 modulation depth state somehow skips restting some gains (screenshot shows the problem).  (noted before in alog 31558).  This could be serious, and could be why we have occasionally lost lock in this state.  I've attached a the log, this is disconcerting because the guardian log reports that it set the gains, but it seems not to have happened.  For the two PDs which did not get set, it also looks like the round step is skipped. 

We accepted the wrong values (neither of these PDs is in use in lock) in SDF so that Adam could make a hardware injection, but these are the wrong values and should be different next time we lock. The next time the IFO locks, the operator should accept the correct values