TITLE: 12/01 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.47 μm/s
QUICK SUMMARY:

• after handoff I had to "fix" the fast shutter
• the "fix" was not overly obvious, test failed, init -> test failed, auto mode -> init -> test failed
• I tried something else
• I ran the ISC_LOCK down script with Down as the chosen state, and when that was done, the fast shutter was ok again
• alignment's great, and H1 is in ENGAGE_SRC_ASC

TITLE: 12/01 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Having troubles relocking, seems 4.7khz violins are acting up
LOG:

IFO was locked when I got here. Jeff was working on tamping down second harmonic violins. Eventually got that taken care of got to observing. After an hour (after everyone else went home) started getting ETMY saturations. Guess I didn't notice the 4.7khz modes ringing up. Haven't been able to relock since, nothing I've tried has gotten these violins to quit playing. ISC_LOCK also keeps failing to go to and come out of down, I've had to select down after each lockloss and then manually go ready. Don't know if that is a problem.
 

Just had a lockloss. It started with a long string of continual EY saturations, some broad "low frequency" noise (roughly 10-50 hz) appeared in the live DARM spectra, then a lump appear around 50 hz. This eventually became a sharp peak at about 47 hz that would get large for a few seconds, then settle down a little. Attached spectra show a measurement just before the lock loss (red) with the 47 hz feature, a measurement with the low frequency stuff before the 47hz peak came up (brown), and a measurement 2 hours ago (green). 47 hz sounds familiar, but I know not from whence.

I've made my spot position plotting scripts much simpler to use, and put them in the svn folder that holds the A2L scripts.

The script to open, modify if needed, and run is /opt/rtcds/userapps/release/isc/common/scripts/decoup/BeamPosition/Run_A2L_plotter.m

This script calls 2 sub-scripts.  The first one (GetA2Lspots.m) will go through the data in ..../scripts/decoup/rec_LHO/ and pull out any measrements that have been taken since the last time this was run.  It will calculate the resulting beam spot positions for the test masses, and save the data.  The second sub-script (PlotTestMassSpotPositions.m) is just a plotter, for which you can change some parameters (start and stop time for the time axis, etc) in the Run_A2L_plotter script.  Or you can just run each script individually, using the Run_A2L_plotter as an example. 

I haven't yet added a few fancy options that I'd like, such as the option to plot the O1 or O2 average spot positions, but the basics are there.

I didn't check in our A2Ldata.mat file, but the scripts are all in the /opt/rtcds/userapps/release/isc/common/scripts/decoup/BeamPosition/ folder.

J. Kissel, E. Merilh, J. Warner, T. Hardwick, J. Driggers, S. Dwyer

Prompted by DetChar worries about glitching around the harmonics of violin modes, Ed, Jim, and I went on an epic campaign to damp the ~1kHz, 2nd harmonic violin modes. These are tricky because not all modes had been successfully damped before, and one has to alternate filters in two filter banks to hit all 8 modes for a given suspension. 

We've either used, or updated Nutsinee's violin mode table, with the notable newly damped entries being 
994.8973    ITMY     -60deg, +gain      MODE9: FM2 (-60deg), FM4 (100dB), FM9 (994.87) 
997.7169    ITMY       0deg, -400gain   MODE9: FM4 (100dB), FM6(997.717)                 VERY Slow
997.8868    ITMY       0deg, -200gain   MODE10: FM4 (100dB), FM6(997.89) 

Also, we inadvertently rung up modes around 4735 Hz, so we spent a LONG time trying to fight that. We eventually won by temporarily turning on the 4735Hz notch in FM3 of the LSC-DARM2 filter bank and waiting a few hours. I had successfully damped the ETMY mode at 4735.09 Hz by moving the band-pass filter in H1:SUS-ETMY_L2_DAMP_MODE9 's FM10 from centered around 4735.5 to centered around 4735 Hz exactly, and using positive gain with zero phase. However, there still remains a mode rung up at 4735.4 Hz but it's from an as-of-yet unidentified test mass, and we didn't want to spend the time exploring. These 4.7 kHz lines have only appeared once before in late October (LHO aLOG 31020).

Attched is a before vs. after ASD of DELTAL_EXTERNAL. I question the calibration, but what's important is the difference between the two traces. Pretty much all modes in this frequency band have been reduced by 2 or 3 orders of magnitude -- better than O1 levels. Hopefully these stick through the next few lock losses and acquisitions.

Thanks again to the above mentioned authors for all their help!

Sheila D., Evan G.

We investigated whether 1080 Hz glitching seen throughout ER10 and into the start of O2 is due to jitter noise. 

It has been seen in various runs on Hveto that a large number of the 1080 Hz glitches are vetoed with channels like H1:IMC-PZT_PIT_OUT_DQ (see, for example, here). Note that the auxiliary channel has glitches over a broad range of frequencies that are used to veto the 1080 Hz glitches.

To test if jitter really could be a culprit, we inject a broadband excitation into IMC PZT YAW and use IMC WFS A DC segment 1 and IMC WFS B DC segment 1 as witnesses for the jitter coupling. In the witness channels, the noise goes up by a factor of 2-4. At the same time, the broad noise feature in DELTAL_EXTERNAL at 1080 Hz basically remains unchanged (see attached figure). It is possible there is some underlying jitter noise at these frequencies (because the DARM spectrum goes up a little at nearby frequencies), but the primary cause of the 1080 Hz doesn't seem to be caused by jitter noise.

We tried excitations in IMC PZT PIT, but this also does not have any effect on the 1080 feature or the glitch level.

The free memory size on the guardian machine is about 4GB. At the current rate of usage we predict a reboot is needed before next Tuesday. At the next opportune time, we will increase the memory size from 12GB to 48GB and perhaps schedule regular reboots on Tuesdays. 

Plot of available memory for the month of November is attached (Y-axis mis-labelled, actually MB).

3:45pm local

Took 26 seconds to overfill CP3 from control room. Increased LLCV to 50% from 17%. Put it back to 17%.

We are having issues accessing nds2 data at the LHO nds2 server that is from the last few days.

I am looking at it now.  The data is available on the cluster at LHO, nds2 is just not seeing it.  I'm updating the frame source lists on the server and will restart it when that is done, hopefully that will clear up the problems.  ETA less than an hour.

h1cam17 was reported down this morning by Whatsup Monitoring system, we will leave the camera down since we are in Observation until is needed it, if someone has an imperative need to use it let cdsadmin know and we will reboot the camera remotely.



From: cds-alerts@LIGO.ORG
Subject: Device is Down (h1cam17.cds.ligo-wa.caltech.edu).
Date: November 30, 2016 at 9:33:38 AM PST
To: carlos.perez@ligo.org

Ping is Down on Device: h1cam17.cds.ligo-wa.caltech.edu (10.106.0.37).
Details:
Monitors that are down include: Ping
Monitors that are up include: 
Notes on this device (from device property page):
This device was scanned by discovery on 10/6/2016 10:12:15 AM.
----------------------------------------
See More
Ping is Down on Device: h1cam17.cds.ligo-wa.caltech.edu (10.106.0.37).
Details:
Monitors that are down include: Ping
Monitors that are up include: 
Notes on this device (from device property page):
This device was scanned by discovery on 10/6/2016 10:12:15 AM.
----------------------------------------

This mail was sent on November 30, 2016 at 09:33:32 AM
Ipswitch WhatsUp Gold

Day Shift: 16:00-00:00 UTC (08:00-16:00 PST)

With pomp & circumstance, O2 started with H1 down (& quickly on its way up) & a livestream of the start O2 going on in the Control Room.  Cheryl took H1 to NLN & after a bit of shuffling, we are now in OBSERVING.

• 16:20 H1 at NOMINAL LOW NOISE
  • Completed LVEA Sweep, Turned off lights at EX
  • Ran 1st A2L
  • Initial restart of HWS computer
  • Usual tweaks of PI MODE28 & 27
• 16:43  UNDISTURBED
  • 1647 Knocked out due SDF Diffs related to HWS camera tweaks
  • Have removed these channels from being monitored
  • Ran a 2nd A2L since we had been at NLN for 23min
• 17:02  UNDISTURBED
  • 17:16 Dropped out intentionally to check if high freq Calib line caused glitching (Kissel)
• 17:27 UNDISTURBED
  • 17:28 Tested INJ_TRANS Guardian Node's INJECT_KILL state by going to it, and it DID NOT drop us from Observing.  Went back to INJECT_SUCCESS.
  • 18:50 Wifi Routers at EX & EY apparently are still ON.  They are going to be remotely powered down (Carlos), but next chance we have to go out and pull their power, we'll repower them ON & then have someone go out and unplug their cables.
  • 19:36:01 Dropped out intentionally for high freq Calib Line (kissel)
• 19:36:34 UNDISTURBED
  • 20:02 Handing off to Ed for the rest of the shift

Activities For NEXT LockLoss:

1)  Hit LOAD on IMC_LOCK

2) Notify Carlos, so he can run to EX & EY to turn OFF the wifi routers.

I have been investigating the spike in the computed value of kappa_tst shown here in the summary pages:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20161122/cal/time_varying_factors/
A closer look reveals a seeming correlation between the DARM line coherence and the values of kappa_tst (see the attached plot). Currently, kappa_tst computed values are gated with PCALY_LINE1 and SUS_LINE1, the only lines used to compute kappa_tst.

Things to note in the plot:
1) kappa_tst diverges from the expected range about a minute after the DARM line coherence uncertainty skyrockets, just the amount of time it takes to corrupt the 128 running median.
2) kappa_tst as computed by CALCS also diverges when the DARM coherence goes bad.
3) Ungated kappa_pu behaves similarly to kappa_tst, but since it is gated with the DARM coherence, the GDS output is not corrupted.