Our Utility provider needed to do some switching of our power feed yesterday and returned it to normal today.  At approximately 1545 PST 29Nov2016 DOE contractor changed our feed from one buss to another at the substation for work that needed to be done on other circuits in the substation.  At approximately 1350 PST 30Nov2016 they restored our circuit to the normal feed.   Fortunately with the new switch in the system we did not see this work.  The IFO remain locked and everything seemed fine.  A cursory look at Line monitors and Magnetometers did not reveal any anomalies.  If any of our PEM monitors did see an event at this time please let us know as this is useful feedback.

Continuing the schedule for this roaming line with a move from 2001.3 to 2501.3 Hz.

Frequency    Planned Amplitude        Planned Duration      Actual Amplitude    Start Time                 Stop Time                    Achieved Duration
(Hz)         (ct)                     (hh:mm)                   (ct)               (UTC)                    (UTC)                         (hh:mm)
---------------------------------------------------------------------------------------------------------------------------------------------------------
1001.3       35k                      02:00                   39322.0           Nov 28 2016 17:20:44 UTC  Nov 30 2016 17:16:00 UTC         days    @ 30 W  
1501.3       35k                      02:00                   39322.0           Nov 30 2016 17:27:00 UTC  Nov 30 2016 19:36:00 UTC         02:09   @ 30 W         
2001.3       35k                      02:00                   39322.0           Nov 30 2016 19:36:00 UTC  Nov 30 2016 22:07:00 UTC         02:31   @ 30 W
2501.3       35k                      05:00                   39322.0           Nov 30 2016 22:08:00 UTC
3001.3       35k                      05:00                   39322.0           
3501.3       35k                      05:00                   39322.0           
4001.3       40k                      10:00                   39322.0           
4301.3       40k                      10:00                   39322.0                
4501.3       40k                      10:00                   39322.0           
4801.3       40k                      10:00                   39222.0           
5001.3       40k                      10:00                   39222.0           

Frequency    Planned Amplitude        Planned Duration      Actual Amplitude    Start Time                 Stop Time                    Achieved Duration
(Hz)         (ct)                     (hh:mm)                   (ct)               (UTC)                    (UTC)                         (hh:mm)
---------------------------------------------------------------------------------------------------------------------------------------------------------
1001.3       35k                      02:00                   39322.0           Nov 11 2016 21:37:50 UTC    Nov 12 2016 03:28:21 UTC      ~several hours @ 25 W
1501.3       35k                      02:00                   39322.0           Oct 24 2016 15:26:57 UTC    Oct 31 2016 15:44:29 UTC      ~week @ 25 W
2001.3       35k                      02:00                   39322.0           Oct 17 2016 21:22:03 UTC    Oct 24 2016 15:26:57 UTC      several days (at both 50W and 25 W)
2501.3       35k                      05:00                   39322.0           Oct 12 2016 03:20:41 UTC    Oct 17 2016 21:22:03 UTC      days     @ 50 W
3001.3       35k                      05:00                   39322.0           Oct 06 2016 18:39:26 UTC    Oct 12 2016 03:20:41 UTC      days     @ 50 W
3501.3       35k                      05:00                   39322.0           Jul 06 2016 18:56:13 UTC    Oct 06 2016 18:39:26 UTC      months   @ 50 W
4001.3       40k                      10:00                   39322.0           Nov 12 2016 03:28:21 UTC    Nov 16 2016 22:17:29 UTC      days     @ 30 W (see LHO aLOG 31546 for caveats)
4301.3       40k                      10:00                   39322.0           Nov 16 2016 22:17:29 UTC    Nov 18 2016 17:08:49 UTC      days     @ 30 W          
4501.3       40k                      10:00                   39322.0           Nov 18 2016 17:08:49 UTC    Nov 20 2016 16:54:32 UTC      days     @ 30 W (see LHO aLOG 31610 for caveats)   
4801.3       40k                      10:00                   39222.0           Nov 20 2016 16:54:32 UTC    Nov 22 2016 23:56:06 UTC      days     @ 30 W
5001.3       40k                      10:00                   39222.0           Nov 22 2016 23:56:06 UTC    Nov 28 2016 17:20:44 UTC      days     @ 30 W (line was OFF and ON for Hardware INJ)

We are having issues accessing nds2 data at the LHO nds2 server that is from the last few days.

I am looking at it now.  The data is available on the cluster at LHO, nds2 is just not seeing it.  I'm updating the frame source lists on the server and will restart it when that is done, hopefully that will clear up the problems.  ETA less than an hour.

A couple people asked today if Pcal calibration lines could be contributing to any of the noise humps. We have tested this previously, (recent LHO aLOGs 31101 and 31035). I looked at the time last night when we shuttered the Pcal Y laser (LHO aLOG 31991).

Attached are power spectra from a few minutes before shuttering, and just after shuttering for the four Pcal Y lines. In blue is the reference time just before shuttering, while red is the time just after shuttering. Observe that the blue Pcal line frequencies (7.9 Hz, 36.7 Hz, 331.9 Hz, and 1083.7 Hz) disappear, but the noise humps around 330 Hz and 1080 Hz remain.

In addition, as noted in LHO aLOG 31991, the glitch rate remained the same during this test.

Conclusions: Pcal does not contribute to the glitching observed in DELTAL_EXTERNAL, and Pcal does not contribute to noise humps at ~330 Hz and ~1080 Hz (laser on versus off comparisons again confirm these).

h1cam17 was reported down this morning by Whatsup Monitoring system, we will leave the camera down since we are in Observation until is needed it, if someone has an imperative need to use it let cdsadmin know and we will reboot the camera remotely.



From: cds-alerts@LIGO.ORG
Subject: Device is Down (h1cam17.cds.ligo-wa.caltech.edu).
Date: November 30, 2016 at 9:33:38 AM PST
To: carlos.perez@ligo.org

Ping is Down on Device: h1cam17.cds.ligo-wa.caltech.edu (10.106.0.37).
Details:
Monitors that are down include: Ping
Monitors that are up include: 
Notes on this device (from device property page):
This device was scanned by discovery on 10/6/2016 10:12:15 AM.
----------------------------------------
See More
Ping is Down on Device: h1cam17.cds.ligo-wa.caltech.edu (10.106.0.37).
Details:
Monitors that are down include: Ping
Monitors that are up include: 
Notes on this device (from device property page):
This device was scanned by discovery on 10/6/2016 10:12:15 AM.
----------------------------------------

This mail was sent on November 30, 2016 at 09:33:32 AM
Ipswitch WhatsUp Gold

Day Shift: 16:00-00:00 UTC (08:00-16:00 PST)

With pomp & circumstance, O2 started with H1 down (& quickly on its way up) & a livestream of the start O2 going on in the Control Room.  Cheryl took H1 to NLN & after a bit of shuffling, we are now in OBSERVING.

• 16:20 H1 at NOMINAL LOW NOISE
  • Completed LVEA Sweep, Turned off lights at EX
  • Ran 1st A2L
  • Initial restart of HWS computer
  • Usual tweaks of PI MODE28 & 27
• 16:43  UNDISTURBED
  • 1647 Knocked out due SDF Diffs related to HWS camera tweaks
  • Have removed these channels from being monitored
  • Ran a 2nd A2L since we had been at NLN for 23min
• 17:02  UNDISTURBED
  • 17:16 Dropped out intentionally to check if high freq Calib line caused glitching (Kissel)
• 17:27 UNDISTURBED
  • 17:28 Tested INJ_TRANS Guardian Node's INJECT_KILL state by going to it, and it DID NOT drop us from Observing.  Went back to INJECT_SUCCESS.
  • 18:50 Wifi Routers at EX & EY apparently are still ON.  They are going to be remotely powered down (Carlos), but next chance we have to go out and pull their power, we'll repower them ON & then have someone go out and unplug their cables.
  • 19:36:01 Dropped out intentionally for high freq Calib Line (kissel)
• 19:36:34 UNDISTURBED
  • 20:02 Handing off to Ed for the rest of the shift

Activities For NEXT LockLoss:

1)  Hit LOAD on IMC_LOCK

2) Notify Carlos, so he can run to EX & EY to turn OFF the wifi routers.

No H2O added. The level is a little below the max line depending on the bubbles. Peter K. has an entry on the notepad that he added water yesterday. Are we duplicating work?

Both filter canisters look clean. There were no active alarms.

J. Kissel

Continuing the schedule for this roaming line with a move from 1501.3 to 2001.3 Hz. We (as in operators and I instead of just I) are making an effort to pay closer attention to this, so we can be done with the schedule sooner and turn off this line for the duration of the run.



Frequency    Planned Amplitude        Planned Duration      Actual Amplitude    Start Time                 Stop Time                    Achieved Duration
(Hz)         (ct)                     (hh:mm)                   (ct)               (UTC)                    (UTC)                         (hh:mm)
---------------------------------------------------------------------------------------------------------------------------------------------------------
1001.3       35k                      02:00                   39322.0           Nov 28 2016 17:20:44 UTC  Nov 30 2016 17:16:00 UTC         days    @ 30 W  
1501.3       35k                      02:00                   39322.0           Nov 30 2016 17:27:00 UTC  Nov 30 2016 19:36:00 UTC         02:09   @ 30 W         
2001.3       35k                      02:00                   39322.0           Nov 30 2016 19:36:00 UTC
2501.3       35k                      05:00                   39322.0           
3001.3       35k                      05:00                   39322.0           
3501.3       35k                      05:00                   39322.0           
4001.3       40k                      10:00                   39322.0           
4301.3       40k                      10:00                   39322.0                
4501.3       40k                      10:00                   39322.0           
4801.3       40k                      10:00                   39222.0           
5001.3       40k                      10:00                   39222.0           


Frequency    Planned Amplitude        Planned Duration      Actual Amplitude    Start Time                 Stop Time                    Achieved Duration
(Hz)         (ct)                     (hh:mm)                   (ct)               (UTC)                    (UTC)                         (hh:mm)
---------------------------------------------------------------------------------------------------------------------------------------------------------
1001.3       35k                      02:00                   39322.0           Nov 11 2016 21:37:50 UTC    Nov 12 2016 03:28:21 UTC      ~several hours @ 25 W
1501.3       35k                      02:00                   39322.0           Oct 24 2016 15:26:57 UTC    Oct 31 2016 15:44:29 UTC      ~week @ 25 W
2001.3       35k                      02:00                   39322.0           Oct 17 2016 21:22:03 UTC    Oct 24 2016 15:26:57 UTC      several days (at both 50W and 25 W)
2501.3       35k                      05:00                   39322.0           Oct 12 2016 03:20:41 UTC    Oct 17 2016 21:22:03 UTC      days     @ 50 W
3001.3       35k                      05:00                   39322.0           Oct 06 2016 18:39:26 UTC    Oct 12 2016 03:20:41 UTC      days     @ 50 W
3501.3       35k                      05:00                   39322.0           Jul 06 2016 18:56:13 UTC    Oct 06 2016 18:39:26 UTC      months   @ 50 W
4001.3       40k                      10:00                   39322.0           Nov 12 2016 03:28:21 UTC    Nov 16 2016 22:17:29 UTC      days     @ 30 W (see LHO aLOG 31546 for caveats)
4301.3       40k                      10:00                   39322.0           Nov 16 2016 22:17:29 UTC    Nov 18 2016 17:08:49 UTC      days     @ 30 W          
4501.3       40k                      10:00                   39322.0           Nov 18 2016 17:08:49 UTC    Nov 20 2016 16:54:32 UTC      days     @ 30 W (see LHO aLOG 31610 for caveats)   
4801.3       40k                      10:00                   39222.0           Nov 20 2016 16:54:32 UTC    Nov 22 2016 23:56:06 UTC      days     @ 30 W
5001.3       40k                      10:00                   39222.0           Nov 22 2016 23:56:06 UTC    Nov 28 2016 17:20:44 UTC      days     @ 30 W (line was OFF and ON for Hardware INJ)

Looking at the current glitch rate, it is elevated compared to previous locks in the last days.

Figure 1: current glitch rates (Nov 30)
Figure 2: Nov 28 glitch rates

Note that the current glitch rates are all elevated. It's easier if you open these in windows you can swap back and forth, or just stare at them side-by-side.

Looking at the SNR distribution, there is a large population of SNR<30 glitches (note large hump instead of linear decay on the log-log plot)
Figure 3: current SNR distribution (Nov 30)
Figure 4: Nov 28 SNR distribution

Now looking at the histogram of glitch SNR versus frequency, it's clear there are more numerous higher SNR glitches at low frequencies, but the higher glitch rate for low SNR glitches seems to be coming mostly from the (new?) 2 kHz and 3 kHz glitches.
Figure 5: current SNR versus frequency (Nov 30)
Figure 6: Nov 28 SNR versus frequency

J. Kissel, S. Aston, P. Fritschel

Peter was browsing through the list of frame channels and noticed that there are some differences between H1 and L1 on PR2 (an HSTS), even after we've both gone through and made the effort to revamp our channel list -- see Integration Issue 6463, ECR E1600316, LHO aLOG 30844, and LLO aLOG 29091.

What difference he found is the result of the LHO-ONLY ECR E1400369 to increase the drive strength of the lower states *some* of the HSTSs. This requires the two sites to have a different front-end model library part for different types of the same suspension because the BIO control each stage is different depending on the number of drivers that have been modified;
At LHO the configuration is
    Library Part        Driver Configuration            Optics
    HSTS_MASTER.mdl     No modified TACQ Drivers        MC1, MC3
    MC_MASTER.mdl       M2 modified, M3 not modified    MC2
    RC_MASTER.mdl       M2 and M3 modified              PRM, PR2, SRM, SR2

At LLO, the configuration is
    Library Part        Driver Configuration            Optics
    HSTS_MASTER.mdl     No modified TACQ Drivers        MC1, MC3, PR2
    MC_MASTER.mdl       M2 modified, M3 not modified    MC2, PRM, SR2, SRM
    RC_MASTER.mdl       M2 and M3 modified       none

This model's DAQ channel list for the MC and RC masters is the same. The HSTS master is different, and slower, because these SUS are used for angular control only: 
                           HSTS (Hz)        MC or RC (Hz)
    M3_ISCINF_L_IN1        2048               16384

    M3_MASTER_OUT_UL       2048               16384
    M3_MASTER_OUT_LL       2048               16384
    M3_MASTER_OUT_UR       2048               16384
    M3_MASTER_OUT_LR       2048               16384

    M3_DRIVEALIGN_L_OUT    2048               4096

Since LLO's PR2 does not have any modifications to its TACQ drivers, it uses the HSTS_MASTER model, which means that PR2 alone is going to show up as a difference in the channel list between the sites that seemed odd Peter -- that L1 had 6 more 2048 Hz channels than H1. Sadly, it *is* used for longitudinal control, so LLO suffers the lack of stored frame rate.

In order to "fix" this difference, we'd have to create a new library part for LLO's PR2 alone that has the DAC channel list of an MC or RC master, but have the BIO control logic of an HSTS master (i.e. to operate M2 and M3 stages with an unmodified TACQ driver). That seems excessive given that we already have 3 different models due to differing site preferences (and maybe range needs), so I propose we leave things as is, unless there's dire need to compare the high frequency drive signals to the M3 stage of PR2 at LLO.

I attach a screenshot that compares the DAQ channel lists for the three library parts, and the two types of control needs as defined by T1600432.

We remotely powered down the CDS WAPs in both mids and end stations by disabling the POE switch ports around 10:45PST. When opportunity arises we'll go to these locations, disconnect the ethernet cables and restart the switch ports.

I have looked at all the A2L data that we have since the last time the alignment was significantly changed, which was Monday afternoon after the PSL PZT work (alog 31951).  This is the first attached plot.

The first data point is a bit different than the rest, although I'm not totally sure why.  Other than that, we're mostly holding our spot positions quite constant.  The 3rd-to-last point, taken in the middle of the overnight lock stretch (alog 32004) shows a bit of a spot difference on ETMX, particularly in yaw, but other than that we're pretty solid.

For the next ~week, I'd like operators to run the test mass a2l script (a2l_min_lho.py) about once per day, so that we can track the spot positions a bit.  After that, we'll move to our observing run standard of running a2l once a week as part of Tuesday maintenence.

The second attached plot is just the last 2 points from the current lock.  First point was taken immediately upon lock, second was take about 30 min into the lock.  The maximum spot movement in the figure appears to be about 0.2mm, but I think that is within the error of the A2L measurement.  I can't find it right now, but once upon a time I ran A2L 5 or 7 times in a row to see how consistent the answer is, and I think I remember the stdev was about 0.3mm. 

The point of the second plot is that at 30W, it doesn't seem to make a big difference if we run a2l immediately or a little later, so we can run it for our once-a-days as soon as we lock, or when we're otherwise out of Observe, and don't have to hold off on going to Observe just for A2L.

I made a few measurements tonight, and we did a little bit more work to be able to go to observe. 

Measurements:

First, I tried to look at why our yaw ASC loops move at 1.88 Hz, I tried to modify the MICH Y loop a few times which broke the lock but Jim relocked right away.  

Then I did a repeat of noise injections for jitter with the new PZT mount, and did repeats of MICH/PRCL/SRCL/ASC injections.  Since MICH Y was about 10 times larger in DARM than pit, (it was at about the level of CHARD in DARM) I adjusted MICH Y2L by hand using a 21 Hz line.  By chaning the gain from 2.54 to 1, the coupling of the line to DARM was reduced by a bit more than a factor of 10, and the MICH yaw noise is now a factor of 10 delow darm at 20Hz.  

Lastly, I quickly checked if I could change the noise by adjusting the bias on ETMX.  A few weeks ago I had changed the bias to -400V, which reduced the 60Hz line by a factor of 2, but the line has gotten larger over the last few weeks.  However, it is still true that the best bias is -400V.  We still see no difference in the broad level of noise when changing this bias. 

Going to observe:

I've added round(,3) to the SOFT input matrix elements that needed it, and to MCL_GAIN in ANALOG_CARM

DIAG main complained about IM2 y being out the nominal range, this is because of the move we made after the IMC PZT work (31951).  I changed the nominal value from -209 to -325 for DAMP Y IN1

A few minutes after Cheryl went to observe, we were kicked out of observe again because of fiber polarization, both an SDF difference becuase of the PLL autolocker and because of a warning in DIAG main.  This is something that shouldn't kick us out of observation mode because it doesn't matter at all.  We should change DAIG_MAIN to only make this test when we are acquiring lock, and perhaps not monitor some channels in SDF observes. We decided the easiest solution for tonight was to fix the fiber polarization, so Cheryl did that. 

Lastly, Cheryl suggested that we orgainze the gaurdian state for ISC_LOCK so that states which are not normally used are above NOMINAL_LOW NOISE, I've renumbered the states but not yet loaded the guardian because I think that would knock us out of observation mode and we want to let the hardware injections happen. 

REDUCE_RF9 modulation depth guardian problem:

It seems like the reduce RF9 modulation depth state somehow skips restting some gains (screenshot shows the problem).  (noted before in alog 31558).  This could be serious, and could be why we have occasionally lost lock in this state.  I've attached a the log, this is disconcerting because the guardian log reports that it set the gains, but it seems not to have happened.  For the two PDs which did not get set, it also looks like the round step is skipped. 

We accepted the wrong values (neither of these PDs is in use in lock) in SDF so that Adam could make a hardware injection, but these are the wrong values and should be different next time we lock. The next time the IFO locks, the operator should accept the correct values

I've scheduled a CBC injection to begin at 9:20 UTC (1:20 PT).

Here is the change to the schedule file:

1164532817 H1L1 INJECT_CBC_ACTIVE 1 1.0 Inspiral/{ifo}/imri_hwinj_snr24_1163501538_{ifo}_filtered.txt

I'll be scheduling more shortly.

[Jenne, JimW, JeffK, Sheila, EvanG, Jamie]

We were ready to try hitting the Intent bit, since SDF looked clear, but kept failing.  We were auto-popped out of Observation.  With Jamie on the phone, we realized that the ODCMASTER SDF file was looking at the Observatory intent bit.  When the Observe.snap file was captured, the intent bit was not set, so when we set the intent bit, SDF saw a difference, and popped us out of Observe.  Eeek! 

We have not-monitored the observatory intent bit.  After doing this, we were able to actually set the bit, and stick in Observe. 

Talking with Jamie, it's perhaps not clear that the ODCMASTER model should be under SDF control, but at least we have something that works for now.