As a benchmark against which to compare upcoming O2 data, I have compiled a list
of narrow lines seen in the H1 DARM spectrum up to 2000 Hz, using 107 hours of
ER10 FScan 30-minute SFTs. There are no big surprises relative to the lines and combs
Ansel and I have reported on previously from ER9 and later data, but below are some
observations. 

Attached figures show  selected band spectra, and a zipped attachment
contains a much larger set of bands. Also attached for reference is a plaintext list of combs, 
isolated lines, PEM-associated lines. etc. In the attached spectra, the red curve is the
ER10 data, and the black curve is the full-O1 data. The label annotations are keyed to
the height of the red curve, but in some cases, those labels refer to lines in the O1 data
that are not (yet) visible in accumulated current data. For the most part, lines seen in 
O1 that don't show up in ER10 nonetheless remain for now in the lines list and still
have labels on the graphs that end up in the red fuzz. If they fail to emerge in O2
data, they will be deleted from future line lists.

Observations:

As noted previously, the 8-Hz / 16-Hz combs seen in O1 are blessedly gone
Thanks to interventions to upgrade timing electronics firmware, the previously dominant low-frequency comb
with 1-Hz spacing and 0.5-Hz offset is greatly reduced (but not eliminated)
Compared to O1, the non-linear upconversion around quad violin modes and harmonics
is much worse in ER10, and it appears that at least some violin mode excitations are
affecting the low frequency band in that one sees there a multitude of sparsely distributed
doublets, triples and quadruplets of lines with a  spacing of about 0.0468 Hz.
Andy Lundgren pointed out that this spacing coincides closely with a beat between
two 2nd-harmonic violin modes he reported on last week. 
If I measure / eyeball those frequencies as seen in the ER10 data set, I get 1009.4414 and 1009.4881 Hz, giving
a beat of 0.0467 Hz (but with an uncertainty of a few tenths of a mHz), consistent with the explanation.
Further credence comes from the proliferation of such doublets / triplets / quadruplets seen in the wings of
the quad harmonics (fundamental and higher). 
There are so many upconverted lines around the quad harmonics that I didn't bother cataloguing them
in those regions on the assumption that they will go away in O2 with improved damping 
(but cataloguing can be revisited later, if needed). 
 With the exception of the violin mode regions, the higher frequency spectrum is generally free
of narrow lines (unlike the region below 100 Hz), but one disturbing feature is the presence of 
non-Gaussian broad disturbances that were not visible in O1. The infamous region around 1083 Hz 
is one example, as is the 100-200 Hz band.


Figures: [ER10 in red and O1 in black; labels are attached to peaks in the ER10 data]

Fig 1 - 0-1000 Hz
Fig 2 - 1000-2000 Hz
Fig 3 - 2-20 Hz
Fig 4 - 20-50 Hz
Fig 5 - 50-100 Hz
Fig 6 - 100-200 Hz
Fig 7 - 490-535 Hz 
Fig 8 - 1000-1000 Hz
Fig 9 - 1800-1900 Hz 

Other attachments:
* zipped file with full set of sub-band spectra
* Lines list with label definitions 

Jeff B. warned me that we would likely be getting a lot of dust monitor alarms after the PEM system was reset this afternoon.  He advised that we can set arbitrary alarm values in the meantime and he will remedy them in the morning.  I have been setting the thresholds to 1000 minor and 10000 major as the alarms occur.

CP4 LTY250 pump level went into high alarm 20 mins ago.  Cell phones have received alarms.  Waiting to talk to Chandra or someone VE...

The current alarm is not the normal "glitchiness" that has been ongoing for the last few days - 20 mins ago this signal jumped from ~95 to 100 and is riding up there.  See attached.

Reloading some Guardian code knocked us out of Observe.  I took this opportunity to run a2l.  PI modes 27 and 28 started ringing up during a2l, so Terra and I tweaked some setting and beat them back down.

J. Kissel, J. Driggers, S. Dwyer, J. Bartlett, D. Barker

Today's (not that big a deal) rabbit hole: while looking for what was preventing the IFO from being OBSERVATION_READY, we found the IFO master node complaining that the IMC_LOCK gaurdian was not ready (i.e. the H1:GRD-IMC_LOCK_OK == False), even though the node was in its nominal state, managed correctly, and in EXEC. The key was that the IMC input power was just ever so slightly out-of-range of a recently added conditional statement that prevented the run portion of the ISS_DC_COUPLED state from ever completing:

        # Hack, since power_adjust() won't change the gain when it's less than 1dB different, 30Nov2016 JCD
        if self.counter == 0:
            if ezca['IMC-PWR_IN_OUTMON'] > 28 and ezca['IMC-PWR_IN_OUTMON'] < 32:
                ezca['IMC-REFL_SERVO_IN1GAIN'] = -8
                self.counter += 1

 -- and today, for the first time, we hit an a low of 27.9 [W], the self.counter never incremented, and the code never advanced because all future conditionals relied on that self.counter to have incremented. Jenne recalls that this conditional statement was added to account for the IMC-REFL_SERVO_GAIN adjustment that happens as the ISS figures out what power it wants to send into the interferometer today (see LHO aLOG 32267 for side conversation about that), and prevent SDF from complaining that this gain was set to -7 dB instead of -8 dB.

The solution: bring the incrementing of the self.counter out of the power check:

        # Hack, since power_adjust() won't change the gain when it's less than 1dB different, 30Nov2016 JCD
        if self.counter == 0:
            if ezca['IMC-PWR_IN_OUTMON'] > 28 and ezca['IMC-PWR_IN_OUTMON'] < 32:
                ezca['IMC-REFL_SERVO_IN1GAIN'] = -8
            self.counter += 1


This bug fix has been loaded, and committed to the userapps repo. 

Once the bug fix was installed, the code was able to successfully run, the IMC_LOCK status cleared, and the IFO happily went into OBSERVATION_READY.

TITLE: 12/07 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
    Wind: 13mph Gusts, 12mph 5min avg
    Primary useism: 0.28 μm/s
    Secondary useism: 0.31 μm/s
QUICK SUMMARY:  Ran through IA at the beginning of the shift.  We are back at Observing at 1:28 UTC.
 

As I was working on a different script I found that both X and Y camera can be talked to simultaneously. So I'm leaving both codes running again. If DIAG_MAIN guardian complains about HWS code stops running please do not comment out the HWS diagnostic code just to get rid of the message. I need to know if they stop running so we can try to troubleshoot it some more.

All of the dust monitor EPICS IOC's, the FMCS EPICS IOC, and the dewpoint EPICS IOC was restarted due to a reboot of the computer on which they were running.

Not sure of the cause yet, everything seemed all good and normal. Running lockloss plots now though and I'll update if I find anything.

The TCSY chiller flow seems to be struggling for the last 40min or so. This doesn't seem to be the normal issue of an air bubble because it is not recovering as it normally would. I saw a WP for the replacement of this flow meter but I didn't see an alog stating that it happened.

Attached is an hour trend of the flow rate.

I used the lockloss2 script that automatically checks for sus saturations and plots them using the lockloss tool, and saw that one of the three locklosses (2016-12-05 16:02:42 UTC) in the last day or so was probably caused by a glitch on SR3.  The attached screenshot shows the timeline, there is clearly a glitch on the top mass of SR3 about 0.2 seconds before the lockloss.  

The dither outputs (which we use for the cage servo) don't show anything unual until after the lockloss, which means that this is not a cage servo problem.  Looking at top mass OSEMINF LF and RT are the two that seem to have a glitch first, at about the same time. 

I've added a lockloss template /ligo/home/ops/Templates/Locklosses/channels_to_look_at_SR3.txt  for any operators who have an unexplained lockloss and want to check if it is simlar to this one.