12:37 PDT DAQ restart to add following VAC channels

+[H0:VAC-LX_IBM_II122_AIP_IC_LOGMA]
+[H0:VAC-LX_IBM_II122_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LX_IBM_II122_AIP_IC_MA]
+[H0:VAC-LX_IBM_II122_AIP_IC_MA_ERROR]
+[H0:VAC-LX_IBM_II122_AIP_IC_VOLTS]
+[H0:VAC-LX_IBM_II122_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LX_IBM_VI122_AIP_PRESS_TORR]
+[H0:VAC-LX_IBM_VI122_AIP_PRESS_TORR_ERROR]
+[H0:VAC-LX_OBM_II136_AIP_IC_LOGMA]
+[H0:VAC-LX_OBM_II136_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LX_OBM_II136_AIP_IC_MA]
+[H0:VAC-LX_OBM_II136_AIP_IC_MA_ERROR]
+[H0:VAC-LX_OBM_II136_AIP_IC_VOLTS]
+[H0:VAC-LX_OBM_II136_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LX_OBM_VI136_AIP_PRESS_TORR]
+[H0:VAC-LX_OBM_VI136_AIP_PRESS_TORR_ERROR]
+[H0:VAC-LY_IBM_II192_AIP_IC_LOGMA]
+[H0:VAC-LY_IBM_II192_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LY_IBM_II192_AIP_IC_MA]
+[H0:VAC-LY_IBM_II192_AIP_IC_MA_ERROR]
+[H0:VAC-LY_IBM_II192_AIP_IC_VOLTS]
+[H0:VAC-LY_IBM_II192_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LY_IBM_VI192_AIP_PRESS_TORR]
+[H0:VAC-LY_IBM_VI192_AIP_PRESS_TORR_ERROR]
+[H0:VAC-LY_OBM_II196_AIP_IC_LOGMA]
+[H0:VAC-LY_OBM_II196_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LY_OBM_II196_AIP_IC_MA]
+[H0:VAC-LY_OBM_II196_AIP_IC_MA_ERROR]
+[H0:VAC-LY_OBM_II196_AIP_IC_VOLTS]
+[H0:VAC-LY_OBM_II196_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LY_OBM_VI196_AIP_PRESS_TORR]
+[H0:VAC-LY_OBM_VI196_AIP_PRESS_TORR_ERROR]
 

I was about to take a screen shot of the beams from the various PSL cavities when all of a
sudden the screen went black.  The laser tripped.  I haven't performed any forensics but it
looks like the same problem as last night.

On Monday, I went to EndX to retrieve data from the anemometers I had around the wind fence, and found the set up kind of trashed. The ultrasonic gust sensor had been blown over, which sounds like a common issue with that setup from the past. The fall, however, broke some of the plastic pieces that the screws holding the head of the sensor together. I think the sensor is still functional, but it will need to be glued back together.

One of the other cup anemometers may have been attacked by a bird, or possibly a vehicle. The cable had been unplugged from the sensor, drug more than 20 feet and the sheath and a couple wires had been cut. I would normally blame a bird for that, but the copper tube I had secured the anemometer to was also bent. It's possible that a gust of wind had bent the post, but neither of the other posts were similarly bent. There are tire tracks up around the fence, but it's hard to say how old they are, it's likely they are left over from the fence install, and none of them come very close to the post. Finally, the batteries in the logger for the 3 cup anemometers had worked free of their connections, so the logger only collected about 3 days worth of data, even though they were installed for a week and a half. The batteries are held in place by a plastic strap, and this strap probably got soft in the sun, which allowed the batteries to work they way free.

I don't know what we can do to keep birds (or bees, they seem to really like the fence) away from the outside instruments, but couple of other things I'll get for next time:

1. Cones or flags to mark a stay clear around the fence.

2. Weights and guy lines for the ultrasonic sensor, after I get it back together.

3. A foam shim to secure the batteries in the cup anemometer datalogger.

Kyle, Chandra

Starting step down of heating of Vertex RGA -> Closed calibration gas bottle isolation valves and reduced variacs (except one supplying turbo inlet) by 10% -> Will need access to continue this process every few hours or so throughout the day.  

[Sheila, Kiwamu, Terra, Jenne]

• ISS 2nd and 3rd loops simultaneously engaged. No dPdTheta instability tonight.
  • Bug in guardian code prevented this working last night:  wrong value sent to a switch to enable it.  Fixed.
• POP_A offsets tonight seemed to help POP sidebands as well as PRC gain, but still not so good for AS90.  Moving AS_C was delicate, resulted in lockloss.
• CoilDrivers state seems to ring up violin modes drastically.
  • Turned off violin mode damping before transitioning test mass L2 coils worked well.  Didn't turn damping back on, since the modes weren't high. 
  • Kiwamu noticed that the higher order violin mode notches for DARM / ETMs were off in frequency.  He has added notches to fix this.
• Switched to LVLN ETMY ESD. 
• Still need to figure out why CHARD Yaw engagement is so rough - it's better after re-doing the initial alignment setpoints, but it's not great.
• Took data for MICH and SRCL feedforward.
• Can't reshape MICH and SRCL loops as used to be done in Noise_Tunings.  Need to measure loops to understand what's up.

Next up:  Try dither loop to hold ETMX spot position in yaw to prevent spot position movement on BS.  Separate SOFT loops to X and Y, use offset of XSOFT yaw to hold ETMX spot position constant.  Alternatively (or simultaneously), dither BS and demodulate with several different signals at the same time, to understand better how the spot is moving.

We ended up moving PR2 (our uncontrolled recycling optic), and walking it with the POP_A offset.  This allowed us to get to a PRC gain of 31.6 by the O1 standard, but without the sideband powers tanking.  We think that this was promising, and will come back to it tomorrow.  Once we decide that we're happy, we should re-do the green initial alignment setpoints yet again (including the beatnote PDs, which weren't done earlier today) to set this as our reference alignment.  Attached is a big screenshot of where we were most happy.  Also attached is the PR2 offset screen, time-machined back to before we started moving PR2. 

TITLE: 09/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Sep
SHIFT SUMMARY: Lots of commissioning, a PSL trip, and a small earthquake.
LOG:

• 01:59 PSL went out, Chiller error, flow sensor 1. Peter was awesome and drove out to the site to get it back on and going for more commissioning.
• 03:14 Dave restarting H1SUSPROCPI

the h1susprocpi model had two entries in the testpoint.par file (dcuids 52 and 71). The correct ID is 71. This was causing a failure to open testpoints on this model. After the testpoint.par file was corrected, I restarted h1susprocpi and testpoints are now available.

Attached are trends of the various flow sensors in the PSL.  Comparison of the timestamps shows that the high power oscillator
tripped because of a flow rate problem with the AMP circuit.  The problem does not lie within the 4 laser heads, leaving either
the front end power amplifier, or the 4 crystal chambers.

    Unfortunately checking the 4 crystal chambers requires dismantling the laser.  The vortex flow sensor in the AMP circuit
is relatively new.  The other object in that circuit is the power amplifier module.  An inspection mirror might be able to
afford a view of the plumbing underneath the housing.

    One can see that the flow rate in the AMP circuit drops before the output of the front end laser drops, and that the output
power of the front end laser drops before the high power oscillator.  I think the sequence of events is the following:
 1. flow rate problem in the AMP circuit trips the flow watch dog of the front end laser
 2. the switching off of the front end laser breaks the injection locking of the high power oscillator
 3. the loss of injection locking results in a power drop in the high power oscillator which then trips the power watch dog
 4. the power watch dog switches the high power oscillator off

WP6121: Jonathan, Carlos, Jim, Dave

Tuesday Aug 30, the Remote Access Control system (RACCESS) was turned on during the times we have operator coverage (Mon-Fri, 8am - 4pm Pacific). This is a 'real life' test of the system, and an opportunity for everyone to get familiar with it before it is on full-time during O2. Any problems with the system should be communicated as an LHO-CDS FRS ticket.

To provide more visibility of who is logged into the border machine (cdsssh) I have expanded the RACCESS portion of the CDS_OVERVIEW medm (center right area).

WP 6131. Jonathan's new daqd code which is running on both h1fw0 and h1fw1 exports more signals via EPICS channels. At 12:42PDT today I restarted the DAQ with a new H1EDCU_DAQ.ini file which includes the full set of EPICS channels. I also added the standard set of channels for the h1fw2 frame writer.

I have extended the DAQ Overview medm to show trend file sizes, and added links to open the detailed screens for fw0 and fw1 which show the full diagnostic suite.

I redid our green initial alignment setpoints, in hopes that CHARD won't have such a large input offset when we try to engage the loops.

Unsuccessful at damping ITMX 15520 Hz PI several times tonight (previously seen here and here). We find that larger damping drive does not equal greater damping: When this mode was test driven and damped after the thermal transient at 50 W, a best gain and phase was found for damping. When the mode began to ring up later, increasing gain (by some large amound but still under saturation) or flipping sign and/or changing phase only resulted in faster ringup. This is true when still far below DAC saturation levels. It seems as if there is some gain sweet spot that must be found. 

--- --- ---

We had three occasions to damp ITMX 15520 Hz mode during the night. During the first, I successfully damped and it then rerang up (perhaps due to offset adjustments?) and lost lock. During the second and third I was not able to damp the mode and avoided lockloss only by decreasing power 50 W --> 25 W.

Below you see the mode first ring up and my gain trial and error response until I settle at 10000 and the mode is fully damped. Soon after, you see the mode ring up again right after the yaw offset step from ~ -0.02 --> -0.01. Note we started with a small negative gain so I just assumed we had actually been ringing up the mode the past few days.

During the second 50 W lock, damping was already running at the gain and phase settings that were effective at damping the first ring up above (+10000 gain, -60deg). Despite this, you see the mode slowly rising ~3 hours into the 50 W lock and my gain adjustments trying to damp. Note that here I start with some mostly successful positive gain (i.e. shallow slope) yet both raising the gain and flipping the gain sign cause the mode to ring up. I tried phase changes at this point too but existing was best. I avoided lockloss by decreasing power to 25 W, allowing mode to ring down enough to damp, then powering back up. I also rechecked my gain and phase at 50 W (post thermal transient time) and it would still drive and damp with a very steep slope. Still, an hourish later the mode began to ring up. I found similar behavior in the third attempt as the second and had to decrease power to avoid lockloss. 

Things to note: 

• After full lock aquisition, this mode takes ~ 3 hours to ring up. This morning, after dropping to 25 W then coming back to 50 W, it took only ~ 1.5 hours to ring up. Suggests something thermal.
• Q_m of ITMX ~= 18M
• Gap between ESD and ITM is larger than for ETMs ==> less actuation per drive
• Kiwamu has found similar higher gain =/= better damping in violin mode damping

Things to try:

• Retry damping immediately after decreasing power; should be able to see within ~1 min of lower power. 
• Try driving only one quadrant; test spatial overlap of ESD and mechanical mode
• Turn up ring heater (most likely ETMX RH) to decrease frequency overlap between ITMX mech mode and optical mode
• Check offset dependence

Expecting PT120 to be 3 x 10-9 torr -> Don't see logged activity to explain -> am on road and not able to trend data etc.

Executive summary: 

* Good news - as expected, the 16-Hz comb due to the OMC length dither is gone (at least at this sensitivity level)
* Bad news - low-frequency 1-Hz combs remain, and some new low-frequency combs & lines have appeared 

Some details:

 I initially looked at 15 hours of 30-minute FScan DELTAL_EXTERNAL SFTs (30 SFTs) generated during ER9 and was aghast at how bad the low-frequency spectrum looked, with a pervasive 0.485308-Hz comb ranging up to its 446th harmonic at 216.4 Hz, but when I exclude the three hours of SFTs when the 2-second ALS- glitches were present, things don't look quite so bad (see figure 1 for a sample without removal and figure 4 with removal). I dewhiten according to the new 6-pole / 6-zero, 0.3 / 30 Hz algorithm.
 The infamous 16-Hz comb due to the OMC length dither tracked down and killed here is gone (at least at this sensitivity level and these SFT statistics). As a result the high-frequency band (up to 2 kHz) is remarkably smooth with only violin modes and sporadic isolated artifacts.
 The 1-Hz comb with 0.5-Hz offset remains pervasive, but other prior 1-Hz or near-1-Hz combs with different offsets (e.g., 0.25 Hz) are not strong.
 On the other hand, there is a new near-1-Hz comb (0.996798-Hz spacing) visible to its 204th harmonic at ~203.3 Hz.
 There is also a new near-2-Hz comb (1.999951-Hz spacing) visible on approximate odd-integer-Hz frequencies, starting from ~9 Hz and visible up to ~175 Hz. This is likely the same 2-Hz comb reported in May by Bryn Pearlstone (which Ansel Neunzert kindly reminded me about today). 
 There is a "new" 56.8406-Hz comb visible to its 11th harmonic at ~625.25 Hz, which in hindsight I can see was buried in the O1 spectrum (I overlooked the pattern and indicated the teeth as isolated lines). This time the pattern was strong enough to jump out at me and to jog my memory that this comb was seen in  H2 1-arm data in 2012 in both the arm feedback channel and a quiet sensor-noise-dominated OSEM channel. This seemed to indicate a DAQ system problem at the time. I can see from O1 NoEMi line lists that this comb is pervasive in ISI, SUS and PEM channels at the corner station and both end stations.
 The old "K" comb-on-comb (0.088425-Hz fine comb attached to teeth of a coarse 76.3235-Hz comb) has more 11 more fine teeth visible on the lowest-frequency coarse-tooth comb.
 The old calibration line at 35.9 Hz has been moved to 35.3 Hz. The new excitation lines at 33.7 and 34.7 Hz are easily visible, but not as strong as the primary calibration lines. I found these changes documented here.
 There are sporadic new isolated lines here and there (indicated in line list - see below)
 There is a "crab-killer" broad bump centered at about 58.6 Hz which degrades sensitivity in the Crab Pulsar band of interest (~59.3 Hz). On the other hand, the whole noise floor is elevated w.r.t. O1, anyway. So it may be premature to worry about the bump.


Figure 1 - spectrum for 50-100 Hz when the 3-hour 2-second-glitches stretch is included (~0.5 Hz lines marked with 'h' for 'half')
Figure 2 - 0-2000 Hz (removing 3-hour bad stretch from here on)
Figure 3 - 20-50 Hz sub-band 
Figure 4 - 50-100 Hz sub-band 
Figure 5 - 100-200 Hz sub-band
Figure 6 - 1300-1400 Hz sub-band (illustration of how clean the high-frequency band is)

Also attached are a larger set of zipped sub-band spectra and a lines list (excluding the bad 3-hour stretch). Note that because the statistics here are two orders of magnitude smaller than used for the full O1 run report, I am not yet removing lines seen then that may yet re-emerge with more accumulated post-O1 data. So many of the line labels in these figures are buried in the noise fuzz for now.

Line label codes in figures:

b - Bounce mode (quad suspension)
r - Roll mode (quad suspension)
Q - Quad violin mode and harmonics
B - Beam splitter violin mode and harmonics
C - Calibration lines
M - Power mains (60 HZ)
O - 1-Hz comb (0.5-Hz offset)
o - weaker 1-Hz and near-1-Hz combs (various offsets, including zero)
H - 99.9989-Hz comb
J - 31.4127 and 31.4149-Hz combs
K - 0.088425-Hz comb-on-comb
t - 1.999951-Hz, 2.07412-Hz and 2.07423-Hz combs
D - 56.8406-Hz comb
x - single line (not all singlets in the vicinity of quad violin modes are marked, given the known upconversion)

Since there seem to be some confusions about which PD has what kind of analog filtering and sent to which channel, here it is.

I'm quite certain about HPO output before AOM (which is "Power monitor PD" in D1002929) and ISS 1st loop monitors, but not that sure about "monitor PD"s in D1002164. E-travellers are incomplete (they don't say which one is installed where), so I'm just listing the nominal values for these "monitor PD"s.

the Verbal Alarms code was logging to the ops home directory. Prior to the move of this home directory (WP5658) I have modified the code to log to a new directory:

/ligo/logs/VerbalAlarms

We restarted the program at 14:04 and verified the log files are logging correctly.