On Tuesday we reconfigured h1fw1 to run the newer code (same code as h1fw0), and put the older code on h1fw2 for comparison purposes.

So:
h1fw0 and h1fw1 are running r4252 from branch-3.0 in svn.
h1fw2 is running as the advLigoRTS-3.0.3 release tagged for O1.

We have not seen frame writer crashes in several weeks now.  However we are seeing h1fw0 produce a small number of frames that are different from what is produced by h1fw1 and h1fw2.

With analysis done by Greg Mendell and David Barker it appears that the daqd does not receive all the data occasionally and issues some re transmit requests for some of the data.  When it does this some of the data is put in out of order.

So apparently data that should arrive in as

1 2 3 4 5

may be stored as:

1 3 4 5 2

I will review the network receive code tomorrow to see if this is what is happening.  We find it curious that it inserts the data it skips later on.

We are not sure what is causing this, but a leading theory is that this may be a network issue between the h1dc0 and h1fw0.  Next Tuesday Dave plans to switch the connections on the switch between h1fw0 and h1fw1.  If the behavior switches between h1fw0 and h1fw1 we will be able to attribute this to a problem with either the port, sfp, or fiber that is currently connecting h1fw0 to the switch.

I will work with Carlos to see if we can get error counters from the switch.

The plot shows that an alignment study last night changed jitter coupling reproducibly. Notice that the different peaks from the PSL seem to change differently, so it seems more complex than something like clipping.

Elli (remotely), Kiwamu,

Today was another day for the SRC gouy phase measurement. Even though I said that I would try out another misalignment configuration (29418), we decided to repeat the same measuremnet/configuration as yesterday with an improved clipping situation.

We will stop doing the measurement for now until we hear from Elli on her offline analysis. Done for now.

[The measurements]

• measurement #1
  •  18:26:27 - 18:36:27 UTC
  • BS yaw excitation with amplitude of 6 urad at 0.2 Hz
• measurement #2
  • 18:39:02 - 18:49:02 UTC
  • BS yaw excitation with amplitude of 3 urad at 0.2 Hz
• measurement #3
  • 18:52:34 - 19:02:34 UTC
  • PR2 yaw excitation with amplitude of 20 urad at 0.2 Hz
• measurement #4
  • 19:07:15 - 19:17:15 UTC
  • PR2 yaw excitation with amplitude of 10 urad at 0.2 Hz

[The setup]

• PSL = 22.4 W
• CO2Y = 286 mW
• ITMY ring heater = 0.5 W
• ifo configuration = single bounce (with ITMY aligned) + slightly misaligned SRM
• There was a large earthquake about 1 hour before I started the measurements. The spot positions on the camera seemed to move more than usual below 0.1 Hz.
• Together with SRM misalignment, I iteratively introduced misalignment on BS and PR2 as well in order to get less clipping on the spot on ASAIR.
• See the alignment sliders in the first and second attachments.
• The clipping on the spot on ASAIR was improved. See the screenshot below. The picture on the right is from yesterday and the picture on the left is from this morning.

[The data]

All the relevant data is available at kiwamu.izumi/Public/measurements/20160831_SRCgouy2/data2

Still thinking about earthquakes, I have 2 main points:

1. We can reduce the number of times ISIs trip during earthquakes by reducing the amount of inertial isolation we do a low frequency. Even if we can't lock, changing seismic configurations may save time in recovery.

2. The current earthquake configuration amplifies microseism (.1-.3hz motion) by a factor of a couple. I have some thoughts on improving that, but we need to balance not making things worse at .1-.3 hz (for the summer, we'll need to do better come November) and rolling off the seismometers quickly below .1hz. I've made a Earth_Quake_V2 to address this, but theres more work here to do.

This morning New Zealand had a pretty big earthquake, with peak .03-.1hz blrms velocties around 10 micron/s at LHO.  We noticed this before the velocities reached around 1 micron/s, so I moved the SEI_CONF manager to the Earth_Quake state. Looking at ground and WD trends for the ISI's over the last two months, it seems that normally the BSC ISIs trip at 1-2 micron/s, and today all of the platforms survived. My first plot shows the ITMY .03-.1hz BLRMS, ITMY ST2 WD MON bit and SEI_CONF state (40 is the "windy" state, 17 is the "earthquake" state) for the earthquake today, the second is a plot of an earthquake from Aug 29th. The earthquake on Aug 29th took out all of test mass ISIs (WD_MON goes from 1 to 4), while none of the chambers tripped today, even though the peak velocities were almost 1.5x as high. Kiwamu was even able to run his Guoy phase measurements, without losing the IMC, or any other platforms. He did this with when the .03-.1hz blrms were sitting around 10 micron/s.

Given the fact that we can't lock with 1+ micron/s, I think it would be relatively low risk to automate switching the seismic configuration, for the case where some number of site  seismometers are showing 1 micron/s motion. Maybe we could add this to the SEI_CONF manager.

I really doubt we could get to NLN or even ALS during an eq like this and this may be harder to do when the microseism is high. I think this shows that even if we can't lock the IFO, it might still be worthwhile going to an earthquake robust state to reduce the amount of re-alignment needed to recover, once the earthquake is over.

I've been looking at data from the earthquake on Aug 31, in alog 29403. My third figure is the stock lockloss plot from the first lockloss during that eq. Looking at ALS in the "earthquake" and "windy" seismic configurations on Sheila's plot in 29403, ALS sees more .15ish hz noise in the earthquake state. This is because the current earthquake state turns off the low frequency sensor correction and so the .1-.3hz motion gets amplified by the gain peaking of the 250mhz blends on the ISIs. On the BSCs this gets amplified twice, because both stages use 250mhz blends.

One way  to get around that is to use some narrow band sensor correction to reduce the .1-.3hz motion. I've tried this a bit during the earthquake today, and it seemed like it could have been helping, but things were obviously moving a lot. For now, I've added and "Earth_Quake_V2" state to the SEI_CONF that adds .1-.3 sensor correction to the earthquake state. Commissioners and operators should try using this state the next time we get  a ~.5micron/s earthquake.

Had to trend and move back PR2 alignment sliders to be able to find IR. After that I had trouble locking DRMI and PRMI so I ran through an initial alignment. No trouble after that.

Kiwamu ran his Gouy phase measurements in the morning during which time there was a 7.1 mag. earthquake off the coast of New Zealand. The PSL also tripped during his measurements and Peter brought it back. I updated h0vaclx and h0vacly to add beam tube annulus ion pumps. John W. turned up the air conditioning in the control room. Kyle has been periodically turning down the heating on the HAM4 RGA bakeout. Terra has been taking PI measurements. Jenne and Terra are now recovering the IFO from a lock loss.

15:48 UTC Karen opening OSB receiving door
15:54 UTC Chandra to HAM4 to check on RGA bakeout
15:59 UTC Kyle to HAM4 RGA to turn down heating
16:02 UTC Chandra back
16:06 UTC Kiwamu starting Gouy phase measurement. Setting IFO to down.
16:09 UTC Kyle back
16:22 UTC Filiberto to end Y to check on Beckhoff end station 3 chassis
16:42 UTC PSL tripped off, Peter investigating
16:53 UTC Travis, Betsy, Mitchell to LVEA for 3rd IFO inventory
17:15 UTC Richard checking on LVEA access door
17:19 UTC Richard done
17:21 UTC Kiwamu to ISCT6
17:54 UTC Kyle to HAM4 RGA to turn down heating
17:58 UTC Kiwamu back
17:58 UTC Kyle back
18:07 UTC Gerardo to LVEA to check on new beam tube annulus ion pumps
18:34 UTC Travis, Betsy, Mitchell out
19:07 UTC Filiberto heading back
19:21 UTC Kiwamu to ISCT6 to dismantle setup used for Gouy phase measurement
19:29 UTC Kiwamu back, done with Gouy phase measurement
19:37 UTC DAQ restart to add vacuum channels for beam tube annulus ion pumps
19:37 UTC Karen to OSB optics lab to clean
19:44 UTC Filiberto turned back on ITM HV (tripped off from vacuum code restart)
20:09 UTC Karen out of optics lab
20:14 UTC Betsy to LVEA west bay to check serial numbers on parts
20:19 UTC Chandra to mid Y
20:22 UTC Kyle to HAM4 RGA to turn down heating
20:23 UTC Travis to LVEA west bay to help Betsy
20:31 UTC Gerardo to LVEA to check signals of beam tube annulus ion pumps
20:33 UTC Kyle done
20:40 UTC Started initial alignment
21:26 UTC Completed initial alignment
21:34 UTC Travis and Betsy done
22:24 UTC Kyle to mid Y
22:25 UTC Jenne and Sheila to ISCT1
22:44 UTC Jenne and Sheila back

WP 6132

I updated the code from svn to add the new beam tube annulus ion pumps (LX IBM 122, LX OBM 136, LY IBM 192, LY OBM 196). It turned out that I connected the LX annulus ion pumps to the old PT110 channels instead of the old PT170 channels and thus the readings for these are currently wrong (the signal cables were landed on the old PT170 channels). This has been fixed in svn but h0vaclx has to be updated and restarted again at some later opportunity.

As expected the restart tripped the ITM ESD high voltages. Filiberto reset them.

I burtrestored both IOCs to 8 am. Dave added the channels to the DAQ.

I'm leaving the work permit open until h0vaclx is updated and restarted again.

Correction on Sep 1:

Some zero/pole numbers in the original entry was incorrect, so I made some corrections below.

Specifically, ILPD has zp=([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k]) (130 and 2.7k were swapped in the original entry). Plots were updated accordingly.

Summary:

• The outer loop OLTF was measured to be very small.
• That's because the inner loop whitening compensation is NOT implemented in the prototype board. This needs to be implemented.
• (This is implemented in the original (old) second loop board.)

What's done and found:

• Polarity of the board was found to be wrong so I and Fil pulled the chassis and flipped the sign using a jumper.
• After the polarity fix, I measured the OLTF of the outer loop servo with 2W PSL output, AC coupling servo enabled, and the maximum board gain of 45.6dB (first attachment).
  • Don't try to learn too much from this "measurement" as I was saturating the electronics, this is not meant to be a precise measurement. But the point is that this is very, very small.
• Unfortunately it turns out that this is what you get when you don't implement the zero-pole correction that I mentioned in point 2. of alog 29293 and its attachment entry.
  • Gist of it is that the intensity noise sensing gain for the inner loop PD is huge for f>3Hz because of double 70mHz zero, and the outer loop board gain needs to be even bigger in order for the outer loop to have any reasonable gain.
• After the above mentioned measurement, negative leg of the outer loop differential output was found to be grounded to the chassis on the output connector. This is not why the OLTF is very small, but we need to fix it.

These things were mostly done during Tuesday maintenance.

Some more explanation:

Since the outer loop injects its signal to the inner loop error point, OLTF of outer loop is basically the ratio of two transfer functions, one from RIN to the output of the outer loop servo board (H_OLBoard) and the other from RIN to the inner loop error point (H_ILPD):

OLTF_OL=H_OLBoard/H_ILPD.

As I noted before (alog 26879 and an entry attached later to that one), the inner loop diode is equivalent of 660 Ohm DC trans impedance and zp=([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k]) . DC output of this inner loop diode is about 1.6V differential (measured). For details you need to look at D1001998.

H_IL = 1.6[V]*zp([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k]).

OTOH, outer loop trans impedance amplifier doesn't have any coloring in the new prototype (the old one has this compensation, see D1300639), and each diode outputs about 130mV or so at 2W. There are four PDs, so the sum of all PDs at DC is about 520mV at 2W. The outer loop chassis has a flat gain of 3, a variable gain (VGA) with -12.8 to 45.6dB range, and a servo filter SERVO=zp([600;9k],[20,20,380]) with the DC gain of 1 (see alog 29293).  (There should be another factor of 2 from SE to DIFF conversion, but the negative leg of DIFF is grounded, so this is not there at the moment.). There's also a MC pole (9k) in H_OLBoard.

H_OLBoard(2W) = 0.52[V]*3*IMC*VGA*SERVO = RIN*1.6[V]*VGA*zp([600],[20,20,380]).

Combining these,

OLTF_OL @2W = H_OLBoard@2W/H_ILPD ~ zp([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k])/zp(600, [20;20;380])/VGA.

Thing is, H_IL is absolutely huge at 3Hz and higher due to double 70mHz zeros, and the outer loop board as of now doesn't have the gain necessary to keep the UGF at some kHz.

Second attachment bottom shows the above theoretical OLTF at 2W with the maximum (45.6dB) VGA and the measured data. Remember that the "measured" is no good measurement because it was saturating, but nevertheless you can see that the measurement is not totally unreasonable.

It's clear to that we need something in the outer loop to counter 70mHz double zeros.

Implementing that would also help AC coupling. As of now, the AC coupling is already sort of aggressive, but not aggressive enough. If I bring up the blue curve in the second plot bottom by 80dB or so to set the outer loop higher UGF at 2kHz, the lower UGF would become .2Hz or so, that might not be good enough of an AC coupling.

12:37 PDT DAQ restart to add following VAC channels

+[H0:VAC-LX_IBM_II122_AIP_IC_LOGMA]
+[H0:VAC-LX_IBM_II122_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LX_IBM_II122_AIP_IC_MA]
+[H0:VAC-LX_IBM_II122_AIP_IC_MA_ERROR]
+[H0:VAC-LX_IBM_II122_AIP_IC_VOLTS]
+[H0:VAC-LX_IBM_II122_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LX_IBM_VI122_AIP_PRESS_TORR]
+[H0:VAC-LX_IBM_VI122_AIP_PRESS_TORR_ERROR]
+[H0:VAC-LX_OBM_II136_AIP_IC_LOGMA]
+[H0:VAC-LX_OBM_II136_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LX_OBM_II136_AIP_IC_MA]
+[H0:VAC-LX_OBM_II136_AIP_IC_MA_ERROR]
+[H0:VAC-LX_OBM_II136_AIP_IC_VOLTS]
+[H0:VAC-LX_OBM_II136_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LX_OBM_VI136_AIP_PRESS_TORR]
+[H0:VAC-LX_OBM_VI136_AIP_PRESS_TORR_ERROR]
+[H0:VAC-LY_IBM_II192_AIP_IC_LOGMA]
+[H0:VAC-LY_IBM_II192_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LY_IBM_II192_AIP_IC_MA]
+[H0:VAC-LY_IBM_II192_AIP_IC_MA_ERROR]
+[H0:VAC-LY_IBM_II192_AIP_IC_VOLTS]
+[H0:VAC-LY_IBM_II192_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LY_IBM_VI192_AIP_PRESS_TORR]
+[H0:VAC-LY_IBM_VI192_AIP_PRESS_TORR_ERROR]
+[H0:VAC-LY_OBM_II196_AIP_IC_LOGMA]
+[H0:VAC-LY_OBM_II196_AIP_IC_LOGMA_ERROR]
+[H0:VAC-LY_OBM_II196_AIP_IC_MA]
+[H0:VAC-LY_OBM_II196_AIP_IC_MA_ERROR]
+[H0:VAC-LY_OBM_II196_AIP_IC_VOLTS]
+[H0:VAC-LY_OBM_II196_AIP_IC_VOLTS_ERROR]
+[H0:VAC-LY_OBM_VI196_AIP_PRESS_TORR]
+[H0:VAC-LY_OBM_VI196_AIP_PRESS_TORR_ERROR]
 

I was about to take a screen shot of the beams from the various PSL cavities when all of a
sudden the screen went black.  The laser tripped.  I haven't performed any forensics but it
looks like the same problem as last night.

Kyle, Chandra

Starting step down of heating of Vertex RGA -> Closed calibration gas bottle isolation valves and reduced variacs (except one supplying turbo inlet) by 10% -> Will need access to continue this process every few hours or so throughout the day.  

Attached are trends of the various flow sensors in the PSL.  Comparison of the timestamps shows that the high power oscillator
tripped because of a flow rate problem with the AMP circuit.  The problem does not lie within the 4 laser heads, leaving either
the front end power amplifier, or the 4 crystal chambers.

    Unfortunately checking the 4 crystal chambers requires dismantling the laser.  The vortex flow sensor in the AMP circuit
is relatively new.  The other object in that circuit is the power amplifier module.  An inspection mirror might be able to
afford a view of the plumbing underneath the housing.

    One can see that the flow rate in the AMP circuit drops before the output of the front end laser drops, and that the output
power of the front end laser drops before the high power oscillator.  I think the sequence of events is the following:
 1. flow rate problem in the AMP circuit trips the flow watch dog of the front end laser
 2. the switching off of the front end laser breaks the injection locking of the high power oscillator
 3. the loss of injection locking results in a power drop in the high power oscillator which then trips the power watch dog
 4. the power watch dog switches the high power oscillator off

Unsuccessful at damping ITMX 15520 Hz PI several times tonight (previously seen here and here). We find that larger damping drive does not equal greater damping: When this mode was test driven and damped after the thermal transient at 50 W, a best gain and phase was found for damping. When the mode began to ring up later, increasing gain (by some large amound but still under saturation) or flipping sign and/or changing phase only resulted in faster ringup. This is true when still far below DAC saturation levels. It seems as if there is some gain sweet spot that must be found. 

--- --- ---

We had three occasions to damp ITMX 15520 Hz mode during the night. During the first, I successfully damped and it then rerang up (perhaps due to offset adjustments?) and lost lock. During the second and third I was not able to damp the mode and avoided lockloss only by decreasing power 50 W --> 25 W.

Below you see the mode first ring up and my gain trial and error response until I settle at 10000 and the mode is fully damped. Soon after, you see the mode ring up again right after the yaw offset step from ~ -0.02 --> -0.01. Note we started with a small negative gain so I just assumed we had actually been ringing up the mode the past few days.

During the second 50 W lock, damping was already running at the gain and phase settings that were effective at damping the first ring up above (+10000 gain, -60deg). Despite this, you see the mode slowly rising ~3 hours into the 50 W lock and my gain adjustments trying to damp. Note that here I start with some mostly successful positive gain (i.e. shallow slope) yet both raising the gain and flipping the gain sign cause the mode to ring up. I tried phase changes at this point too but existing was best. I avoided lockloss by decreasing power to 25 W, allowing mode to ring down enough to damp, then powering back up. I also rechecked my gain and phase at 50 W (post thermal transient time) and it would still drive and damp with a very steep slope. Still, an hourish later the mode began to ring up. I found similar behavior in the third attempt as the second and had to decrease power to avoid lockloss. 

Things to note: 

• After full lock aquisition, this mode takes ~ 3 hours to ring up. This morning, after dropping to 25 W then coming back to 50 W, it took only ~ 1.5 hours to ring up. Suggests something thermal.
• Q_m of ITMX ~= 18M
• Gap between ESD and ITM is larger than for ETMs ==> less actuation per drive
• Kiwamu has found similar higher gain =/= better damping in violin mode damping

Things to try:

• Retry damping immediately after decreasing power; should be able to see within ~1 min of lower power. 
• Try driving only one quadrant; test spatial overlap of ESD and mechanical mode
• Turn up ring heater (most likely ETMX RH) to decrease frequency overlap between ITMX mech mode and optical mode
• Check offset dependence

Jeff K, Chris Whittle

We used the interometer uptime to test the ITM charge measurement script by taking data for ITMX. With the previously used excitation frequency of 20.1 Hz being used for other measurements, we opted for an excitation at 11.5 Hz. At this frequency, we found an excitation amplitude of 1k counts to be sufficient for a good SNR. We have taken data at bias voltages between -10k and 10k counts, but have yet to perform post-processing.

We additionally tested whether flipping the analog voltage out switches to the quadrants had any effect on the locked interferometer. No effects were observed for flipping on/off quadrant voltages in various orders. This test was performed on just ITMX and with no actual output voltage to the quadrants.

Expecting PT120 to be 3 x 10-9 torr -> Don't see logged activity to explain -> am on road and not able to trend data etc.

The new prototype board was modified to satisfy some of the things in the requirement (T1600064) as listed in 1., 3. and 5. below. I also listed things that were not modified but would have been good if recommendations in T1600064 were implemented.

I wouldn't claim that we won't modify it further, but this is just the current state.

In this entry you'll be able to see the electronics modifications in the attached, but the photos are too small to read the original component values, so read the original drawing D1600298 as necessary.

1. Readback of the PD array signal after the servo loop switch ("SUM PE MON")  needed to be DC coupled.

The "whitening" was originally AC-coupled (second attachment) and therefore it was not compatible with digital AC coupling described in the requirement.

We bypassed the big caps in the input of the opamps with some resistors so it acts as one single pole at 2.7k with DC gain of 50 (second attachment, mod 1).

To make the OLTF measurement easier, we made the same change to the whitening downstream of the summation for the excitation DAC output.

The servo board output readback was also AC-coupled, but we want to see if everything is working fine including DC injected into the inner loop board.

We made the output whitening a true whitening (second attachment, mod 2). 0.35Hz zero might be too low, though.

2. The board doesn't implement the analog compensation for the in the inner loop filtering upstream of the error point.

The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z, p) = ([3; 3; 130], [0.07, 0.07]). T1600064 reccomends to implement the same thing in the outer loop somewhere to cancel this effect to make the loop design simpler.

Since this is not compensated in the prototype board, the outer loop OLTF would become huge for f<3Hz. This means that the digital AC coupling servo should work REALLY hard to effectively AC-couple the entire outer loop at, say, f<1Hz, without reducing the outer loop gain at 10Hz and up.

We do not implement this by analog cut and paste job (no opeamp available for this on the board), but we'll try to make it work by an aggressive digital AC-coupling filter.

3. Integrator as a boost doesn't go well with digital AC coupling so it was converted to a usual boost with finite DC gain.

The error point of the digital AC coupling servo is kept zero at DC (i.e. a pole at zero Hz). The outer loop prototype had an integrator as a boost.

Of course they don't go well as any tiny DC difference between the AC coupling error point and the integrator input will be integrated and eventually the servo runs away. But we don't really need a pole at zero Hz.

We changed it such that the first boost acts as 50Hz LPF with DC gain of 10, and when added to a flat unity gain it becomes 500:50 boost (third attachment) (but see the next entry).

4. Boosts were implemented as three parallel integrators added to a flat unity gain.

We had a choice of unity gain, zp=50:0, 100:0 or 150:0. See the first and the third attachment.

As described in 3. above, we already changed one integrator to []:50 LPF so the boost becomes 500:50.

We didn't fix the parallel summation, it might be OK to go without any boost or using just one.

But T1600064 shows our standard practice of connecting boosts in series.

5. Outer loop servo filters on the board were also changed.

Originally the servo filtering was something like (z,p)=(100, [20^2;40]) (fourth attachment). As described in 2. above, 1st loop effectively adds ([3^2; 130], [0.07^2]). IMC adds another pole at ~8kHz.

Everything added together it's like ([3^2; 100; 130], [0.07^2; 20^2; 40; 8k]), and this might be OK.

But we made a change so the board became ([380;9k],[20^2;600]) by changing one of zero-Ohm resistors in the second stage and one C-R pair in the third stage  (fourth attachment). Everything added together it would be   ([3^2;130;380;9k], [0.07^2; 20^2;600; 8k]).

This change might not have been necessary. We just felt that letting a big 4.7uF capacitor and a zero-Ohm resistor to determine AC gain at around 10kHz in the first as well as the second stage was maybe too much of an uncertainty in the AC gain.

6. Sallen-Key at 0.1Hz for Digital AC coupling path might be too aggressive

For the moment the Sallen-Key is disabled for the digital AC coupling path (so the only analog filtering is a 10Hz LPF) to make things easier as we try to make it work.

Since there seem to be some confusions about which PD has what kind of analog filtering and sent to which channel, here it is.

I'm quite certain about HPO output before AOM (which is "Power monitor PD" in D1002929) and ISS 1st loop monitors, but not that sure about "monitor PD"s in D1002164. E-travellers are incomplete (they don't say which one is installed where), so I'm just listing the nominal values for these "monitor PD"s.