Jenne, Sheila, Patrick

While at ENGAGE_REFL_POP_WFS we took the ISI configuration state from windy to earthquake. This made the cavity power buildup signals noticeably worse. We therefore took the ISI configuration state back to windy.

This occurred around 18:12 UTC.

   These are the 7 day trends for the OpLevs. FAMIS Task #4691. 

The pump diode current readout for diode box 3 has been mis-behaving for a while now.  This time it was caught in the
act.  The set diode current is 50.2 A.  The readout is clearly wrong, not least of which the power supplies cannot
deliver more than ~65 A but also the laser power does not vary during this time.

    Might be an intermittent fault with the Beckhoff terminal that reads the output of the power supply.

Please when the laser trips, disable the locking for the PMC, FSS, and ISS.  It doesn't make
much sense to keep ramping the NPRO PZT when there's no chance of any of the cavities acquiring lock.

CDS:
Continuation of framewriter debugging
Tues: reinstall common mode board

SEI:
Tues: offload BS HEPI DC bias to large springs

VAC:
h0vaclx restarted to fix annulus ion pump channels. Richard reset ITM ESD high voltage.
Kyle turning off pumps

When CO2Y laser had an issue (alog2856) I added a 1 degree offset to the rotation stage calibration to trick it to deliver the right amount of power. That offset has been removed and the CO2Y RS now delivers the right power again.

For higher accuracy the RS random walk script has to be run.

Attached is a capture of the PSL's TwinCAT status screen.  Same problem as previously, namely a flow error in the
AMP cooling circuit.

    In bring the laser back up, I took the opportunity to reboot h1pslctrl0.  This has temporarily fixed the problem
with the diode current readout of diode box 3.

[Sheila, Jenne, Kiwamu, Terra]

We have reset the initial alignment offsets, including realigning several paths on ISCT1, to match the alignment that we liked from yesterday (alog 29430).  So far, we think that it's good.  We have powered up part way twice since, and the power recycling gain no longer drops very much!  It still drops a bit, so we could consider going a bit farther, but this is probably plenty good to move on and focus entirely on low noise efforts.  Now at 40W it is something like 31, rather than something in the mid/low twenties.  Plots coming tomorrow.

In order to do this, we did the following (hardware alignment changes noted in bold):

• Lock using old alignment, at DC Readout move most of the way to the new alignment using PR2 sliders and POP_A offsets.
• Relieve the POP_X centering loop PZT on ISCT1 by *very slowly* moving the knobs on the steering mirror that goes to POP_X.  We were able to do this while locked, and PRC2 was using POP_X in-loop. 
• We then tried to power up with the new alignment to ensure that it was good.  It seemed pretty good (PRC gain stayed roughly constant), but we lost lock due to dPdTheta. 
• For our next lock, we had to lower the threshold at which the POP_X centering loop triggers on, since the beam was partly off the WFS.  (spoiler alert: once we finished all alignment work for the night, this is no longer necessary.  POPX is already quite centered before the centering loop comes on.)
• Once we were relocked with our PR2 slider motion and POPA offsets in place, we set the green input pointing QPD offsets such that the green beams were maximally co-aligned with the IR in the arms, and the green WFS signals were all zero.  This was done at DC Readout, 2W.  We also reset the green ITM camera setpoints.
  • We lost lock while doing this, but the Yarm was already complete.  Sheila did a test InitialAlignment of just the green Yarm and saw that the test masses didn't move according to their oplevs from where we wanted them by more than a few tenths of a microradian.
  • For the Xarm, we put the test masses back to the locations that the oplevs said they wanted to be from the 2W DC Readout lock, and then finished setting the QPD and camera offsets.  The test masses had to be moved by several microradians each in both pitch and yaw to get the unlocked mirrors pointed the same way that they were while being controlled by the ASC loops.
• We centered the green beams on ISCT1 on the green transmission PDs and on the transmission cameras.
• We put PR3 and BS back to the places that their oplevs said they were during our 2W DC Readout lock, then aligned both the ALS Comm and ALS Diff beatnote PDs.
• TJ did an initial alignment for us using the new set points.
• We aligned POPAIR_A and POPAIR_B, although neither was very far off.
• We checked that the DRMI ASC works using the POP_A offsets that we like.  This is now in the guardian, but at some point we should picomotor to recenter POPA in this new place.
  • PRC1 and SRC1 and SRC2 are once again turned on in the DRMI ASC guardian.
  • This makes CHARD come on much more nicely (although we have still had a few locklosses trying to engage these loops).  The gain has been redistributed in the guardian so it also comes on more slowly, but it's apparently still not perfect yet.  At least engaging CHARD makes the recycling gain go up rather than down, which it had been doing until this afternoon (the recycling gain would come back once the PRC1 loop was closed around POPA).
• On our first power up with the new alignment, we lost lock due to the dPdTheta instability.  Kiwamu measured the suppression at the resonance frequency, and increased the gain of the ISS 3rd loop by a factor of 3.
  • Upon our next power up, it seems as though we can't increase the power quite as fast, without causing trouble.  Tomorrow we will try with the ISS 3rd loop gain only up by a factor of 2, and we will also try increasing the power more slowly.
  • Sheila tweaked the CSOFT pitch loop shape so that we no longer have a marginally stable point around 0.45 Hz - this will allow us to increase the gain such that we can help hold against the instability.
• On our next power up, we lost lock with an ISS saturation.  We think this is due to the 3rd loop gain being too high / increasing the power too fast.
• A few minutes later, the PSL shut off.  Commissioning to be continued tomorrow when more photons are available.

8:13 UTC

TITLE: 09/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Sep
SHIFT SUMMARY: I arrived at the tail end of the 7.2 EQ in New Zealand, so commissioning efforts were just starting again. They continued throughout the night, working hard as always.
 

On Tuesday we reconfigured h1fw1 to run the newer code (same code as h1fw0), and put the older code on h1fw2 for comparison purposes.

So:
h1fw0 and h1fw1 are running r4252 from branch-3.0 in svn.
h1fw2 is running as the advLigoRTS-3.0.3 release tagged for O1.

We have not seen frame writer crashes in several weeks now.  However we are seeing h1fw0 produce a small number of frames that are different from what is produced by h1fw1 and h1fw2.

With analysis done by Greg Mendell and David Barker it appears that the daqd does not receive all the data occasionally and issues some re transmit requests for some of the data.  When it does this some of the data is put in out of order.

So apparently data that should arrive in as

1 2 3 4 5

may be stored as:

1 3 4 5 2

I will review the network receive code tomorrow to see if this is what is happening.  We find it curious that it inserts the data it skips later on.

We are not sure what is causing this, but a leading theory is that this may be a network issue between the h1dc0 and h1fw0.  Next Tuesday Dave plans to switch the connections on the switch between h1fw0 and h1fw1.  If the behavior switches between h1fw0 and h1fw1 we will be able to attribute this to a problem with either the port, sfp, or fiber that is currently connecting h1fw0 to the switch.

I will work with Carlos to see if we can get error counters from the switch.

The plot shows that an alignment study last night changed jitter coupling reproducibly. Notice that the different peaks from the PSL seem to change differently, so it seems more complex than something like clipping.

Elli (remotely), Kiwamu,

Today was another day for the SRC gouy phase measurement. Even though I said that I would try out another misalignment configuration (29418), we decided to repeat the same measuremnet/configuration as yesterday with an improved clipping situation.

We will stop doing the measurement for now until we hear from Elli on her offline analysis. Done for now.

[The measurements]

• measurement #1
  •  18:26:27 - 18:36:27 UTC
  • BS yaw excitation with amplitude of 6 urad at 0.2 Hz
• measurement #2
  • 18:39:02 - 18:49:02 UTC
  • BS yaw excitation with amplitude of 3 urad at 0.2 Hz
• measurement #3
  • 18:52:34 - 19:02:34 UTC
  • PR2 yaw excitation with amplitude of 20 urad at 0.2 Hz
• measurement #4
  • 19:07:15 - 19:17:15 UTC
  • PR2 yaw excitation with amplitude of 10 urad at 0.2 Hz

[The setup]

• PSL = 22.4 W
• CO2Y = 286 mW
• ITMY ring heater = 0.5 W
• ifo configuration = single bounce (with ITMY aligned) + slightly misaligned SRM
• There was a large earthquake about 1 hour before I started the measurements. The spot positions on the camera seemed to move more than usual below 0.1 Hz.
• Together with SRM misalignment, I iteratively introduced misalignment on BS and PR2 as well in order to get less clipping on the spot on ASAIR.
• See the alignment sliders in the first and second attachments.
• The clipping on the spot on ASAIR was improved. See the screenshot below. The picture on the right is from yesterday and the picture on the left is from this morning.

[The data]

All the relevant data is available at kiwamu.izumi/Public/measurements/20160831_SRCgouy2/data2

Still thinking about earthquakes, I have 2 main points:

1. We can reduce the number of times ISIs trip during earthquakes by reducing the amount of inertial isolation we do a low frequency. Even if we can't lock, changing seismic configurations may save time in recovery.

2. The current earthquake configuration amplifies microseism (.1-.3hz motion) by a factor of a couple. I have some thoughts on improving that, but we need to balance not making things worse at .1-.3 hz (for the summer, we'll need to do better come November) and rolling off the seismometers quickly below .1hz. I've made a Earth_Quake_V2 to address this, but theres more work here to do.

This morning New Zealand had a pretty big earthquake, with peak .03-.1hz blrms velocties around 10 micron/s at LHO.  We noticed this before the velocities reached around 1 micron/s, so I moved the SEI_CONF manager to the Earth_Quake state. Looking at ground and WD trends for the ISI's over the last two months, it seems that normally the BSC ISIs trip at 1-2 micron/s, and today all of the platforms survived. My first plot shows the ITMY .03-.1hz BLRMS, ITMY ST2 WD MON bit and SEI_CONF state (40 is the "windy" state, 17 is the "earthquake" state) for the earthquake today, the second is a plot of an earthquake from Aug 29th. The earthquake on Aug 29th took out all of test mass ISIs (WD_MON goes from 1 to 4), while none of the chambers tripped today, even though the peak velocities were almost 1.5x as high. Kiwamu was even able to run his Guoy phase measurements, without losing the IMC, or any other platforms. He did this with when the .03-.1hz blrms were sitting around 10 micron/s.

Given the fact that we can't lock with 1+ micron/s, I think it would be relatively low risk to automate switching the seismic configuration, for the case where some number of site  seismometers are showing 1 micron/s motion. Maybe we could add this to the SEI_CONF manager.

I really doubt we could get to NLN or even ALS during an eq like this and this may be harder to do when the microseism is high. I think this shows that even if we can't lock the IFO, it might still be worthwhile going to an earthquake robust state to reduce the amount of re-alignment needed to recover, once the earthquake is over.

I've been looking at data from the earthquake on Aug 31, in alog 29403. My third figure is the stock lockloss plot from the first lockloss during that eq. Looking at ALS in the "earthquake" and "windy" seismic configurations on Sheila's plot in 29403, ALS sees more .15ish hz noise in the earthquake state. This is because the current earthquake state turns off the low frequency sensor correction and so the .1-.3hz motion gets amplified by the gain peaking of the 250mhz blends on the ISIs. On the BSCs this gets amplified twice, because both stages use 250mhz blends.

One way  to get around that is to use some narrow band sensor correction to reduce the .1-.3hz motion. I've tried this a bit during the earthquake today, and it seemed like it could have been helping, but things were obviously moving a lot. For now, I've added and "Earth_Quake_V2" state to the SEI_CONF that adds .1-.3 sensor correction to the earthquake state. Commissioners and operators should try using this state the next time we get  a ~.5micron/s earthquake.

Had to trend and move back PR2 alignment sliders to be able to find IR. After that I had trouble locking DRMI and PRMI so I ran through an initial alignment. No trouble after that.

Kiwamu ran his Gouy phase measurements in the morning during which time there was a 7.1 mag. earthquake off the coast of New Zealand. The PSL also tripped during his measurements and Peter brought it back. I updated h0vaclx and h0vacly to add beam tube annulus ion pumps. John W. turned up the air conditioning in the control room. Kyle has been periodically turning down the heating on the HAM4 RGA bakeout. Terra has been taking PI measurements. Jenne and Terra are now recovering the IFO from a lock loss.

15:48 UTC Karen opening OSB receiving door
15:54 UTC Chandra to HAM4 to check on RGA bakeout
15:59 UTC Kyle to HAM4 RGA to turn down heating
16:02 UTC Chandra back
16:06 UTC Kiwamu starting Gouy phase measurement. Setting IFO to down.
16:09 UTC Kyle back
16:22 UTC Filiberto to end Y to check on Beckhoff end station 3 chassis
16:42 UTC PSL tripped off, Peter investigating
16:53 UTC Travis, Betsy, Mitchell to LVEA for 3rd IFO inventory
17:15 UTC Richard checking on LVEA access door
17:19 UTC Richard done
17:21 UTC Kiwamu to ISCT6
17:54 UTC Kyle to HAM4 RGA to turn down heating
17:58 UTC Kiwamu back
17:58 UTC Kyle back
18:07 UTC Gerardo to LVEA to check on new beam tube annulus ion pumps
18:34 UTC Travis, Betsy, Mitchell out
19:07 UTC Filiberto heading back
19:21 UTC Kiwamu to ISCT6 to dismantle setup used for Gouy phase measurement
19:29 UTC Kiwamu back, done with Gouy phase measurement
19:37 UTC DAQ restart to add vacuum channels for beam tube annulus ion pumps
19:37 UTC Karen to OSB optics lab to clean
19:44 UTC Filiberto turned back on ITM HV (tripped off from vacuum code restart)
20:09 UTC Karen out of optics lab
20:14 UTC Betsy to LVEA west bay to check serial numbers on parts
20:19 UTC Chandra to mid Y
20:22 UTC Kyle to HAM4 RGA to turn down heating
20:23 UTC Travis to LVEA west bay to help Betsy
20:31 UTC Gerardo to LVEA to check signals of beam tube annulus ion pumps
20:33 UTC Kyle done
20:40 UTC Started initial alignment
21:26 UTC Completed initial alignment
21:34 UTC Travis and Betsy done
22:24 UTC Kyle to mid Y
22:25 UTC Jenne and Sheila to ISCT1
22:44 UTC Jenne and Sheila back

WP 6132

I updated the code from svn to add the new beam tube annulus ion pumps (LX IBM 122, LX OBM 136, LY IBM 192, LY OBM 196). It turned out that I connected the LX annulus ion pumps to the old PT110 channels instead of the old PT170 channels and thus the readings for these are currently wrong (the signal cables were landed on the old PT170 channels). This has been fixed in svn but h0vaclx has to be updated and restarted again at some later opportunity.

As expected the restart tripped the ITM ESD high voltages. Filiberto reset them.

I burtrestored both IOCs to 8 am. Dave added the channels to the DAQ.

I'm leaving the work permit open until h0vaclx is updated and restarted again.

Unsuccessful at damping ITMX 15520 Hz PI several times tonight (previously seen here and here). We find that larger damping drive does not equal greater damping: When this mode was test driven and damped after the thermal transient at 50 W, a best gain and phase was found for damping. When the mode began to ring up later, increasing gain (by some large amound but still under saturation) or flipping sign and/or changing phase only resulted in faster ringup. This is true when still far below DAC saturation levels. It seems as if there is some gain sweet spot that must be found. 

--- --- ---

We had three occasions to damp ITMX 15520 Hz mode during the night. During the first, I successfully damped and it then rerang up (perhaps due to offset adjustments?) and lost lock. During the second and third I was not able to damp the mode and avoided lockloss only by decreasing power 50 W --> 25 W.

Below you see the mode first ring up and my gain trial and error response until I settle at 10000 and the mode is fully damped. Soon after, you see the mode ring up again right after the yaw offset step from ~ -0.02 --> -0.01. Note we started with a small negative gain so I just assumed we had actually been ringing up the mode the past few days.

During the second 50 W lock, damping was already running at the gain and phase settings that were effective at damping the first ring up above (+10000 gain, -60deg). Despite this, you see the mode slowly rising ~3 hours into the 50 W lock and my gain adjustments trying to damp. Note that here I start with some mostly successful positive gain (i.e. shallow slope) yet both raising the gain and flipping the gain sign cause the mode to ring up. I tried phase changes at this point too but existing was best. I avoided lockloss by decreasing power to 25 W, allowing mode to ring down enough to damp, then powering back up. I also rechecked my gain and phase at 50 W (post thermal transient time) and it would still drive and damp with a very steep slope. Still, an hourish later the mode began to ring up. I found similar behavior in the third attempt as the second and had to decrease power to avoid lockloss. 

Things to note: 

• After full lock aquisition, this mode takes ~ 3 hours to ring up. This morning, after dropping to 25 W then coming back to 50 W, it took only ~ 1.5 hours to ring up. Suggests something thermal.
• Q_m of ITMX ~= 18M
• Gap between ESD and ITM is larger than for ETMs ==> less actuation per drive
• Kiwamu has found similar higher gain =/= better damping in violin mode damping

Things to try:

• Retry damping immediately after decreasing power; should be able to see within ~1 min of lower power. 
• Try driving only one quadrant; test spatial overlap of ESD and mechanical mode
• Turn up ring heater (most likely ETMX RH) to decrease frequency overlap between ITMX mech mode and optical mode
• Check offset dependence

S. Dwyer, J. Kissel, C. Gray

After successfully recovering the IMC's VCO and recovering the IMC (29264), we were able to get up through LOCKING_ARMS_GREEN in the lock acquisition sequence. However, we found that ALS COMM failed caused lock losses during the next step (LOCKING_ALS), when the input for IMC length control in its Common Mode Chassis was switched from the IMC's PDH output to the ALS COMM PLL output. The ALS COMM PLL output is connected to IN2 of the IMC chassis that had a new daughter board installed in the star-crossed ISC rack H1-ISC-R1 today (LHO aLOG 29250). After fighting through MEDM screen confusion* at the racks, we found that OUT2 (an analog pickoff pick-off just after the input gain circuit) indicated a ~2.5 [V] offset, even with IN2 terminated with 50 [Ohms]. 

Suspecting that this symptom was indicative that the input gain circuit (circled in red in MEDM screen capture) was yet another causality of the unfortunate rack power mishap today (LHO aLOG 29253), we've replaced the entire chassis (which lives in U14 of H1-ISC-R1) with a spare we found in the EE shop -- S/N S1102627 (or Board S/N S1102627MC). 

Notably, this spare does not have one of the new daughter boards on which Chris has worked so hard.

We're not suggesting this swap be permanent, but we make the swap for tonight at least, so we can hopefully make forward progress. We suggest that IN2 and/or the input gain stage of SN S1102626 be fixed tomorrow, and the chassis restored so we can employ the new daughter board.

Other Details:
- Before removing the chassis, we powered down the entire rack using the voltage sequencer around the back at the top of the rack. 
- After installing the rack, we were sure to have all cables connected appropriately before turning the rack power on again (via the sequencer again).
- We added a few labels to the IMC's PDH output and the ALS COMM PLL output cables such that they're easier to follow and reconnect in the future.

*MEDM Screen Confusion -- whether IN1 or IN2 is fed into OUT2 of all common mode chassis is selectable on their MEDM screens. For the IMC's common mode board (at least for SN S1102626), the MEDM screen's indication of the status of that switch is exactly backwards. When the screen indicates that IN1 is feeding OUT2, IN2 is feeding OUT2, and vice versa. #facepalm