TITLE: Jan 21 EVE Shift 00:00-07:11UTC (16:00-23:11:00 PDT), all times posted in UTC

STATE Of H1: Down

SUPPORT: Sheila,Evan, Gabriele

INCOMING OPERATOR: N/A

SHIFT SUMMARY:

• IFO Locked re-locked for commissioning ≈ 5 times

• Last lock of the night is hopefully going to be a keeper. Sheila and I are going to try to get the SDF cleared for Observing mode. She suggests that the safe.snaps probably need to be updated.

• Ezca Connection error ≈ 07:00UTC caused lockloss when Sheila attempted to clear it.

• No one home at Livingston. We’re checking out (07:11UTC)

We have a lot of out of date safe.snaps, which mean that after our power outage we have a lot of red on sdf, which Ed and I have spent some time fixing but there is still a lot to go.

UPdating safe.snaps would help make the next power outage recovery easier. 

We are going to leave the IFO in down and hope to sort out SDF in the morning. 

[Sheila, Evan, Gabriele]

The goal is to cancel the PRCL length chane induced by driving MICH using only the BS. For this reason we plan to implement a path from MICH control signal to PR2.

1. we injected a MICH motion using a differential drive on the two ITMs, and a PRCL motion using a common drive on the two ITMs. The PRCL error signal (POP_9I) sees the common mode about 20 times more than the differential mode. So we conclude that POP-9I sees MICH about a factoir 20 less than PRCL. It's therefore a good reference to disentangle MICH and PRCL in the BS drive
2. we injected broad band noise in tBS and measured the transfer function to POP_9I. Let's call this transfer function TF_POP_BS. This measures how much PRCL we do when we drive BS
3. we injected broad band noise in PR2 and measured the transfer function to POP_9I. Let's call this transfer function TF_POP_PR2.
4. we fit the ratio TF_POP_BS / TF_POP_PR2 with a 6th order filter (see figure) and implemented this filter in the PR2 ISC input filter banks. We set the driving matrix to send MICH to PR2, and activated the filter

We checked the result with a MICH line at 132 Hz: when the compensation is on, the line amplitude in the PRCL error signal is reduced by at least a factor 10, as expected from the fit residual (figure 2).

We're not leaving this in the guardian.

Mid-SHIFT SUMMARY:

• IFO Locked at NLN 78Mpc 02:57UTC

• Commissioners working 54Mpc

• SDF is LIT up Red. Sheila mentioned that it probably hasn’t been cleared since the power outage?

• µSei Z-axis is riding above the 90%ile. EQ bands are ≈.08µm/s. Winds are ≤15mph

STATE OF H1: Relocking
SHIFT SUMMARY: Moved RM1 and RM2. Ran initial alignment. Obtained first lock at NLN since power outage. Lost lock shortly afterwards to a 6.6 magnitude earthquake in Mexico (see attached). Earthquake took about 5 hours to ring down. Manually damped ITMX and ITMY roll modes.
SUPPORT: Jenne, Evan, Sheila
INCOMING OPERATOR: Ed

ACTIVITY LOG:
17:02 UTC Bubba and Chris into end X mechanical room to lubricate fans (WP 5685)
17:45 UTC Bubba back from from end X
18:47 UTC Bubba and Chris to end Y mechanical room to lubricate fans (WP 5685)
19:18 UTC Rick and Liyuan to LVEA and optics lab to look for optics
20:11 UTC Rick and Liyuan done
20:30 UTC Filiberto to Y2-8 to check on solar powered battery charge
21:05 UTC Kyle to end X air handling room to take measurements for building a stand for the ion pump
22:08 UTC Filiberto done at Y2-8 and going to X2-8
22:19 UTC Kyle back
22:21 UTC Dick G. to LVEA to look at RF racks
22:53 UTC Jeff B. to optics lab to work on dust monitors
22:59 UTC Filiberto done at X2-8
23:04 UTC Nutsinee and Jim B. to end Y VEA to turn on HWS
23:19 UTC The tidal common length integrator limits (H1:LSC-X_COMM_CTRL_LIMIT, H1:LSC-Y_COMM_CTRL_LIMIT) were being reached. Evan had me change them from 10 to 20.
23:27 UTC Nutsinee and Jim B. back
23:30 UTC Dick G. back
23:33 UTC Jeff B. back. He turned off the dust monitors in the optics labs.
00:34 UTC Kyle to mid Y to overfill CP3

1635 - 1650 hrs. local -> Back and forth Y-mid 

Next scheduled over fill Saturday, Jan. 23rd

Jim B, Nutsinee

Yesterday I managed to restart ETMX HWS software but wasn't able to connect to EY HWS camera via telnet (this camera is plugged in to an external power supply). Today Jim and I went out, removed the relay box for troubleshooting, and plugged EY HWS camera directly to the power supply (no way of powering it on and off remotely at the moment). I trended  the cylindrical power of both EX and EY HWS to see if the software had been running before the power outage yesterday. None of them had been running since the past 10 days. ETMY cylindrical power trend shows that the software in fact hasn't been running since the lost power outage in October (I couldn't get ETMX data). Since nobody noticed this and no one ever told me that it wasn't running, I hope it wasn't needed.

TITLE: Jan 19 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Locking

DAY OPERATOR: Patrick

BRIEF SUMMARY: Recovering from a very large earthquake off the coast of Western Mexico. Currently the IFO bounce and roll modes are being monitored and manually damped. This appears to be the reason for lock losses around the RF_DARM stage.

Plot attached shows TMSX and TMSY damping signals before and after the power outage, and TMSX signals are significantly more noisy after the power outage.

model restarts logged for Wed 20/Jan/2016

Power outage at 07:45 PST. Many systems were restarted, full log file attached.

model restarts logged for Tue 19/Jan/2016

No restarts reported

model restarts logged for Mon 18/Jan/2016

No restarts reported

The LHO IM (IM1, IM2, IM3, IM4) alignments change, for the same alignment offset (drive), after a HAM2 ISI trip, or like yesterday, a power outage.

IM2's algnment changes the most, so I have only include it in this alog.

The first attachment is a chart that shows how the IM2 alignment changed, and how much IM2 and the HAM2 ISI shook during this event, summary here:

The second attachment is the full data trends for these channels (not in the same order) during the shaking.

At LHO, it is a requirment to check IM alignments after a HAM2 ISI trip, because these alignment shifts regularly occur after the optics experince the shaking.

Other alignment change / shaking events to be posted.

It would be helpfull to have SDF files for multiple interferometer states, to help us diagnose configuration changes when we aren't locked.  Since we just lost lock due to an EQ, we know that at least for some ISC and non tripped suspensions we should be in the normal down states.  

I've saved down.snaps for LSC, LSCAUX, ASC, ASCIMC (has some diffs due to a script that sets values with too much precision), OMC, ALSEY, ALSEX, ISCEX, ISCEY,SUSETMX, SUSETMY

Now we can start using these as a reference or try to implement Staurt's script to have the guardian load them.

tl;dr: Alignment of IMs and RMs should be checked, then re-run an initial alignment.

Xarm green alignment

Ed was struggling with the Xarm green alignment.  We set the guardian to locked no slow, no wfs.  I then turned on the loops one at a time.  Usually the camera centering loops are the problem, but they were the easy ones this time.  Eventually it was DOF2 that was causing trouble, so I had DOFs 1 and 3 closed and touched TMS by hand to get the error signals for DOF2 closer to zero.  I was able to close all the loops, and let the alignment run like normal after that.

Xarm IR alignment

Not really sure what the problem is here, but it's getting late and I'm getting frustrated, so I'm going to see if I can move on with just hand aligning the input. 

I suspect that the IMs need some more attention, so if Cheryl (or someone else following Cheryl's procedures) could check on those again in the morning, that would be great. 

Also, I'm not sure if the RMs got any attention today, but the DC centering servos are struggling.  I've increased the limits on DC servos 1 and 2, both pitch and yaw (they used to all be 555, now they're all 2000).  I also increased H1:SUS-RM2_M1_LOCK_Y_LIMIT from 1000 to 2000. 

Allowing the INP ASC loops to come on is consistently causing the arm power to decay from 0.85ish to lockloss. I didn't touch the ITM or ETM, but I'm not getting any more power by adjusting PR2 or IM4. 

MICH Dark not locking.  Discovered that BS's M2 EUL2OSEM matrix wasn't loaded, so no signal being sent out.  Hit load, MICH locked, moved on.

Bounce needed hand damping (guardian will prompt you in DARM_WFS state), roll will probably need it too.  This isn't surprising, since it happens whenever the ISI for a quad optic trips.  Recall that you can find the final gains (and the signs) for all of these in the ISC_LOCK guardian.  I like to start with something small-ish, and increase the gain as the mode damps away.  No filters need to be changed, just the gains.  My starting values are in the table below, and I usually go up by factors of 2 or 3 at a time. 

I lost lock while damping the bounce mode (in DARM_WFS state), and the DRMI alignment is coming back much worse than the first few DRMI locks I had.  

I don't actually have a lot of faith in my input beam alignment, so I probably wouldn't be happy with any ASC loop measurements I take tonight even if I got the IFO locked.  Since we have an 8am meeting, I'm going to call it a night, and ask the morning operator to check my alignment and fix anything that sleepy-Jenne messed up.

STATE OF H1: Still recovering from power outage.

ACTIVITY LOG (some missing):
15:44 UTC Unexpected power outage
16:38 UTC Fire department through gate to check on RFAR boxes
16:56 UTC Richard, Jeff B. and Jim W. to end stations to turn on high voltage and HEPI pumps
17:14 UTC Richard turning on h1ecaty1
17:14 UTC Jason and Peter to LVEA to look for an optic
17:39 UTC Richard and company done at end X and going to corner station to turn on TCS chillers and HEPI pumps
17:49 UTC Filiberto to end X to reset fire panel
17:56 UTC Vacuum group touring LVEA (6 or 7 people)
17:57 UTC HEPI pump stations and TCS chillers started in corner station
18:18 UTC Filiberto back from end X
18:24 UTC Jeff B. and Jason to LVEA to start TCS X laser
18:49 UTC Hugh bringing up and isolating HAM ISIs
19:16 UTC Richard and Jim W. to end stations to turn on ALS lasers
19:30 UTC Sheila to CER to power cycle all of the Beckhoff chassis
22:55 UTC Jason to LVEA to look at optical levers
23:02 UTC Jason back
23:34 UTC Dave restarting the EPICS gateway between the slow controls network and the frontend network in an attempt to fix an issue with the Beckhoff SDF
23:47 UTC Dave restarting the Beckhoff SDF code

Other notes:
End stations: end Y IRIG B chassis power cycled; end X, end Y high voltage turned on; end X, end Y HEPI pump station computers started; end X, end Y ISI coil drivers chassis reset; end Y Beckhoff computer turned on
Corner station: TCS chillers turned on; HEPI pump controllers turned on (Jeff B. had to push power button on distribution box); Sheila power cycled all Beckhoff chassis in CER; Sheila turned on the AOS baffle PD chassis in LVEA

I started the h0video IOC on h0epics2.

Conlog was running upon arrival. However it crashed during the recovery. It still needs to be brought back.

Joe D. and Chris worked on beam tube enclosure sealing.

There was some trouble starting the corner station HEPI pump controller computer.

There was some trouble finding the strip tool template for the wind speeds. I'm not sure if it was found or if TJ created a new one.

Things to add to the 'short power outage recovery' document:
- Turn on high voltage supplies
- Push reset on ISI coil drivers
- HEPI pump controller and pumps
- TCS chillers
- TCS lasers
- ALS lasers
- Turn on video server

We have begun recovery from an unexpected power outage that occurred at 7:44 PST and lasted for 3 minutes (as noted from the UPS).

Summary and discussion

Radar plots of the sensing matrix for PRCL, MICH, and SRCL are attached (where MICH is the usual beamsplitter drive). This was taken in the nominal O1 configuration.

Something is strange with the response in POP9; driving beamsplitter seems to result in the same response as driving PRM. Certainly we expect some amount of POP9I response from driving the beamsplitter, since this drives PRCL, but it seems strange that they are almost exactly equal.

For POP45, beamsplitter appears in both I and Q; again, this is not surprising. However, its contribution in I seems to dominate over the SRM contribution. This could potentially be bad, because we only do PRCL → SRCL subtraction (not MICH → SRCL subtraction).

Sensing matrix is as follows:

There is some ambiguity in the signs here that still needs to be resolved.

The magnitudes of the PRM ⇝ POP9I and SRM ⇝ POP45I elements agree fairly well with what is necessary to explain the open-loop transfer functions for PRCL and SRCL (3.6 W/µm and 0.13 W/µm, respectively). For MICH, there is some frequency-dependent residual between the model and the measurement (which has been observed before). The gain needed to make them match around the UGF is 0.6 W/µm.

Details

I added a 132.1 Hz notch to the PRCL, MICH, and SRCL SFMs. Then one by one I drove PRM, SRM, and BS with a 132.1 Hz line for a few minutes, using the digital oscillator just after the LSC SFMs.

I took the time series for POP9I/Q and POP45I/Q during this time, and demodulated them with 0.1 Hz low-passing.

All times are 2015­­–01–13 Z. The actuator calibrations are based on digital filters, the suspension models, and the suspension electronics.

• PRM:
  • Oscillator amplitude: 1000 ct
  • Injection time: 16:19:40 to 16:23:40
  • Calibration: −3.0×10⁻¹⁶ m/ct
• SRM:
  • Oscillator amplitude: 2000 ct
  • Injection time: 16:25:35 to 16:30:35
  • Calibration: −3.0×10⁻¹⁷ m/ct
• BS:
  • Oscillator amplitude: 20 ct
  • Injection time: 16:33:05 to 16:37:05
  • Calibration: 3.8×10⁻¹⁶ m/ct (including digital f² inversion filters)

As for sensors, the calibrations are 2.2×10⁸ ct/W for POP9 and and 2.3×10⁸ ct/W for POP45, based on LHO#24959 plus an additional 30 dB of whitening gain.

**Short version: Increased RY input motion (maybe HEPI, maybe wind/ground) causes ISI X loops to ring up when running 45mhz blends. The suspension/tidal is not the cause. The 90mhz blends seem to be immune to this. Other than using 90mhz blends, I'm not sure how to fix the ISI's configuration, short term to prevent the ISI from ringing up. But we should put a StripTool of the end station ISI St1 CPS locationmons somewhere in the control room so operators can see when ground tilt has rung up an ISI. Alternatively, we could add a notification to VerbalAlarms or the DIAG node when an ISI has been moving something like 10 microns peak to peak for several minutes.

This morning while the IFO was down for maintenance, Evan and I looked at ETMX to see if we could figure out what is causing the ISI to ring up. First we tried driving the L1 stage of the quad to see if some tidal or suspension drive was the cause. This did not have on the ISI, so I tried driving on HEPI. When I drove HEPI X, the ISI rang up a bit, but no more than expected with the gain peaking of the 45mhz blends. When I drove HEPI in RY, however, the ISI immediately rang up in X, and continued to ring for several minutes after I turned the excitation off. The attached image shows the ISI CPS X(red), RY (blue), HEPI IPS RY(green) and X (magenta). The excitation is visible in the left middle of the green trace, also visible in the sudden increase in the red trace. I only ran the excitation for 300 seconds (from about 1134243600 to 1134243900), but the ISI rang for twice that. After the ISI settled down I switched the blends to the 90mhz blends and drove HEPI RY again. The ISI moved more in X but it never rang up even after I increaed the drive by a factor of 5. The second plot shows the whole time series, same color key. The large CPS X motion (with barely noticeable increase in the IPS RY) is the oscillation with the 45mhz blend , the larger signal on the IPS RY (with small increase in CPS X) is with 90mhz blends. The filter I used for each excitation was zpk([0 0],[ .01 .01 .05 .05], 15111).