Plot attached shows TMSX and TMSY damping signals before and after the power outage, and TMSX signals are significantly more noisy after the power outage.

model restarts logged for Wed 20/Jan/2016

Power outage at 07:45 PST. Many systems were restarted, full log file attached.

model restarts logged for Tue 19/Jan/2016

No restarts reported

model restarts logged for Mon 18/Jan/2016

No restarts reported

The LHO IM (IM1, IM2, IM3, IM4) alignments change, for the same alignment offset (drive), after a HAM2 ISI trip, or like yesterday, a power outage.

IM2's algnment changes the most, so I have only include it in this alog.

The first attachment is a chart that shows how the IM2 alignment changed, and how much IM2 and the HAM2 ISI shook during this event, summary here:

The second attachment is the full data trends for these channels (not in the same order) during the shaking.

At LHO, it is a requirment to check IM alignments after a HAM2 ISI trip, because these alignment shifts regularly occur after the optics experince the shaking.

Other alignment change / shaking events to be posted.

It would be helpfull to have SDF files for multiple interferometer states, to help us diagnose configuration changes when we aren't locked.  Since we just lost lock due to an EQ, we know that at least for some ISC and non tripped suspensions we should be in the normal down states.  

I've saved down.snaps for LSC, LSCAUX, ASC, ASCIMC (has some diffs due to a script that sets values with too much precision), OMC, ALSEY, ALSEX, ISCEX, ISCEY,SUSETMX, SUSETMY

Now we can start using these as a reference or try to implement Staurt's script to have the guardian load them.

Added 100 mL of H2O to the H1 PSL crystal chiller.

After moving RM1 and RM2, I performed an initial alignment and took the IFO to NLN. The lock lasted a couple of minutes before being taken out by a large earthquake.

DRMI 1F locked, then dipped in power, and recovered (see attached screenshot).

I trended back the positions of RM1 and RM2 for the last 3 days (see attached). The differences appeared significant, so I used the alignment sliders to bring them back.

RM1:
Pitch: 380 -> 416
Yaw: 632 -> 331

RM2:
Pitch: -1149 -> -856
Yaw: 2664 -> 1306

tl;dr: Alignment of IMs and RMs should be checked, then re-run an initial alignment.

Xarm green alignment

Ed was struggling with the Xarm green alignment.  We set the guardian to locked no slow, no wfs.  I then turned on the loops one at a time.  Usually the camera centering loops are the problem, but they were the easy ones this time.  Eventually it was DOF2 that was causing trouble, so I had DOFs 1 and 3 closed and touched TMS by hand to get the error signals for DOF2 closer to zero.  I was able to close all the loops, and let the alignment run like normal after that.

Xarm IR alignment

Not really sure what the problem is here, but it's getting late and I'm getting frustrated, so I'm going to see if I can move on with just hand aligning the input. 

I suspect that the IMs need some more attention, so if Cheryl (or someone else following Cheryl's procedures) could check on those again in the morning, that would be great. 

Also, I'm not sure if the RMs got any attention today, but the DC centering servos are struggling.  I've increased the limits on DC servos 1 and 2, both pitch and yaw (they used to all be 555, now they're all 2000).  I also increased H1:SUS-RM2_M1_LOCK_Y_LIMIT from 1000 to 2000. 

Allowing the INP ASC loops to come on is consistently causing the arm power to decay from 0.85ish to lockloss. I didn't touch the ITM or ETM, but I'm not getting any more power by adjusting PR2 or IM4. 

MICH Dark not locking.  Discovered that BS's M2 EUL2OSEM matrix wasn't loaded, so no signal being sent out.  Hit load, MICH locked, moved on.

Bounce needed hand damping (guardian will prompt you in DARM_WFS state), roll will probably need it too.  This isn't surprising, since it happens whenever the ISI for a quad optic trips.  Recall that you can find the final gains (and the signs) for all of these in the ISC_LOCK guardian.  I like to start with something small-ish, and increase the gain as the mode damps away.  No filters need to be changed, just the gains.  My starting values are in the table below, and I usually go up by factors of 2 or 3 at a time. 

I lost lock while damping the bounce mode (in DARM_WFS state), and the DRMI alignment is coming back much worse than the first few DRMI locks I had.  

I don't actually have a lot of faith in my input beam alignment, so I probably wouldn't be happy with any ASC loop measurements I take tonight even if I got the IFO locked.  Since we have an 8am meeting, I'm going to call it a night, and ask the morning operator to check my alignment and fix anything that sleepy-Jenne messed up.

There is water all over the floor in the hallway leading to the control room from the water cooler leaking. 

I have dried up much of it, and left a cone in the hall.  I moved the electronics that are stored in that area out of the way.  I also opened an FRS ticket (hopefully not an ECR....).

TITLE: Jan 20 EVE Shift 20:00-04:00UTC (12:00-20:00 PDT), all times posted in UTC

STATE Of H1: Commissioning

SUPPORT: Jenne, Sheila, Jeff and Evan

DAY OPERATOR: Patrick

SHIFT SUMMARY:

Power outage recovery ongoing. Patrick still here. After some ESD woes and a few journeys to end stations the ESD situation seems to be resolved for now. Another trip to EX to power cycle TMS coil drivers to clear suspect oscillations occurring in the 1Khz neighborhood. Arm alignment was a daunting task and is still going on as Y arm WFS seem to be a destructive force rather than help, at this point. Jenne is working on it. There was a missing beatnote in the PLL for X-end ALS that was repaired, prior.

Is being reported by the verbal alarms script.

JeffK and Ed noticed that the ALS X spot was flashing much faster than normal, and they looked at the spectra of the TMSX OSEMs and saw large oscillations at a few kHz.  This seems to be a little worse than the situation in July (alog 20118).  

Ed and I went to the end station and power cycled both coil driver chassis, but the noise is still there.  Tomorrow (or tonight if we can't lock with this noise) someone should perhaps try Vern's trick of giving the connector an impulse force (a good ol' whack), which fixed the problem back in July.

Attached is a screenshot comparing the TMSX OSEMs with the TMSY OSEMs that do not have this problem.  Unfortunately, we only record these channels at 256 Hz, so we don't know exactly how long these peaks have been around (since this morning's power outage? longer?).

Error logged: Jan 20 09:18:39 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Incorrect string value: 'xEBxE26x1A?' for column 'value' at row 1: Error code: 1366: SQLState: HY000: Exiting.

I believe this may have happened after one of the end station Beckhoff computers was brought back online.

STATE OF H1: Still recovering from power outage.

ACTIVITY LOG (some missing):
15:44 UTC Unexpected power outage
16:38 UTC Fire department through gate to check on RFAR boxes
16:56 UTC Richard, Jeff B. and Jim W. to end stations to turn on high voltage and HEPI pumps
17:14 UTC Richard turning on h1ecaty1
17:14 UTC Jason and Peter to LVEA to look for an optic
17:39 UTC Richard and company done at end X and going to corner station to turn on TCS chillers and HEPI pumps
17:49 UTC Filiberto to end X to reset fire panel
17:56 UTC Vacuum group touring LVEA (6 or 7 people)
17:57 UTC HEPI pump stations and TCS chillers started in corner station
18:18 UTC Filiberto back from end X
18:24 UTC Jeff B. and Jason to LVEA to start TCS X laser
18:49 UTC Hugh bringing up and isolating HAM ISIs
19:16 UTC Richard and Jim W. to end stations to turn on ALS lasers
19:30 UTC Sheila to CER to power cycle all of the Beckhoff chassis
22:55 UTC Jason to LVEA to look at optical levers
23:02 UTC Jason back
23:34 UTC Dave restarting the EPICS gateway between the slow controls network and the frontend network in an attempt to fix an issue with the Beckhoff SDF
23:47 UTC Dave restarting the Beckhoff SDF code

Other notes:
End stations: end Y IRIG B chassis power cycled; end X, end Y high voltage turned on; end X, end Y HEPI pump station computers started; end X, end Y ISI coil drivers chassis reset; end Y Beckhoff computer turned on
Corner station: TCS chillers turned on; HEPI pump controllers turned on (Jeff B. had to push power button on distribution box); Sheila power cycled all Beckhoff chassis in CER; Sheila turned on the AOS baffle PD chassis in LVEA

I started the h0video IOC on h0epics2.

Conlog was running upon arrival. However it crashed during the recovery. It still needs to be brought back.

Joe D. and Chris worked on beam tube enclosure sealing.

There was some trouble starting the corner station HEPI pump controller computer.

There was some trouble finding the strip tool template for the wind speeds. I'm not sure if it was found or if TJ created a new one.

Things to add to the 'short power outage recovery' document:
- Turn on high voltage supplies
- Push reset on ISI coil drivers
- HEPI pump controller and pumps
- TCS chillers
- TCS lasers
- ALS lasers
- Turn on video server

We have begun recovery from an unexpected power outage that occurred at 7:44 PST and lasted for 3 minutes (as noted from the UPS).

Robert, Sheila, Evan, Gabriele

I tried to look at one of Robert's injections from yesterday, and we noticed a dangerous bug, which had previously been reported by Annamaria and Robert 20410.  This is also the subject of https://bugzilla.ligo-wa.caltech.edu/bugzilla3/show_bug.cgi?id=804

When we changed the Stop frequency on the template, without changing anything else, the noise in DARM changes.  

This means we can't look at ISI, ASC, PEM, or SUS channels at the same time as DARM channels and get a proper representation of the DARM noise, which is what we need to be doing right now to improve our low frequency noise.  Can we trust coherence measurements between channels that have different sampling rates?

This is not the same problem as reported by Robert and Keita alog 22094

people have looked at the DTT manual and speculate that this could be because of the aggressive whitening on this channel, and the fact that DTT downsmaples before taking the spectrum.  

If there is no near term prospect for fixing the problem in DTT, then we would want to have less aggressive whitening for CAL_DELTA_L_EXTERNAL

Summary and discussion

Radar plots of the sensing matrix for PRCL, MICH, and SRCL are attached (where MICH is the usual beamsplitter drive). This was taken in the nominal O1 configuration.

Something is strange with the response in POP9; driving beamsplitter seems to result in the same response as driving PRM. Certainly we expect some amount of POP9I response from driving the beamsplitter, since this drives PRCL, but it seems strange that they are almost exactly equal.

For POP45, beamsplitter appears in both I and Q; again, this is not surprising. However, its contribution in I seems to dominate over the SRM contribution. This could potentially be bad, because we only do PRCL → SRCL subtraction (not MICH → SRCL subtraction).

Sensing matrix is as follows:

There is some ambiguity in the signs here that still needs to be resolved.

The magnitudes of the PRM ⇝ POP9I and SRM ⇝ POP45I elements agree fairly well with what is necessary to explain the open-loop transfer functions for PRCL and SRCL (3.6 W/µm and 0.13 W/µm, respectively). For MICH, there is some frequency-dependent residual between the model and the measurement (which has been observed before). The gain needed to make them match around the UGF is 0.6 W/µm.

Details

I added a 132.1 Hz notch to the PRCL, MICH, and SRCL SFMs. Then one by one I drove PRM, SRM, and BS with a 132.1 Hz line for a few minutes, using the digital oscillator just after the LSC SFMs.

I took the time series for POP9I/Q and POP45I/Q during this time, and demodulated them with 0.1 Hz low-passing.

All times are 2015­­–01–13 Z. The actuator calibrations are based on digital filters, the suspension models, and the suspension electronics.

• PRM:
  • Oscillator amplitude: 1000 ct
  • Injection time: 16:19:40 to 16:23:40
  • Calibration: −3.0×10⁻¹⁶ m/ct
• SRM:
  • Oscillator amplitude: 2000 ct
  • Injection time: 16:25:35 to 16:30:35
  • Calibration: −3.0×10⁻¹⁷ m/ct
• BS:
  • Oscillator amplitude: 20 ct
  • Injection time: 16:33:05 to 16:37:05
  • Calibration: 3.8×10⁻¹⁶ m/ct (including digital f² inversion filters)

As for sensors, the calibrations are 2.2×10⁸ ct/W for POP9 and and 2.3×10⁸ ct/W for POP45, based on LHO#24959 plus an additional 30 dB of whitening gain.