Added 100 mL of H2O to the H1 PSL crystal chiller.

After moving RM1 and RM2, I performed an initial alignment and took the IFO to NLN. The lock lasted a couple of minutes before being taken out by a large earthquake.

DRMI 1F locked, then dipped in power, and recovered (see attached screenshot).

I trended back the positions of RM1 and RM2 for the last 3 days (see attached). The differences appeared significant, so I used the alignment sliders to bring them back.

RM1:
Pitch: 380 -> 416
Yaw: 632 -> 331

RM2:
Pitch: -1149 -> -856
Yaw: 2664 -> 1306

tl;dr: Alignment of IMs and RMs should be checked, then re-run an initial alignment.

Xarm green alignment

Ed was struggling with the Xarm green alignment.  We set the guardian to locked no slow, no wfs.  I then turned on the loops one at a time.  Usually the camera centering loops are the problem, but they were the easy ones this time.  Eventually it was DOF2 that was causing trouble, so I had DOFs 1 and 3 closed and touched TMS by hand to get the error signals for DOF2 closer to zero.  I was able to close all the loops, and let the alignment run like normal after that.

Xarm IR alignment

Not really sure what the problem is here, but it's getting late and I'm getting frustrated, so I'm going to see if I can move on with just hand aligning the input. 

I suspect that the IMs need some more attention, so if Cheryl (or someone else following Cheryl's procedures) could check on those again in the morning, that would be great. 

Also, I'm not sure if the RMs got any attention today, but the DC centering servos are struggling.  I've increased the limits on DC servos 1 and 2, both pitch and yaw (they used to all be 555, now they're all 2000).  I also increased H1:SUS-RM2_M1_LOCK_Y_LIMIT from 1000 to 2000. 

Allowing the INP ASC loops to come on is consistently causing the arm power to decay from 0.85ish to lockloss. I didn't touch the ITM or ETM, but I'm not getting any more power by adjusting PR2 or IM4. 

MICH Dark not locking.  Discovered that BS's M2 EUL2OSEM matrix wasn't loaded, so no signal being sent out.  Hit load, MICH locked, moved on.

Bounce needed hand damping (guardian will prompt you in DARM_WFS state), roll will probably need it too.  This isn't surprising, since it happens whenever the ISI for a quad optic trips.  Recall that you can find the final gains (and the signs) for all of these in the ISC_LOCK guardian.  I like to start with something small-ish, and increase the gain as the mode damps away.  No filters need to be changed, just the gains.  My starting values are in the table below, and I usually go up by factors of 2 or 3 at a time. 

I lost lock while damping the bounce mode (in DARM_WFS state), and the DRMI alignment is coming back much worse than the first few DRMI locks I had.  

I don't actually have a lot of faith in my input beam alignment, so I probably wouldn't be happy with any ASC loop measurements I take tonight even if I got the IFO locked.  Since we have an 8am meeting, I'm going to call it a night, and ask the morning operator to check my alignment and fix anything that sleepy-Jenne messed up.

There is water all over the floor in the hallway leading to the control room from the water cooler leaking. 

I have dried up much of it, and left a cone in the hall.  I moved the electronics that are stored in that area out of the way.  I also opened an FRS ticket (hopefully not an ECR....).

TITLE: Jan 20 EVE Shift 20:00-04:00UTC (12:00-20:00 PDT), all times posted in UTC

STATE Of H1: Commissioning

SUPPORT: Jenne, Sheila, Jeff and Evan

DAY OPERATOR: Patrick

SHIFT SUMMARY:

Power outage recovery ongoing. Patrick still here. After some ESD woes and a few journeys to end stations the ESD situation seems to be resolved for now. Another trip to EX to power cycle TMS coil drivers to clear suspect oscillations occurring in the 1Khz neighborhood. Arm alignment was a daunting task and is still going on as Y arm WFS seem to be a destructive force rather than help, at this point. Jenne is working on it. There was a missing beatnote in the PLL for X-end ALS that was repaired, prior.

Is being reported by the verbal alarms script.

JeffK and Ed noticed that the ALS X spot was flashing much faster than normal, and they looked at the spectra of the TMSX OSEMs and saw large oscillations at a few kHz.  This seems to be a little worse than the situation in July (alog 20118).  

Ed and I went to the end station and power cycled both coil driver chassis, but the noise is still there.  Tomorrow (or tonight if we can't lock with this noise) someone should perhaps try Vern's trick of giving the connector an impulse force (a good ol' whack), which fixed the problem back in July.

Attached is a screenshot comparing the TMSX OSEMs with the TMSY OSEMs that do not have this problem.  Unfortunately, we only record these channels at 256 Hz, so we don't know exactly how long these peaks have been around (since this morning's power outage? longer?).

Error logged: Jan 20 09:18:39 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Incorrect string value: 'xEBxE26x1A?' for column 'value' at row 1: Error code: 1366: SQLState: HY000: Exiting.

I believe this may have happened after one of the end station Beckhoff computers was brought back online.

STATE OF H1: Still recovering from power outage.

ACTIVITY LOG (some missing):
15:44 UTC Unexpected power outage
16:38 UTC Fire department through gate to check on RFAR boxes
16:56 UTC Richard, Jeff B. and Jim W. to end stations to turn on high voltage and HEPI pumps
17:14 UTC Richard turning on h1ecaty1
17:14 UTC Jason and Peter to LVEA to look for an optic
17:39 UTC Richard and company done at end X and going to corner station to turn on TCS chillers and HEPI pumps
17:49 UTC Filiberto to end X to reset fire panel
17:56 UTC Vacuum group touring LVEA (6 or 7 people)
17:57 UTC HEPI pump stations and TCS chillers started in corner station
18:18 UTC Filiberto back from end X
18:24 UTC Jeff B. and Jason to LVEA to start TCS X laser
18:49 UTC Hugh bringing up and isolating HAM ISIs
19:16 UTC Richard and Jim W. to end stations to turn on ALS lasers
19:30 UTC Sheila to CER to power cycle all of the Beckhoff chassis
22:55 UTC Jason to LVEA to look at optical levers
23:02 UTC Jason back
23:34 UTC Dave restarting the EPICS gateway between the slow controls network and the frontend network in an attempt to fix an issue with the Beckhoff SDF
23:47 UTC Dave restarting the Beckhoff SDF code

Other notes:
End stations: end Y IRIG B chassis power cycled; end X, end Y high voltage turned on; end X, end Y HEPI pump station computers started; end X, end Y ISI coil drivers chassis reset; end Y Beckhoff computer turned on
Corner station: TCS chillers turned on; HEPI pump controllers turned on (Jeff B. had to push power button on distribution box); Sheila power cycled all Beckhoff chassis in CER; Sheila turned on the AOS baffle PD chassis in LVEA

I started the h0video IOC on h0epics2.

Conlog was running upon arrival. However it crashed during the recovery. It still needs to be brought back.

Joe D. and Chris worked on beam tube enclosure sealing.

There was some trouble starting the corner station HEPI pump controller computer.

There was some trouble finding the strip tool template for the wind speeds. I'm not sure if it was found or if TJ created a new one.

Things to add to the 'short power outage recovery' document:
- Turn on high voltage supplies
- Push reset on ISI coil drivers
- HEPI pump controller and pumps
- TCS chillers
- TCS lasers
- ALS lasers
- Turn on video server

Upon arrival, on the CDS overview, under the Slow Controls diagnostics, all of the corner station Beckhoff PLCS were green and updating. All of the end station Beckhoff PLCS had white boxes.

Corner Station:
I was informed that the TCSX laser was not working. There were errors on some of the terminals in the system manager, so I asked Sheila to powercycle all of the chassis in the CER. This seemed to fix the errors and the TCSX laser. She also found all of the AOS Baffle PD chassis in the LVEA off and turned them on. I burtrestored all the PLCs for h1ecatc1 to 6:10 AM PST.

End Stations:
Richard turned on or powercycled h1ecaty1 (I do not know if he found it off). I believe the Beckhoff PLCS for end Y turned green and started updating on the CDS overview after this. I was able to log into h1ecatx1. I found that the EPICS IOC had not started cleanly (see attached screenshot). I quit it and started it again. The Beckhoff PLCS for end X turned green and started updating on the CDS overview after this.

The EtherCAT vacuum gauges did not come back cleanly. All of them were reporting a pressure of flat 0. We have not been able to bring back the gauges in the beam tube enclosures and they have been disabled in the system managers. At end Y, on Vacuum Gauge ETM, under Process Data, Daniel hit 'Load info from device'. On Vacuum Gauge NEG, under Process Data, Daniel toggled the checkboxes under PDO Assignment. For some reason, once he did this and activated the configuration and relinked the variables, the gauges started reporting reasonable values. This did not work for me at end X. If I just hit 'Load info from device' or toggled the checkboxes, and then hit activate configuration, the gauges would start reporting reasonable values. However, as soon as I tried to relink the variables, the data went invalid again. I ended up having to remove and add back the gauges in the system manager and relink them. After that they seemed to work. None of these changes have been (and probably should not be) commited to subversion. We do not know the root cause of these problems.

I burtrestored all the PLCS for h1ecatx1 to 6:10 AM PST. At some point I did the same for h1ecaty1, but I do not remember if the IOC got restarted since. Things seem to be working for it for now, so I am not attempting to burtrestore it again.

Gabriele, Sheila

The message:

We had a look at the upconversion we would expect from the quadratic response of DC readout.  The message is that the large DARM residual at around 3Hz (due to our LSC feedforward) limits our noise near 10Hz, and upconversion around the calibration lines is about a factor of 4 below DARM at 40 Hz. 

Details:

The DCPD photocurrent is proportional to the power on them:

P_DC offset current (20mA),  G_opt is the optical gain (3.3 mA/pm), and x_0 is the darm offset (12 pm).

If we inject a line we expect to see upconversion at the second harmonic, the amplitude of the second harmonic (seen in the DCPDs) over the fundamental should be:

This is the explanation for the upconversion mentioned in alogs 25001 and 21240 .

Since the quadratic term should be small we can approximate the DARM residual, and use it to predict the noise in the DC PDs due to upconversion:

We took the data from a time when I was injecting a 6 Hz line into DARM and made this projection.  The upconversion of the 6 Hz peak predicts the 12 Hz peak well. 

The rms of our DARM residual is dominated by bad feedforward around 3 Hz, this is upconverted and limits our sensitivity at around 10 Hz.  There is also upconversion around the calibration lines that is about a factor of 4 below DARM near 40 Hz. 

Peter, Jason, Jeff B, Nutsinee

The TCS CO2X laser was down due to the power outage this morning along with many other systems. Jeff Bartlett reported that the chiller was tripped and he restarted it. We later restarted the power supply on the mezzaine which then brought the system back up. However, the timeseries of the flow alarm didn't imply that the alarm was tripped when the chiller was shut off and the laser output continued to have non-zero reading for over an hour until the frontend was restarted. Moral of the story is, some of the channels would continue to record fault values after a power outage until the frontend gets restarted. Another moral of the story, maybe we should move TCS CO2 power supply to the CER.

This afternoon I took measurements of the DHARD Yaw loop at different PSL powers.  In addition to general characterization of the O1 IFO, I will use this data to verify the ASC loop model.  Once we're confident in the loop model at powers that we can measure, we will use it to try to design ASC filters that we can use for high power operation in a few months.

In the first attached screenshot and .xml file, the measurement at 2 W is blue, the measurement at 10 W is orange, and the measurement at 20 W is red. 

The 2 W measurement was taken at the DC_READOUT state, and only FM6 of the DHARD Yaw filter bank was engaged. This measurement was taken from a lock stretch earlier in the day, using 40 points at 3 avg each. In the xml file, the 2 W data is saved as references 0-4.  In the second screenshot and .xml file, I include some higher resolution measurements of the peaks, with 5 avg for each point.

The 10 W measurement was taken at INCREASE_POWER, and both FM2 and FM6 were engaged.  This measurement used 60 points at 5 avg each. In the xml file, the 10 W data is saved as references 5-9.   I had modified the lscparams.py guardian code to stop the power increase at 10 W, but I have reverted that change, so everything should still be as normal.

The 20 W measurement was taken at INCREASE_POWER, and both FM2 and FM6 were engaged.  This measurement used 60 points at 5 avg each. In the xml file, the 20 W data is the "live" traces.

The 10W and 20W measurements today are broadly consistent with the measurements from 31 July 2015 (alog 20084), which is good.

Robert, Sheila, Evan, Gabriele

I tried to look at one of Robert's injections from yesterday, and we noticed a dangerous bug, which had previously been reported by Annamaria and Robert 20410.  This is also the subject of https://bugzilla.ligo-wa.caltech.edu/bugzilla3/show_bug.cgi?id=804

When we changed the Stop frequency on the template, without changing anything else, the noise in DARM changes.  

This means we can't look at ISI, ASC, PEM, or SUS channels at the same time as DARM channels and get a proper representation of the DARM noise, which is what we need to be doing right now to improve our low frequency noise.  Can we trust coherence measurements between channels that have different sampling rates?

This is not the same problem as reported by Robert and Keita alog 22094

people have looked at the DTT manual and speculate that this could be because of the aggressive whitening on this channel, and the fact that DTT downsmaples before taking the spectrum.  

If there is no near term prospect for fixing the problem in DTT, then we would want to have less aggressive whitening for CAL_DELTA_L_EXTERNAL