Kiwamu reported in 22781 about a couple lock losses due to the drive to HEPI hitting the 250um limit.  HEPI can handle more than this but should not need to if the PSL (likely Ref Cav) temperature does not trend off somewhere.

Attached are several PSL Temp channels for 14 days with the limit hit time noted on the run away Tidal Channel.  Certainly many of the PSL Temp channels show a trend that correlates pretty well with the run away.  In iLIGO we had tight temp control of the Ref Cav, do we have a temp of the Ref Cav now or is there another signal that would best represent that?

A summary of the O1 stochastic hardware injection (spanning 1129673117-1129673717) recovery is available in the DCC:
https://dcc.ligo.org/G1501327

Josh, Laura, Andy, Duncan, TJ,

Laura noticed that on 10/23/15 LSC X COMM CTRL was hitting its software limit (10 um) while H1 was locked, according to the nice software limits page set up by Duncan M on the summary pages. We've previously seen common mode tidal control hit software limits back on (17654 and 18748 and I remember but can't find the commissioning work to improve this). The attached plots show 1) the times when the software limits were hit, both near to the end of lock, but in lock (likely leading to lockloss), 2) the 24-hour min/max plot for the control channel, 3-6) the control signal and GDS CALIB STRAIN for the two saturation times (four saturation segments, two at each time). It looks like sometimes lock is acquired with the common-mode control being already quite close to its limits. This happens rarely, this is the only day in the past week we have seen it. 

I reset the H1 PSL power watchdog at 15:45 UTC (8:45 PDT).

Attached are trends of the Cartesian and the Local Position Sensors from which they are derived.  Unlike the HEPI after a trip, even the Local positions as well as the servo'd cartesian positions are bang on.  The ISI platform is in the same place.

Richard has transitioned LVEA to laser safe. All doors are locked. PSL still up. IMC still locked.

TITLE:  Oct 27 OWL Shift 7:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Maintenance

SUPPORT: Jenne, Kiwamu

LOCK DURATION: ~6.5hours

INCOMING OPERATOR: Nutsinee

END-OF-SHIFT SUMMARY:  IFO is locked and observing for ~6.5hours @ ~ 80Mpc. EQ sei bands are ~.22 microns/s. µSei is still a bit elevated to about .4 microns/s. Wind is calm.

SUS E_T_M_Y saturating (Oct 27 10:49:18 UTC)

SUS E_T_M_Y saturating (Oct 27 14:02:30 UTC)
SUS E_T_M_Y saturating (Oct 27 14:18:15 UTC)

ACTIVITY LOG:

07:17 Kiwamu on site to do some magic

07:30 Kiwamu manually engaging ISS second Loop

07:51 Observing mode

08:01 Lockloss

09:25  H1 back into Observing mode after much tweaking of IMs.

09:40 Oscillation in DHARD appeared in the ASC signals. Kiwamu lowered the gain to this loop to save the lock. (Commissioning Mode)

09:53 H1 back to Observing

11:13  Kiwamu is done and leaving the site.

12:04  restarted GraceDB script again

14:18  Chris, Joe and Bubba staging materials for grouting activities

14:34 Jeff is going to be taking a trailer around the side of the building and staging the forklift for work with Jody.

14:40 Richard out to LVEA to transition to LASER SAFE

15:00 Sprague on site

MID-SHIFT SUMMARY: Kiwamu has managed to correct some alignment offsets which occurred in the IMs during yesterday’s earthquake recovery. These alignment corrections along with some ASC gain adjustments seem to have removed the oscillations that were occurring. There was also some difficulty with engaging the ISS Second Loop. It had to be done manually during each locking sequence. It is my understanding that the ISS issues are previously scheduled to be addressed during normal weekly maintenance. the IFO is locked and observing for just a little over an hour now at ~ 80Mpc. EQ sei bands are ~.22 microns/s. µSei is still a bit elevated to about .5 microns/s. Wind is calm. I had to reset the GraceDB External Alarm Notification script. Livingston appeared on the BNS range graph about 15 minutes ago

Ed, Patrcik, Kiwamu,

In addition to the ISS issue (alog 22863), tonight we had two other issues with the alignment loops (DHARD instability and SRC2 instability, reported in alog 22859). After some investigation, we found that IM1 and IM2 had different alignment than usual, mainly in pitch. Even though we are not sure how exactly this affected all three issues we had tonight, we realigned them so that the OSEMs read the same values as the past. This resulted in two positive consequences as follows.

(1) Realigning IM1 and IM2 brought IM4 trans and POP_B QPDs back to where they were.

(2) The realignment increased the carrier power in both PRC and arm cavities by 5 or 6 %.

See the first attachment for the increased carrier power. We aligned IM1 and 2 while the interferometer was locked at low noise. We accepted the new IM alignments in SDF (see the third attachment). Even though we still suffer by the DHARD instability, at least the SRC2 instability seems to have gone away after the realignment. I did not attempt to maximize the carrier power but it makes me think that there might be a "sweep spot" for IMs' alignment where the carrier power is maximized in the interferometer. Also, note that IM4 trans sum increased as well with the newly aligned IMs, but only 0.6% perhaps due to the beam on IM4 trans getting centered.

IM1 and IM2 moved likely due to the untrip of HAM2

According to trends, it seems that both IM1 and 2 changed the alignment at the time when Ed untripped HAMs 2 and 3 (alog 22835). This effect is something we already knew (alog 22719). For some reason, this time it changed mostly IM1 and IM2 in pitch. See the second attachment for the trend. IM1 moved by about 1 urad, and IM2 moved by about 50 urad as seen by the OSEMs. Also, yaw of IM1 and 2 also moved, but neither IM4 trans nor POP_B see a significant change in yaw. The MC mirrors seems to have been unchanged.

DHARD is still unstable

I was hoping that the realignment of IMs would magically fix all the issues we had, but the DHARD instability seems to remain. The symptom is that both pitch and yaw DHARD oscillate at around 0.6 Hz. As was suggested by Jenne, we decreased their loop gain which fixed the instability. I have no idea why they oscillate at 0.6 Hz and why the gain reduction helped. The pit gain is set to 7 (which was 10 originally), and the yaw gain is set to 10 (was 15 originally). For this lock stretch, we accepted these values in SDF and this means that they will show up in the SDF next time when we lock the interferometer. See the fourth attachment for the SDF.

To those who will relock next time

Since we have aligned IM1 and IM2 in situ when the interferometer was locked at low noise, the interferometer alignment in the cold state will be somewhat far from the optimum. I would recommend running the full initial alignment before relocking.

10:35UTC

These all appear to be test injections that occured between Oct 5-6. I looked at the page out of curiosity and I saw this list on the home tab. I'll defer attention to these to someone who was involved in those activities. 

09:25UTC

07:51UTC

TITLE: Oct 27 OWL Shift 7:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Locking

OUTGOING OPERATOR: Patrick

QUICK SUMMARY: Jenne and Patrick discussing the state of affairs with Doug at Livingston. They’ve been down for 30+ hours and we’re currently having issues with ISS second loop not engaging. Kiwamu is on his way in. It looks as if my shift will get cut short when L1 goes into maintenance day (04:00PDT)

Yet again, we had trouble with engaging the ISS. 

I put in a notification during ENGAGE_ASC_PART3 for the operator to check the diffracted power level if it is more than 0.8% away from nominal.  While it's probably okay if it's within 1% (other alogs say 2%, but I've had more consistent success with 1%), I chose a somewhat smaller number out of an abundance of caution. 

However, even though the diffracted power was within a few tenths of a percent of nominal, the ISS would stall, and not engage.

As part of the PREPARE_ISS state, the guardian changes some setpoint values such that the H1:PSL-ISS_SECONDLOOP_SIGNAL_OUTPUT channel becomes small.  We need to do this so that the DC-coupled ISS doesn't kick us out of lock when it is engaged.  (For unknown reasons, this convergence has seemed to be more successful when the diffracted power is close to 8%).  This SECONDLOOP_SIGNAL_OUTPUT wasn't converging though.  It was bouncing around between +/-5 counts.  The absolute value of this channel must stay below 2.5 counts in order for the ISS to close the second loop. 

In the end, what I did, was cheat and change the gain H1:PSL-ISS_SECONDLOOP_SIGNAL_GAIN to 0.1 so that the ISS second loop would turn on.  Then, VERY slowly I turned the gain back after we had arrived at NomLowNoise.  As I increased the gain in steps of 0.1, the diffracted power dropped by half a percent or so.  So, at each step, I waited for the diffracted power to come back near 7% or so before going to the next step.  While this worked once, it has failed a few times after that, so let's not make it the new plan.

Patrick and I called Kiwamu, and followed the instructions in aLog 22449 to manually turn on the second loop (basically, turn on the button next to "second loop input to first loop" on the second loop screen, when the second loop output (near bottom right of that screen) is close to zero).  Anyhow, that is making the diffracted power go super low (less than 1% sometimes), until the outer loop loses lock, and the diffracted power jumps up really high (45% or more).  At this point, it's not at all clear why this isn't working, so Kiwamu is on his way in to try to diagnose things.  

We've checked in with Mike, and since LLO is still down, if Kiwamu and Ed can't figure out the ISS, we'll call off the rest of the owl shift. 

Daniel, Sheila, Evan

Summary

Over the past 45 days, we had two instances where the common-mode length control on the end-station HEPIs hit the 250 µm software limiter. One of these events seems to have resulted in a lockloss.

Details

The attached trends show the ISC drives to the HEPIs, the IMC-F control signal, and the IMC-F offloading to the suspension UIMs over the past 45 days. One can see two saturation events: one on 25 September, and another on 11 October.

We survived the event on 11 October: the EY HEPI hit the rail, and counts began to accumulate on the EY UIM, but the control signal turned around and HEPI came away from the rail. On the 25th of September, both end-station HEPIs hit the rail, and after about 2 hours of counts accumulating on the UIMs, IMC-F ran away (second attachment). Note that both HEPI drives started at 0 µm at the beginning of the lock stretch.