TITLE: 12/07 [OWL Shift]: 08:00-16:00 UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Unlocked
OUTGOING OPERATOR: Cheryl
QUICK SUMMARY:

Cheryl has been having trouble locking MICH in initial alignment. Two earthquakes have just hit as I came on shift:

5.6
34km NW of Illapel, Chile
2015-12-07 07:51:11 UTC 48.4 km deep

7.2
109km W of Murghob, Tajikistan
2015-12-07 07:50:07 UTC 28.7 km deep

A number of watchdogs have begun to trip.

From the cameras:
The lights are off in the LVEA.
The lights are off in the PSL enclosure.
I can not tell if the lights are on or off at mid X and mid Y.
The lights are off at end X.
The lights are off at end Y.

The 0.03 - 0.1 Hz (earthquake) seismic band has so far risen to ~ 5 um/s with the earthquakes.
The 0.1 - 0.3 Hz (microseism) seismic band is above the 90 percentile and has risen above .5 um/s with the earthquakes.

The winds are between 0 and 20 mph.

The ISI blends are on the nominal 45 mHz configuration.

From pinging:
CDS WAP is off at the LVEA.
CDS WAP is on at mid X.
CDS WAP is on at mid Y.
CDS WAP is off at end X.
CDS WAP is off at end Y.

Ops Eve Summary: 00:01-08:00UTC (16:00-23:59PT)

State of H1: unlcoked

Incoming Operator: Patrick Thomas

Helpers: Keita, Vern, Mike

Details:

• H1 has been unlocked for about 12 hours.
• Corey did an Initial Alignment, and I tried locking
• with no locking, I decided to do an Initial Alignment
• no change in locking, MICH DARK would not lock
• looked for other things, found a behavior on BSC2 ISI that appears to be different from behvior two days ago, on 12/5
• still not sure that BSC2 ISI anything related to not locking MICH DARK

7:00UTC, no locking

I decided to do an Initial alignment and look more closely that usual, and again made it to MICH DARK which isn't locking.

I looked at the BSC2/BS ISI and saw that when MICH DARK is trying to lock, the ISI stage 2 is dark red on the ST2_BLRMS monitors.

Plots attached show MICH DARK trying to lock and not trying to lock.

Maybe this is normal, but lots of shaking is the symptom of the MICH DARK not locking, and ther BLRMS readbacks are showing a lot of shaking in the BSC2/BS ISI.

State of H1: 9+ hours, no locks past DRMI

Current issues:

• During initial alignment, I could not get MICH_DARK to lock.
• I called Keita, who logged in to take a look.
• Alignment is good, so it's not clear why DARK_MICH isn't locking.
• I reloaded and ran each guardian in the ISC screen through it's INIT state, hoping to find that that cleared up the issue with DARK MICH, but it didn't.
• Attached is a plot of ASAIR A RF45 Q ERR and MICH out showing ther signal running away.
• Additional assistance is being sought.

TITLE:  12/6 DAY Shift:  16:00-00:00UTC (08:00-16:00PDT), all times posted in UTC     

STATE of H1:   Lock Acquitistion

Incoming Operator:  Cheryl

Support:  None

Quick Summary:

Eerily similar to my shift yesterday:  First half H1 locked, and then 2nd half unlocked.  This time there was winds which forced me to not attempt locking.  A good note (highlighted in GREEN below) is that I was able to (1) get through an alignment, (2) have decent looking DRMI locking/flashing, & (3) able to lock PRMI (although it wasn't needed since alignment still looked decent).  Still NLN elludes me (seems like I've not taken H1 to NLN since early October!!)  :(

Shift Log:

• 16:48 LY Beam Manifold Ion Pump went RED on overview (no alarms)
• 19:15 - 22:30 wind avg speeds reach above 20mph.
• 20:20  Lockloss (winds picking up & useism high as range died at the end of that lock)
• 22:30 - 22:50 Initial Alignment.  Notes Below:
  • ALSy was better, but had to offload by hand (i.e. tweak pit & yaw of ETMy)
  • Input Align:  X-arm finally locked!  (because seismically quieter?).  Offloaded IM4 & PR2...but saw that IM4 seemed to get pitch & yaw values after they were offloaded to zero.   (is that normal?  I did not offload by hand)
  • SRC Broke lock after 2min while waiting for control signals to converge; but finally able to run OFFLOAD
  • Alignment went well!
• 22:50 Lock acquisition:  finally made it past CHECK_IR!  DRMI AS port flashing look GOOD with decent flashing for POP 18 & 90.  But after 11min, Guardian went DOWN.  :-/  Looks like it's due to ETMx ringing up (had a peak to peak of 10,000nm).  Cleared this up by switching to the 90mHz Blend briefly for the X-axis, and then switching back to the 45mHz Blend.  
• FIND_IR takes on the order of 2-3min
• 23:33 Attempt Lock PRMI (after 4min PRMI locked!).  Only made minor changes to BS/PRM:  POP18 was at 124 & POP90 was at 190.  Went back to DRMI locking....actually, not clear of how to transition back.  Do I take ISC LOCK to AUTO and then request DRMI?  I was a little confused.  about that.
• It looked like a PRMI locked, and then HAM5 ISI & SRM WD tripped....did I not restore DRMI locking correctly??
• Resetting WDs
• PRMI relocked again (!).  Not sure if I recovered from PRMI....anyway, Cheryl is starting from scratch with this alignment (hopefully it locks up with this pesky useism!)

Beam Manifold Ion Pump looks like it is railed at 10 at the beginning of the shift.

After the 20:20 lockloss, checked to see if I could get any where in locking, but H1 could not get past FIND_IR (never saw any X-arm locks).  Why are X-Arm locks suddenly harder?  Is it really due to all this seismic?

The EY BSCISI Location monitor had over 10000nm oscillations.  When it died, I tried switching Y-axis to 90mHz (this brought oscillations down, but did not improve anything with locking, so I returned 45mHz Blend).

Started an Initial Alignment.  ALS was sort of fine (Y-arm would break lock every few seconds/minutes).  INPUT_ALIGN would not lock the X-ARM.  At this point, I took locking DOWN to wait out the wind (will watch the End Station drifts on the 0.03-0.1Hz DMT Viewer to see when winds die down).

Ambient emperatures are up and it appears that the LVEA is warming quickly so I have dialed down one heater;

HC 2B has been lowered to 9ma (one stage) from 13ma (2 stages).

Winds have started to avg up around 20mph site-wide (gusts on X-arm up to 40mph).  Might be dying down?

Range dropped down to 74Mpc with winds.

EY BSCISI Location monitor has had some big oscillations of +/-10,000nm over the last few minutes.  SUS EY on the Tidal FOM (nuc1) has some clear sinusoids recently, too.

From 11-11:30amPST, Air Handler MAJOR alarm for LVEA 2B ("as usual"?)

Attached are trends of useism & wind from August to present for E8/O1. 

MICROSEISM:  Clearly, useism has been growing with the season.  The worst storm was around 11/12 (but current one looks to be approaching that level, too).

WIND:  What made 11/12 storm worse was that it ALSO had winds over 40mph!  (For this current storm, winds have generally been under 10mph, BUT over the last few hours we had rain & now the winds are picking up to around 20mph (with an EX gust near 40mph). 

Will continue to monitor (the H1 range is running great at 78Mpc and no ETMy saturations thus far *knock on wood*).  And End Station ISIs look good with no ring-ups on the StripTool.

TITLE:  12/6 DAY Shift:  16:00-00:00UTC (08:00-16:00PDT), all times posted in UTC     

STATE of H1:  NLN at 78Mpc

Outgoing Operator:  Jim

Quick Summary:

It's been 90min (chatted with Jim for first hour).  Great work on helping H1 ride through these extremely rough seas of ground motion!  Last night after reading Cheryl's alog about rung up ISI, I wanted to ask Jim:  

What is a rung up ISI?  

• So, a rung up ISI won't necessarily trip a watch dog.  
• Problem child here is mainly:  EX BSCISI Stage1 X
• It can ring up during rough seismic noise (like now).
• Symptom:  large sinusoidal motion for the End Station ISIs.  (+/- 10,000nanometers is bad!  5000nm is avg.  20000nm is worst observed)

Watch End Station Location Monitors:  I have a striptool up and I am watching H1:ISI-ETM[x & y]_ST1_CPS_[x & y]_LOCATIONMON.  The y-axis is set to +/- 15,000nm.  (signals currently about +/- 1000nm with not a very strong/visible sinusoid).

useism continues to be bad.  Looks like it reached its worse during Cheryl's EVE and continues to stay elevated (LVEA has had some data points above 2um/s; easily averaging 1.1um/s.)

If we drop out of lock, Jim says I should NOT have to do anything with the ISIs (let Guardian do its thing).  IF, I get a rung up EX, he says I could watch the EX Location Monitor, and wait for it to calm and then try enabling the X-direction 90mHz BLEND.  (but that's only an iffy proposition)

Basically, we are in one of those unique periods when seismic is noisy enough to push us into this out-of-norm SEI state.

O1 day 80

model restarts logged for Sat 05/Dec/2015 No restarts reported

Title: 12/5 owl Shift 8:00-16:00 UTC

State of H1: NLN

Shift Summary:Rough night, EX ISI was rung up (microseism? IDK), took several hours to get locked

Activity log:

Cheryl had no luck relocking earlier because the EX ISI was misbehaving. I spent several hours poking at it, I'm not sure the things I tried had any effect, but it's possible we want to start engaging the last couple of control loops (St2 RX/RY) on this ISI. ISS 2nd loop engagement was also an issue.

It's been a rough one so far. ETMX ISI was rung up when I came in. I've tried a bunch of things, I'm not sure what or IF I've fixed it. Currently, the Z sensor correction is running on the ISI and the RY loop is running on St2. I've switched everything back to nominal, I'll keep watching the ISI, but for now everything looks OK. Shifting the Z sensor correction was what I thought earlier was the fix, but when I got almost to NLN, EX ISI rang up and blew the lock. I just gotten to NLN I've now turned on the St2 RY isolation loop, and the ISI may have settled down, but I've been here before. The current St2 RY blend is a 750mhz blend that injects a lot of GS13 noise at low frequency and CPS noise at high frequency, so it's not ideal, and I have a hard time believing that this configuration is really necessary for the ISI's stability but it's difficult to argue with qualified success.

Attached are a few spectra of the EX ISI St2 sensors. First plot is the RX (blue), RY(brown) GS13s, with GS13 noise curve(black). You can see turning on the loop makes the microseism better by a little, but worse at the blend cross-over and at a couple hz. Maybe the high microseism is whats giving us problems? Next plot are the CPSs red is RX, green is RY, I kept GS13 noise in for scale. Again, microseism is suppressed, but GS13 noise is making the low frequency worse by almost an order of magnitude, below 70mhz.

Turning on the St2 RY loop looked like it settled the ISI down, but I've backed everything out, and I think I can go back to observing. The EX ISI still needs monitoring.

Ops Eve Summary: 00:01-08:00UTC (16:00-23:59PT)

State of H1: down, ETMX ISI is oscillating

Help: Keita and Vern

Incoming Operator: JimW, who's looking into the ISI now

Shift Summary:

• difficulties getting H1 to lock,
• but after an initial alignment,
• the DRMI did lock more than once,
• but woulnd't hold,
• the switch of ETMX ISI blend filters from (quiet90 to 45mHz),
• threw the ISI into oscillation, 
• and while those oscillations have at times appeared to be dampled,
• they came back and prevented locking,
• so I switched ETMX blend filters to  x dof = 45mHz, y dof = quite90,
• after remembering that Jim had done this with some success in the past,
• however this did not fix the oscillations either

Currently:

• useism at CS is at 1um/s and increased throughout this shift, and appears to still be increasing
• winds are below 5mph