Ops Owl Shift Summary

TITLE: 11/25 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC

STATE of H1: Down due to earthquakes

INCOMING OPERATOR: Ed

SHIFT SUMMARY: Anamaria left couple of suggestions of things that I can do to help improve the range. I was going to try for an hour but caused a lockloss in the process. Have been having trouble locking all my shift. There're only two ISI configurations I could try without using BRSX (which is still rung up). USEISM_NOBRSXY took me the furthest towards NLN (I was in NLN once with this configuration) but it seems unstable at ~37 mHz.

LOG:

9:22 Went to get food+cofee and came back to find PI mode 27 rung up pretty high. I was able to damp it.

14:53 Noticed an earthquake showing up on the FOM. USGS reported a 5.2M and 6.5M earthquake in Tajikistan. Terramon predicted the worst of it should arrive a little bit after 7am local time. I tried to switch ISI config to LARGE_EQ_NOBRSXY but lost lock in the process. Could switching the ISI state have caused the lockloss? The FOM reads about 0.8 um/s when I did the switch.

          About 20 mins later a verbal alarm warned me about the earthquake. For it to be useful I propose that the threshold is set lower (probably half way lower than the current threshold) so people have time to react accordingly.

15:05 GRB alert. Bad timing, Universe.

15:09 Kyle arrived, going into LVEA.

15:34 Kyle out

Measured PT180 bake temps -> made minor adjustments

~0710 - 0730 hrs. local -> Kyle in and out of the LVEA  

Survey of temperatures is as expected, 85C +/- 10C.  Abiding by manufacturer's spec. of 80C at flange of gauge ( = 65C at PCB end of sensor tube).  As such, have low expectations of efficacy of bake-out.  At least the setup doesn't seem to intrude on the IFO folk's efforts so maybe they will forget about it and we can extend the bake period (if we can't go hotter, maybe we can go longer?).  

Also, I removed the access ladder that I had left leaning against BSC8's dome.  

Lost lock while switching ISI config to get ready for a big earthquake

Staying down as 5.2M and 6.5M earthquakes in Tajikistan arrive at the site. I switched ISI config from WINDY_NO_BRSX_V2 to LARGE_EQ_NOBRSXY.

ITMX HWS code stopped

"Image acquisition failed error on Nov 23 2016 11:52:23 UTC." I killed the zombie process and reran the code.

Ops Owl Shift Transition

TITLE: 11/25 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC

STATE of H1: Observing at 69.4735Mpc

OUTGOING OPERATOR: Corey

CURRENT ENVIRONMENT: Wind: 5mph Gusts, 3mph 5min avg Primary useism: 0.04 μm/s Secondary useism: 0.66 μm/s

QUICK SUMMARY: H1 was locked when I arrived. POPAIR dropped during the first half and hour into the shift but it seems stable now. So is the range.

Eve Shift Summary

TITLE: 11/25  EVE Shift: 0:00-08:00 UTC (16:00-0:00 PST), all times posted in UTC
STATE of H1:  PEM Calibration
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:

• Seismic is still noisy (high useism & winds are blustery at 10-20mph).  
• BRSX is still down.
• See this alog for Guardian operation for the weekend.

LOG:

• H1 taken to NLN by:
  • Last few steps of Guardian have been in flux.  Just wanted a recap.  Basically after NOISE TUNINGS, we skip right to NLN.  (So if you select NLN, that will be the path you take.)
  • We are skipping PR3_REFL WFS because the PRC Gain (cyan trace on front StripTool was 29....should be 30+ in order to use PR3_REFL_WFS)
• 0:25 Lockloss due to either Robert's Injection, or SRC1 Loop issue (Kiwamu tried to fix the latter).
• Kyle onsite for BSC8 work.
• 1:19 H1 taken to NLN; NOT run an A2L (Cleared its SDF diffs during this lock)
  • Injection Incursions (Robert):  (1:22-1:34, 2:23-2:31, 3:26-3:36
  • 2:40 Mode28 rung up (Phase -30 --> +30 brought it down)
  • 2:57 Mode26 slowly ringing up (Phase -20 --> -80 took care of it).
  • 3:49-3:53 A2L run
  • 4:03  Lockloss:  EY ISI/SUS/TMS tripped as well as HAM6 ISI
• 5:38 NLN, 5:42 OBSERVING.
  • 5:52 PI Mode28 rung up (phase 30 --> -60 took care of it)
  • 6:02 PI Mode 27(?) (phase -200 -->-180)
  • 6:06 COMMISSIONING:  Closed AS & REFL Beam Diverters & ran A2L
  • 6:11 OBSERVING
  • 6:12 Lockloss
• Been a bit of a rough go for the last 1-2 hrs of the shift.  Locklosses at various places & from the last lock to the DRMI locks, something looks noisy.  
• Did see a "Tidal Limit" within 10% come up during one attempt at locking (looked at trends and they didn't look bad.  (Haven't received it since.)
• 7:42 NLN/OBSERVE & handing over to next shift at 8:00utc.

Current/Turkey-Weekend: (1) Guardian Operations For Last Steps & (2) SRC1

Wanted to make/consolidate a few notes regarding operations as of late & for this weekend

PRC Gain & PR3_REFL_WFS

The last few shifts/weeks we have been waivering on the PR3 (POP vs REFL) WFS steps.  As of this moment, we are bypassing BOTH of these steps in Guardian.  Kiwamu said that if we had higher PRC Gain, we could try for the POP WFS, but the PRC Gain (PR_GAIN_OUT16 on front StripTool) must be 30 or more....it has been hovering at 29, so we are skipping this step.  

Sheila tried to improve the PRC gain by tweaking on the PRC alignment over the last few weeks.  This sounds like a work in progress.  For the weekend, we will ignore these PR3 (POP/REFL) WFS steps.

Beam Diverters

Another step we have been skipping toward the end of the Guardian steps is the CLOSING BEAM DIVERTERS.  We want to keep the POP Beam Diverter open, but perhaps for better performance and consistency for PEM Injections, we have been closing the REFL & AS_AIR Beam Diverters Manually (i.e. Sitemap/LSC/Beam Diverters).  When you close the AS_AIR, it will give you a glitch in DARM, but it's not too scary.

POP90 & SRC1 pit/yaw Loop

There had been some issues with the SRC1 Loop.  Basically it would push the alignment such that the POPAIR_B_RF90 (on the front StripTool) trace would increase (a bad thing).  So, if you see this trace increase, you should:

• Go to the SRC1 Pit & Yaw Loops
• Zero their gains
• Then tweak on the SRM to bring the POPAIR_B_RF90 trace down. 
• Keep the SRC1 gains at zero.

Have not had issues with this for the (3) locks since Kiwamu left.  (Knock on wood.)

Weekly PSL Chiller Reservoir Top-Off

Topped Crystal Chiller with 250mL.  This Closes FAMIS 6498.

(The Sheet is full, & should be given to Jeff B.)

Mid-Shift Status......EY ISSUE!!! (as I started this alog)

Fairly noisy seismically (useism has been holding at 0.7um/s for last 18hrs & winds are at a steady 9mph).

H1 has been also holding at a steady range of 70Mpc for the last 3hrs (even through some short injections by Robert);  Noticeable 

.......

Ugh!  

4:02:  LOCKLOSS:  EY Went Crazy!

As I was typing this alog, H1 dropped out of lock.  There were no noticeable issues (POP90 had been flat for hours, PI Modes were buried, seismic was noisy, but no changes).  However, in addition to the usual HAM6ISI trip, EY's ISI, ETMy, & SUS all tripped.

I reset everyone's Watchdog, but when trying to run ALS, I was getting PDH Alerts (not enough light).  Looked further and found that the ETMy DID NOT come back to a normal state:

• It had many filter banks in a RED state (mainly their outputs OFF), ranging from M0DAMP, to Optic Align (offsets OFF), to L1 LOCK, etc. etc..  I know there were some gains which were also zero-ed.  Too many things to note on here.
• The M0 output channels were also either frozen (for top half ones) or zero for bottom half ones (this was different from ETMx).
• ODC was also in a RED state for ETMy
• I tried to use ETMy's Guardian to restore ETMy (I had to put it in Manual since it was MANAGED by ALIGN IFO).  I tried several states, but the issues noted above never were remedied.  
• I sent texts to Betsy, Jeff, & Richard

So I opened ETMx, and then went through everything that looked different with ETMy.  After a while, ETMy started to look fine.  The ODC even came back green.  At around this point, I talked with Betsy and I snapped a photo of ETMy with my phone to her & at this point it looked OK.  We looked at the safe.snaps, and the Diffs for ETMy looked known & acceptable.  

Additionally, ALS was locking in the Y-arm.  So we are hesitantly going to say we are back, but I hope there isn't a switch, gain, filter, etc. which I could have missed.

OK, back at it.

SDF Updates: SDF GREEN, For Now.

Since H1 has been running suprisingly stable (no PIs rung up [speak of the devil---28 perked up as I typed] & SRC1 has been servoing fine), I went ahead and addressed DIFFERENCES on the SDF.  Going a node at a time, below I list how many DIFFS each had, and also give a summary note about them as well.  Attached you will also find a snapshots for all the DIFFs before I made them GREEN (well, except for SUSETMY...I forgot to snap one).

I reckon some of these will be RED again soon due to changes:  PI's always change, Calibration may change, etc.

SUSIM:  1  Total Diffs

ACCEPTED:  TRAMP for IM3 Pit 

LSC:  6 Total Diffs

ACCEPTED all Gain changes.

ASC:  33 Total Diffs

ACCEPTED:  PRC2_Y Filter difference, POP90 Gain, a few offsets, many matrix element changes, and many ODC channels.

REVERTED (2) TRamps

OMC:  3 Total Diffs

ACCEPTED:  ODC channels

PSLISS:  1 Total Diffs

ACCEPTED:  Ref_signal (has been moving around last few days)

ISIETMX:  9 Total Diffs

ACCEPTED all 9 filter changes (this is due to no BRSx for next few days)---For these Seismic SDFs, we probably want to update these when we are back to nominal environmental conditions and have BRSx back.  Since the SEI Config could change a bit depending on weather & health of BRS's, will we NOT monitor these?

SUSETMX:  1 Total Diffs

ACCEPTED:  GAIN for the L1 Lock Gain of ETMx  (its nominal WAS 0.0)

SUSETMY:  2 Total Diffs  (forgot to take snapshot!)

ACCEPTED:  P2L & Y2L TRAMPS

ALSEX:  3 Total Diffs

ACCEPTED:  These are all "EXTRA_AI" ??

ALSEY:  3 Total Diffs

ACCEPTED:  These are all "EXTRA_AI" ??

SUSPROCPI:  4 Total Diffs

ACCEPTED gain & phase changes

CS ECAT PLC1:  8 Total Diffs

All related to "pico"

CALCS:  1 Total Diffs

Value for Pause

CALEX:  1 Total Diffs (forgot to take snapshot!)

Value for SINGAIN

~1705 - 1715 hrs. local -> Kyle in and out of LVEA -> Measured temps and made adjustments @ BSC8 PT180 bake

"Slowly but surely..."  I'll need BSC8 access in the morning and again tomorrow evening.  

Check out the attached image.  This looks like a step response that couldn't be the result of a real pressure change.  I would guess it has something to do with the shared 24 DC supply between PT180 and PT120B and that pulling the fuse for PT180 changed the supply's load - or something - but nothing related to heating the PT180 hardware.

Ops EVE Shift Transition

TITLE: 11/25 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 22mph Gusts, 15mph 5min avg---> Hoping it doesn't get windier!
    Primary useism: 0.06 μm/s
    Secondary useism: 0.66 μm/s---> this is pretty high.  Hope it stays flat or (obviously drops)
QUICK SUMMARY:

• Arrived to see Kiwamu & Ed taking H1 to NLN eventually.
• PRC Gain at 29, so no PR3_REFL_WFS step in Guardian (must be 30 or more to attempt it.  This trace is the cyan trace on front Strip Tool))
• Appear to still have issues with SRC1 Loop in that POP90 tends to be driven higher.  (when this happens, zero the gain in SRC1 P&Y & tweak SRM to minimize POP90 (bright green trace on front StripTool).
• Had a lockloss and back up now.  Robert is out trying more Injections.
  • Took two attempts to make it to NLN
  • First attempt required BS tweak at Check Fringes & BS/PRM tweaks at LOCK PRMI.
  • Kyle also tiptoed out around BSC8 while attempting lock.
• Jim was here addressing SEI system.  Because we are in a bit of a noisy seismic state (fairly high useism & winds over 15mph), we are almost at a state where the current ISI_CONFIG will be pushed beyond its control.  Currently running at:  WINDY_NO_BRSX_V2 (a new state).
  • Continue to NOT have BRSx (this is due to ring up from yesterday).  Jim may address tomorrow....or Tues.  
  • So we want to run in states which DO NO use BRSx
• Had PSL 0.3um MAJOR alarm in the 100s (but this is probably tied to windy day).

AS36 I matrix modfied again

Ed, Kiwamu, Sheila (on phone)

The attached is the new AS36 I matrix which was able to stabilize the SRC1 loop, at least for 30 minutes. I changed only one element, SEG3 -> PIT element. It was -0.487 and now -0.6. Before this modfication, Ed and I closed the SRC1 loop with the old element. Seemingly issue was in pitch which ran away very slowly on a time scale of 5 minutes or so. We then opened the pitch loop and manually aligned SRM so that it maximizes the AS90/POP90 ratio. Then, looking at the error signal, we have determined the desired value for the element. No physics involved.

For OPS,

If the SRC1 loop is still unstable or slowly drifts (which should be visible as increased POP90 on the striptool), the first action should be to open the SRC1 loop by simply zeroing the loop gains.

Summary -Day Shift

• The Earthquake was the biggest limiting factor of the day

17:10 Waiting at NOISE_TUNINGS. Losing recycling gain (AS90 over POP90) trace. Turned off SRC loops turned off as suggested by Nutsinee to try and hand align SR2.

17:15 ...never got to try aligning SR2 - LockLoss

21:10 aaaaand we're back. Initial alignment

21:15 Jim W on site.

21:25 Jim headed down to EX to deal with BRS

22:40 Jim headed back from EX. Begin locking.

00:00 Kiwamu has H1. Handing off to Corey

BRSX still down

I came in today to see if I could resurrect BRSX, but I was unsuccessful. I re-enabled the damping at one point to see if it was still working and after a while the damper table seemed to just stop servoing. I also damped the BRS by hand a couple times, but it would ring back up, for no apparent reason. It's all very strange, I'm not sure whats going on.  I am going to turn a couple of sensor correction taps off to keep operators from accidentally engaging the BRS, until I have a chance to talk to Krishna about how to fix BRSX.

For now, I have defined a couple of states for SEI_CONF to help keep the IFO running. There is a WINDY_NOBRSX_V2, that puts ETMX on 45 mhz blends, but keeps the other chambers on their 250mhz blends with low freq sensor correction. There is also a USEISM_MOD_WIND_NOBRSX, if the wind is low and microseism is too high.

IMC not locking

IMC trends showed substantial movement in Pitch from the lock 12 hours ago. Clearing the WFS histories got it re-locked. ASC re-aligning.

MC1 - 7µrad

MC2 - 19µrad

Mid-Shift Summary - Day

• My only chance of hand aligning my way through SRCL to get to NLN was thwarted by a pesky lockloss.
• While trying to relock, MICH alignment got ugly
• While trying to get MICH re-aligned, there was a pretty large EQ in Central America that tripped the arm ISIs
• Untripping ISIs at 1um/s and making preperation to try re-locking. Kiwamu is here addressing the recycling gain/SRCL ugliness.
• winds aren't an issue so far
• PSL dust monitor sounding regularly

Large EQ near El Salvador

≈19:02UTC the wave arrived. All arm ISIs tripped at 19:07.