Current/Turkey-Weekend: (1) Guardian Operations For Last Steps & (2) SRC1

Wanted to make/consolidate a few notes regarding operations as of late & for this weekend

PRC Gain & PR3_REFL_WFS

The last few shifts/weeks we have been waivering on the PR3 (POP vs REFL) WFS steps.  As of this moment, we are bypassing BOTH of these steps in Guardian.  Kiwamu said that if we had higher PRC Gain, we could try for the POP WFS, but the PRC Gain (PR_GAIN_OUT16 on front StripTool) must be 30 or more....it has been hovering at 29, so we are skipping this step.  

Sheila tried to improve the PRC gain by tweaking on the PRC alignment over the last few weeks.  This sounds like a work in progress.  For the weekend, we will ignore these PR3 (POP/REFL) WFS steps.

Beam Diverters

Another step we have been skipping toward the end of the Guardian steps is the CLOSING BEAM DIVERTERS.  We want to keep the POP Beam Diverter open, but perhaps for better performance and consistency for PEM Injections, we have been closing the REFL & AS_AIR Beam Diverters Manually (i.e. Sitemap/LSC/Beam Diverters).  When you close the AS_AIR, it will give you a glitch in DARM, but it's not too scary.

POP90 & SRC1 pit/yaw Loop

There had been some issues with the SRC1 Loop.  Basically it would push the alignment such that the POPAIR_B_RF90 (on the front StripTool) trace would increase (a bad thing).  So, if you see this trace increase, you should:

• Go to the SRC1 Pit & Yaw Loops
• Zero their gains
• Then tweak on the SRM to bring the POPAIR_B_RF90 trace down. 
• Keep the SRC1 gains at zero.

Have not had issues with this for the (3) locks since Kiwamu left.  (Knock on wood.)

Weekly PSL Chiller Reservoir Top-Off

Topped Crystal Chiller with 250mL.  This Closes FAMIS 6498.

(The Sheet is full, & should be given to Jeff B.)

Mid-Shift Status......EY ISSUE!!! (as I started this alog)

Fairly noisy seismically (useism has been holding at 0.7um/s for last 18hrs & winds are at a steady 9mph).

H1 has been also holding at a steady range of 70Mpc for the last 3hrs (even through some short injections by Robert);  Noticeable 

.......

Ugh!  

4:02:  LOCKLOSS:  EY Went Crazy!

As I was typing this alog, H1 dropped out of lock.  There were no noticeable issues (POP90 had been flat for hours, PI Modes were buried, seismic was noisy, but no changes).  However, in addition to the usual HAM6ISI trip, EY's ISI, ETMy, & SUS all tripped.

I reset everyone's Watchdog, but when trying to run ALS, I was getting PDH Alerts (not enough light).  Looked further and found that the ETMy DID NOT come back to a normal state:

• It had many filter banks in a RED state (mainly their outputs OFF), ranging from M0DAMP, to Optic Align (offsets OFF), to L1 LOCK, etc. etc..  I know there were some gains which were also zero-ed.  Too many things to note on here.
• The M0 output channels were also either frozen (for top half ones) or zero for bottom half ones (this was different from ETMx).
• ODC was also in a RED state for ETMy
• I tried to use ETMy's Guardian to restore ETMy (I had to put it in Manual since it was MANAGED by ALIGN IFO).  I tried several states, but the issues noted above never were remedied.  
• I sent texts to Betsy, Jeff, & Richard

So I opened ETMx, and then went through everything that looked different with ETMy.  After a while, ETMy started to look fine.  The ODC even came back green.  At around this point, I talked with Betsy and I snapped a photo of ETMy with my phone to her & at this point it looked OK.  We looked at the safe.snaps, and the Diffs for ETMy looked known & acceptable.  

Additionally, ALS was locking in the Y-arm.  So we are hesitantly going to say we are back, but I hope there isn't a switch, gain, filter, etc. which I could have missed.

OK, back at it.

SDF Updates: SDF GREEN, For Now.

Since H1 has been running suprisingly stable (no PIs rung up [speak of the devil---28 perked up as I typed] & SRC1 has been servoing fine), I went ahead and addressed DIFFERENCES on the SDF.  Going a node at a time, below I list how many DIFFS each had, and also give a summary note about them as well.  Attached you will also find a snapshots for all the DIFFs before I made them GREEN (well, except for SUSETMY...I forgot to snap one).

I reckon some of these will be RED again soon due to changes:  PI's always change, Calibration may change, etc.

SUSIM:  1  Total Diffs

ACCEPTED:  TRAMP for IM3 Pit 

LSC:  6 Total Diffs

ACCEPTED all Gain changes.

ASC:  33 Total Diffs

ACCEPTED:  PRC2_Y Filter difference, POP90 Gain, a few offsets, many matrix element changes, and many ODC channels.

REVERTED (2) TRamps

OMC:  3 Total Diffs

ACCEPTED:  ODC channels

PSLISS:  1 Total Diffs

ACCEPTED:  Ref_signal (has been moving around last few days)

ISIETMX:  9 Total Diffs

ACCEPTED all 9 filter changes (this is due to no BRSx for next few days)---For these Seismic SDFs, we probably want to update these when we are back to nominal environmental conditions and have BRSx back.  Since the SEI Config could change a bit depending on weather & health of BRS's, will we NOT monitor these?

SUSETMX:  1 Total Diffs

ACCEPTED:  GAIN for the L1 Lock Gain of ETMx  (its nominal WAS 0.0)

SUSETMY:  2 Total Diffs  (forgot to take snapshot!)

ACCEPTED:  P2L & Y2L TRAMPS

ALSEX:  3 Total Diffs

ACCEPTED:  These are all "EXTRA_AI" ??

ALSEY:  3 Total Diffs

ACCEPTED:  These are all "EXTRA_AI" ??

SUSPROCPI:  4 Total Diffs

ACCEPTED gain & phase changes

CS ECAT PLC1:  8 Total Diffs

All related to "pico"

CALCS:  1 Total Diffs

Value for Pause

CALEX:  1 Total Diffs (forgot to take snapshot!)

Value for SINGAIN

~1705 - 1715 hrs. local -> Kyle in and out of LVEA -> Measured temps and made adjustments @ BSC8 PT180 bake

"Slowly but surely..."  I'll need BSC8 access in the morning and again tomorrow evening.  

Check out the attached image.  This looks like a step response that couldn't be the result of a real pressure change.  I would guess it has something to do with the shared 24 DC supply between PT180 and PT120B and that pulling the fuse for PT180 changed the supply's load - or something - but nothing related to heating the PT180 hardware.

Ops EVE Shift Transition

TITLE: 11/25 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 22mph Gusts, 15mph 5min avg---> Hoping it doesn't get windier!
    Primary useism: 0.06 μm/s
    Secondary useism: 0.66 μm/s---> this is pretty high.  Hope it stays flat or (obviously drops)
QUICK SUMMARY:

• Arrived to see Kiwamu & Ed taking H1 to NLN eventually.
• PRC Gain at 29, so no PR3_REFL_WFS step in Guardian (must be 30 or more to attempt it.  This trace is the cyan trace on front Strip Tool))
• Appear to still have issues with SRC1 Loop in that POP90 tends to be driven higher.  (when this happens, zero the gain in SRC1 P&Y & tweak SRM to minimize POP90 (bright green trace on front StripTool).
• Had a lockloss and back up now.  Robert is out trying more Injections.
  • Took two attempts to make it to NLN
  • First attempt required BS tweak at Check Fringes & BS/PRM tweaks at LOCK PRMI.
  • Kyle also tiptoed out around BSC8 while attempting lock.
• Jim was here addressing SEI system.  Because we are in a bit of a noisy seismic state (fairly high useism & winds over 15mph), we are almost at a state where the current ISI_CONFIG will be pushed beyond its control.  Currently running at:  WINDY_NO_BRSX_V2 (a new state).
  • Continue to NOT have BRSx (this is due to ring up from yesterday).  Jim may address tomorrow....or Tues.  
  • So we want to run in states which DO NO use BRSx
• Had PSL 0.3um MAJOR alarm in the 100s (but this is probably tied to windy day).

AS36 I matrix modfied again

Ed, Kiwamu, Sheila (on phone)

The attached is the new AS36 I matrix which was able to stabilize the SRC1 loop, at least for 30 minutes. I changed only one element, SEG3 -> PIT element. It was -0.487 and now -0.6. Before this modfication, Ed and I closed the SRC1 loop with the old element. Seemingly issue was in pitch which ran away very slowly on a time scale of 5 minutes or so. We then opened the pitch loop and manually aligned SRM so that it maximizes the AS90/POP90 ratio. Then, looking at the error signal, we have determined the desired value for the element. No physics involved.

For OPS,

If the SRC1 loop is still unstable or slowly drifts (which should be visible as increased POP90 on the striptool), the first action should be to open the SRC1 loop by simply zeroing the loop gains.

Summary -Day Shift

• The Earthquake was the biggest limiting factor of the day

17:10 Waiting at NOISE_TUNINGS. Losing recycling gain (AS90 over POP90) trace. Turned off SRC loops turned off as suggested by Nutsinee to try and hand align SR2.

17:15 ...never got to try aligning SR2 - LockLoss

21:10 aaaaand we're back. Initial alignment

21:15 Jim W on site.

21:25 Jim headed down to EX to deal with BRS

22:40 Jim headed back from EX. Begin locking.

00:00 Kiwamu has H1. Handing off to Corey

BRSX still down

I came in today to see if I could resurrect BRSX, but I was unsuccessful. I re-enabled the damping at one point to see if it was still working and after a while the damper table seemed to just stop servoing. I also damped the BRS by hand a couple times, but it would ring back up, for no apparent reason. It's all very strange, I'm not sure whats going on.  I am going to turn a couple of sensor correction taps off to keep operators from accidentally engaging the BRS, until I have a chance to talk to Krishna about how to fix BRSX.

For now, I have defined a couple of states for SEI_CONF to help keep the IFO running. There is a WINDY_NOBRSX_V2, that puts ETMX on 45 mhz blends, but keeps the other chambers on their 250mhz blends with low freq sensor correction. There is also a USEISM_MOD_WIND_NOBRSX, if the wind is low and microseism is too high.

IMC not locking

IMC trends showed substantial movement in Pitch from the lock 12 hours ago. Clearing the WFS histories got it re-locked. ASC re-aligning.

MC1 - 7µrad

MC2 - 19µrad

Mid-Shift Summary - Day

• My only chance of hand aligning my way through SRCL to get to NLN was thwarted by a pesky lockloss.
• While trying to relock, MICH alignment got ugly
• While trying to get MICH re-aligned, there was a pretty large EQ in Central America that tripped the arm ISIs
• Untripping ISIs at 1um/s and making preperation to try re-locking. Kiwamu is here addressing the recycling gain/SRCL ugliness.
• winds aren't an issue so far
• PSL dust monitor sounding regularly

Large EQ near El Salvador

≈19:02UTC the wave arrived. All arm ISIs tripped at 19:07.

Shift Summary - Day Transition

TITLE: 11/24 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Locking
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
    Wind: ~17mph Gusts, 20mph 5min avg
    Primary useism: 0.15 μm/s
    Secondary useism: ~0.71 μm/s
QUICK SUMMARY:

Ops Owl Shift Summary

TITLE: 11/24 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC

STATE of H1: Meh

INCOMING OPERATOR: Ed

SHIFT SUMMARY: Anamaria left couple of suggestions of things that I can do to help improve the range. I was going to try for an hour but caused a lockloss in the process. Have been having trouble locking all my shift. There're only two ISI configurations I could try without using BRSX (which is still rung up). USEISM_NOBRSXY took me the furthest towards NLN (I was in NLN once with this configuration) but it seems unstable at ~37 mHz.

LOG:

9:08 Lockloss as I tried to engage SRC1 P. Not sure how. I think that's because I forgot to turn on the -20dB filter and turn off the integrator. Sorry. I was going to try moving SR2 and hope that SRM follows.

10:12 Locklosses couple of times during PRMI to DRMI transition. Turned out SRM pitch got stuck in some bad alignment. I moved SRM during PRMI locked then transitioned. ASC was able to engage without issue.

10:18 Lockloss 1 minute into NLN.

10:48 Lockloss the twice at SWITCH_TO_QPDs.

11:13 Lockloss at CARM_ON_TR. I wonder if the useism and wind is becoming a problem.

11:25 Another lockloss at SWITCH_TO_QPDS.

~11:50 Tried to engage BRSX after another lockloss at CARM on TR but it's still rung up. Reverted the configuration to no BRX.

12:02 Tried switching to USEISM_NOBRSXY. It's not in the reccomend list. But with BRSX rung up I kinda run out of options.

          Made through SWITH_TO_QPDS with USEISM_NOBRSXY configuration

12:30 Arrived at NOISE_TUNINGS. Staying here for half an hour.

         While staying here I noticed AS90/POP90 dropped suddenly. I tried several things to improve AS90/POP90. Moving SR2 seems to work. See alog31813

13:08 Run a2l

13:14 lockloss.

13:27 Went to LVEA to toggle noise eater. Checked with LLO ops (Jeramy)

13:50 Kyle to BSC8 to continue his work from yesterday.

14:32 Kyle out

14:41 Lost lock a few times at PRMI_LOCKED and DRMI_LOCKED. Tidal seems unstable. Recently lost lock at SWITCH_TO_QPDS. I'm switching the seismic config back to Windy no BRSX

15:12 Switching ISI config back to USEISM_NOBRSXY after 3 locklosses at FIND_IR. Wind reached 20 mph.

15:39 Lockloss at LOWNOSIE_ASC (IMC-F ran away at 37 Hz). PRMI striptool screenshot attached.

Bake of pressure gauge PT180 (mounted on BSC8 dome) started, CDS indicated pressure value for PT180 is invalid until further notice

(see WP #6332)

~0605 - 0630 hrs. local -> Kyle in and out of LVEA 

Attempted to de-energize PT180 by removing the fuse at terminal #111 in the Y-arm vacuum rack.  Noted that the old PT180A signal wiring had been lifted from the terminal strip and was deliberately zip-tied out of the way.  The cabling to PT180B was still landed having only the black conductor fused at terminal #111.  Curiously, the indicated pressure value on the MEDM screens did not change as the result of removing this fuse.  Next, I removed the external gauge electronics, isolated PT180 from the site vacuum volume via closing the 1 1/2" metal-sealed isolation valve and then opened the pump port valve so as to pump the PT180 volume with the locally mounted turbo.  Heating was started with the initial variac settings of 20%.

Unfortunately, we are using the BPG402-SE version of this gauge - where the "S" indicates "short" nipple.  This limits the gauge bake temperature to a modest 80C as measured at its CF flange as the internal PCB can't be thermally radiated at greater temps.  This may be a significant limitation as the current theory is that water vapor is the source of the undesired gauge behavior and a relatively short duration bake at relatively low temperatures won't likely remove very much water.  (I can cheat a little and adjust the non line-of-site surfaces to much higher temps.)  

SUMMARY:  

Disregard the indicated pressure value shown for PT180 as it is not valid (the value at the time of gauge de-energization is latched in Beckhoff?) 
Also, I will need access to the LVEA later this evening and twice tomorrow, Friday, to make measurements and adjustments. 

ASC SRC is doing weird thing to SR2

As I was sitting at Noise Tunings I noticed AS90/POP90 suddenly dropped. First I tried adding some offsets to ASC SRC pitch. I couldn't make a different. Moving SRM also seems to have no effect but moving SR2 I was able to improve AS90/POP90. But something kept dragging SR2 back to where it was in both pitch and yaw (bad place). Anamaria mentioned that she turned off SRC during the last lock strech so I tried and SR2 suddenly came back to its happy place. I was able to put AS90/POP90 back to where it was.

Ops OWL mid-shift summary

Kept losing lock during switch to QPDs. I wonder if useism+wind is a problem now (useism has reached 1 um/s while wind is ~10 mph). It seems to help the detector stay longer in the state if I sit at TR CARM a little. I will try to stay there a little longer.  Switched ISI config to USEISM_NOBRSXY and was able to move on.

Indicated LN2 level of CP4 sporadic and of questionable validity

We have known for some time that the indicated level of CP4's LN2 fluctuates too rapidly to be physically likely and is more probably due to some transient GN2 phenomenon which effects the validity of the differential transducer.  Occasionally this indicated level will deviate enough as to enter into the alarm region and will then get the attention of the CR operator (Corey just asked about this).  I guess we'll need to investigate this more seriously.  Chandra is aware of this issue but is gone until next week.  

The Vacuum Team gets notified via automated text messages of these alarms.  

Until this gets resolved, CR operators only need contact the Vacuum Team Members if CP4 stays in alarm for a prolonged period (say hours). 

Note to self:  

Confirm that the valve that routes LN2 from the dewar through the Ambient-Air vaporizor is fully closed.