Jeff K, Dave:

For each IOP model:

1) if safe.snap and OBSERVE.snap were links to separate but equal files, changed OBSERVE link to point to safe.snap file

2) if safe.snap and OBSERVE.snap were links to separate and different files, made safe.snap = OBSERVE.snap and changes OBSERVE link to point to safe.snap file

only h1iopoaf1 needed no changes. 13 IOP models matched 1) and 16 IOP models matched 2) making 30 total IOPs.

Following my discovery that h1iopiscey was not setting its DACDT to On until last week, today Jeff and I turned on h1iopiscex's DACDT.

So far its looking promising following the replacement of sw-ex-h1daq Netgear by Brocade. 24+ hours and no further CRC errors. We should know more definitely by the end of the week.

BTW: we were able to see corresponding switch port errors on the Netgear using LibreNMS plots, details in the CRC wiki page  This gives us confidence that replacing the switch will fix this error.

[Georgia, Jenne]

• In prep for the measurement, at around 11:50am local we turned on the ETMY bias with 9 V (a little less than the 9.5 V value that ETMX uses).  There is a lot of low frequency noise added, so we're sat here for ~7 min in case the data is useful, but then we're trying the other sign of the bias.  We don't think that we need lower noise for our measurement (exciting the mode with various ESD patterns on different optics) though, so it should be fine either way. 
  • Keita points out that maybe we should think about grounding the ETMY bias input, since we aren't using it at all for lock acquisition.
• We were working on exciting the mode (not succesful yet), but lost lock.  I suspect the lockloss is unrelated to our work, since none of the PI modes were any higher than they had been the whole lock, our excitation hadn't changed in a long time, and Robert had said that he was planning to do some laser vibrometer work requiring contact with some HEPI piers.  So, I suspect that the lockloss was a combo of Robert's work and the fact that we're in EQ mode. Robert and crew were still in the H2 part of the LVEA, hadn't gotten near HEPI piers yet.
• Acquisition guardian change: PRMI_ASC_OFFLOADED now turns off the PRMI ASC filter banks that were in use, prior to our trying to acquire DRMI lock. It looked like we were still sending signals out to the optics after we had lost PRMI when the SRM came back into alignment, which probably didn't help DRMI acquisition (although it did catch by itself).
• We are going to excite this time in the damping filter for PI mode 24, since then our RMS monitor won't be directly contaminated by our excitation, and will just monitor the actual peak height.  New channel: H1:SUS-PI_PROC_COMPUTE_MODE24_DAMP_EXC, with a narrow bandpass in the awggui. 
  • Previous channel: H1:SUS-PI_PROC_COMPUTE_MODE24_BP_EXC.
  • Had also tried exciting using the OMC deacon infrastructure, but that's not really what we want to be using for this identification measurement.  We'll re-look at that when we're ready to implement deacon alignment for the OMC.
• To be continued in a comment by Georgia....

Per WP 8256, took SEI systems down at the Corner and independently at Endy, to check the fluid accumulators' gas charge.  Not one accumulator needed charging in the corner station nor at EndY.  Did not attempt EndX as Pcal work was going on.

Then EndX will not take very long, maybe 1 hour and we'll do that as soon as tolerable.  I'll venture that the 7 accumlators at EndX will be fine but that venture will likely be all it takes to insure that at least one will need charging so better plan on checking.

TITLE: 06/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 10mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.06 μm/s 
QUICK SUMMARY: No immediate issues.

TITLE: 06/26 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
LOG:

THe 48Hz peak is "breathing" rather prominently in DARM.

12:36 UTC Latency is 7.06 secs and Trig Dur is .512 - It's a keeper.

7:45UTC After the SC outputs on Y Arm settled back down I made the swit5ch back to nominal windy

TITLE: 06/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Just posted an alog summarizing post-LONG maintenance Day recovery, locking, and......MORE EARTHQUAKES!  (so please see alog 50198)
LOG:

• MY Vacuum work continues, GVs closed & Maintenance continues.
• 22:52 Baluns swap (Marc)
• 23:15 LVEA sweep!
  • 0:26 Sweep complete!
• 23:35 RF work done (Dick & Matthew)
• 23:48 EX Pcal work finishing up (Niko)
• 0:12 MY VAC work complete (Chandra, Kyle)
• 0:26 Both end stations swept
  • EY entry door did not let me in with FOB.
• 0:31-1:07 Initial Alignment
• 0:34 SEI Config back to WINDY
• 1:08 Locking Begins
  • PRMI looked pretty flat w/ small flashing, so tried CHECK MICH FRINGES, but it looked good.
  • PRMI looked even worse after my small tweaks to BS after the FRINGES check.....BUT: went to lock PRMI & it looked more dead/flat than last time....so I tried a few tweaks with PRM....And then BOOM:  PRMI locked up!  I tried a few tweaks of PRM & BS, but didn't get too much improvement (so I went to PRMI_ASC and let ASC optimize).
  • 2:10 NOMINAL LOW NOISE!
• 1:26 TCS Chillers FAMIS check (Cheryl)
• 2:25 LARGE EQ noted on Verbal form Japan.  
  • S&P waves must have been to much, because these are what dropped me out before I could do anything.
• 4:00 Reattempting locking.  Many locklosses around FIND IR.
  • 4:45 Tried lowering Fiber polarization for X & Y (marginal improvement).
  • 5:00 Finally making it past CHECK IR after the earthquakes.  (EQ band [0.03-0.1Hz] was down to about 0.1um/s when I was finally able to make progress.)
  • 5:27 (while taking H1 to NLN) 6.4 Coasta Rica EQ on VERBAL....recommendation is LARGE_EQ_NOBRSXY....I'm wanting to save this lock, so trying just EARTHQUAKE state & crossing fingers.
    • Did not see on USGS for many minutes, but saw it on this website.
    • 5:35 see on nuc5's 0.1-0.3Hz band first.....well, maybe on CPSFF a few minutes earlier.
  • 5:42 NOMINAL LOW NOISE!!
  • 5:47 OBSERVING!!  (After accepting SDF diffs!)
• 6:06 GRB Alert received & I started looking thru procedure.
  • 6:08 LLO Operator was faster than me and contacted me via teamspeak to confirm if I received it.
  • 6:10 Determined the Trigger Duration was larger than 1.024sec (it was 2.048sec), so IGNORING!
  • I relayed note about ignoring GRB in the TeamSpeak chat window in case Virgo is watching.
• Leaving H1 in EARTHQUAKE state  (and OBSERVING) as I hand off to Ed.

TITLE: 06/26 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    Wind: 10mph Gusts, 7mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

5:47UTC (10:47pmPDT):  Finally back to OBSERVING!!!

Most of tonight's shift H1 has been down (first two hours was due to today's Long Maintenance Day & after that it has been trying to lock while battling a shaking earth!).

Attempt #1:

Once we were given the keys to H1, worked on getting H1 back to OBSERVING.  From 0:30-2:10utc there was an a quick alignment and then running through locking.  Unfortunately, as we were going through SDF Diffs, received notification of an incoming EQ from Russia (Verbal said it was from Japan).  Either way, the S&P waves were big and fast and they took us down after being at NLN for only a few dozen minutes (no observing!!).  Attempted to transition to EARTHQUAKE, eventhough EQ Plot recommended LARGE_EQ_NOBRSXY (which would prob break lock!).

2:30 - 4:00UTC Waited out the EQ.  Did not attempt locking!

Attempt #2:

• 4:00 - 5:00 Tried locking, but still couldn't get past CHECK IR (EQ band was above 0.1um/s).
• 5:00: finally make it past CHECK IR!  And proceeded through ISC_LOCK.
• 5:27 Another EQ!!!!  6.4 from Costa Rica which recommended LARGE_EQ_NOBRSXY again!  UGH!!!!   Since I was in the latter steps of ISC_LOCK, and wanted to try and stay locked, I opted to, once again, only go for EARTHQUAKE, because I did NOT want to break lock!!!! As I watched all earthquake signals AND ISC_LOCK progress, I crossed my fingers.  [see 2nd attachment for screenshot of everything I was looking at for this EQ!]
• 5:47 made it to Observing!!  (even with the Earth A'shakin'!)
• And here we stand.  I am still in the EARTHQUAKE state (which appears to have saved the day/night!)

SDF Diffs:  [see 1st attachement!]

• For attempt #1 I was only able to address a few diffs.  Called Hugh to take care of the HAM6 ISI CPS threshold which was counting down (this required a Reset on the WD screen...this was due to baluns work).
• Georgia ACCEPTED some ASC diffs, but these came again during Attempt #2.  Wondering if we should UNMONITOR them instead.
• Jason warned me of a teeny tiny decimal place diff for PSL ISS.  I ACCEPTED, but once again, it came up on my 2nd attempt.  Should we UNMONITOR?
• Then there were a handful of PLC related diffs...most (but not all) these diffs were related to Beckhoff stuff, specifically rotation stages (for PSL, TCS, etc.).  I ACCEPTED these.

Where We Stand:

We are still rung up from the Costa Rica EQ, so Ed is going to leave H1 in EARTHQUAKE state.  (our range doesn't seem to mind and has decent [& increasing] range).

During some SDF cleanup work today I noticed that the OBSERVE.snap file for h1iopiscey was modified Monday evening 17th June to permit the DAC Duotone loopback to be ON. 

Following the kernel panic of h1iscey that evening, the safe.snap had turned the DACDT ON. Transitioning to the OBSERVE.snap reference raised a sdf-diff, because this file had this setting as OFF. The operator accepted the new ON setting to transition to observation mode.

This channel is not in the DAQ, but a rudimentary trend can be obtained from the hourly autoburt (see below). This shows that the DAC DT loop back has been OFF since the start of O3, transitioning to ON Monday 17jun2019.

I'll follow up with the calibration group as to what this setting should be.

I will also work on making all IOP SDFs having only a safe.snap file, with OBSERVE.snap being a symbolic link to it.

• 17/14:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 OFF
• 17/15:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 OFF
• 17/16:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 OFF
• 17/17:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 OFF
• 17/18:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 OFF
• 17/19:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 ON
• 17/20:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 ON
• 17/21:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 ON
• 17/22:10/h1iopisceyepics.snap:H1:FEC-93_DACDT_ENABLE 1 ON

WP8247 upgrade EX DAQ network switch to Brocade (Ruckus ICX-7150-24)

Dave:

The upgrade of the switch itself went as planned. Problems arose when I activated a port on sw-ex-h1fe to connect the new switch to the ADMINLAN.

After having previously done this upgrade at EY in March of this year, the switch swap-out went smoothly. With the proper tools (e.g. right-angled screwdriver), removing the obscured rack screws went well. Removing the side of the rack (with key) and the left 'ear' on the outgoing Netgear and the incoming Brocade allowed the switches to be retracted into the rack and removed though the side.

Powering down the Netgear sw-ex-h1daq did not cause any other systems' DAQ data to go bad. Once the Brocade sw-ex-h1daq was started, all the EX front ends and the pem at MX recovered their DAQ data.

To connect the new switch to the ADMIN LAN we are currently running an ADMIN ethernet cable over from the local h1fe switch (the tagging of this VLAN from the MSR is not currently working). I converted a slow controls port to an ADMIN port (0/7) on sw-ex-h1fe for this purpose. Unfortunately this was a tagged ADMIN port (from when we used to pair front end FE-LAN and IPMI on a single port) which caused some network issues mentioned in other alogs.

I'll keep this workpermit open to cover fixing the tagging of the ADMIN LAN from the MSR to the end station switches.

Beckhoff code changes:

Daniel, Patrick, Hugh:

As reported elsewhere, corner and end station Beckhoff restarts for new code were made. The Beckhoff SDF faux-models required restarts.

Due to time constraints and other activities requiring the DAQ, these two items were postponed.

• ISI and SEIPROC model updates
• DAQ Restart

Note that this means the new slow controls channels created today are not in the DAQ.

After today's Long Maintenance Day, handed the keys to H1 at 0:30utc (5:30pmPDT).  Cheryl started an alignment & that was straightforward.  Then went for locking, and for the most part that was fine.  At 2:10utc (7:10pmPDT) we made it back to NOMINAL LOW NOISE!

While going through some SDF diffs, H1 had a few glitches, and then 20min later, got notification of an incoming earthquake (6+ from Japan & the S/P waves were large).  EQ Response plot recommeded going to the LARGE EARTHQUAKE, but since I didn't want to lose lock, I thought I would try EARTHQUAKE.  Shortly after this transition or at the same time, we saw the S & P waves.  Then we lost lock at 2:29utc (7:29pmPDT)!

Now it's time for lunch.  And missed out a few minutes of OBSERVING!  But will give it another go after the earth settles down.

Here are some MC_F spectra.