J. Kissel, for the Relocking / Commissioning Teams

A progress update (picking up from when the future looked bleak), so I don't have to write a giant log at the end. (And to be honest, I'm not staying.)

Executive summary thus far: We're still struggling with the restoration settings after a major front-end computer outtage (like an RCG upgrade). Not because we aren't restoring to a good time, but it's that the "good time" (which has, thus far, been "the last time we were locked in low noise") is not the right time for some settings, especially those that are part of lock-acquisition. These settings will slowly but surely rear their head, and we just have to code them into the Guardian as we go, because they are almost always NOT monitored by SDF because they're part of filter banks under guardian control.

Also, big earthquakes are a huge hindrance to recovery from a rough maintenance day. We should should schedule those for Friday nights.

Here's what happened in the after noon / early evening:

13:00 All models have finished there upgrade and are up and running
      Quickly see that HAM SUS IOP output is OK
      Quickly see that IPC errors from SEI are gone
      Richard immediately to ETMX to finish cabling up ETMY ESD LVLN Driver

13:10 Discover IMC WFS front end model has been mistakenly named ASCIMC on the top-level making all MEDM channels go white (because they're now called H1:ASCIMC-... instead of H1:IMC-...)

13:20 All SEI and SUS "recovered" (i.e. brought to ALIGNED and FULLY ISOLATED)

13:30 PR3 oplev commissioning begins
      h1ascimc front end model naming bug fixed, back up and running
      Discover ITMs are getting a HUGE signal blasted in from ASC
      DAQ restart

13:40 Find ASC loops that are blasting the ITMS, loops turned OFF and history cleared. 
            << LESSON LEARNED -- We should Run the ISC_LOCK DOWN script after a full computer restart 
      
14:00 Discovered ETMX ESD wiring had not been updated for the new LVLN ESD wiring (LHO aLOG 20003)
      Resume wiring up ETMX ESD LVLN Driver

14:20 IMC recovered briefly after trouble with *some* unknown setting has been globally BURT restored

14:22 IMC lost again, because ISS is railing.

14:25 Jim and Dave head to EY to update BIOS settings on fast front-end (For the record, we've chosen NOT to revert the EY fast front-end to slow front-end, we've JUST upgraded the BIOS settings)

14:40 IMC recovered
            << LESSON LEARNED earlier global burt restore restored to a time of full IFO lock, 
               which had the ISS 2nd loop engaged. Without full IFO, it doesn't make sense to 
               have the 2nd loop ON. 2nd loop turned OFF, IMC OK.
      EX LVLN Driver Wiring Complete
      EX Charge measurements launched

15:00 Jim and Dave finish ETMY front-end machine BIO upgrades
      Found New ISI wiener filters are ON (as is standard for any new filter bank) and just 
               feeding STS signals straight to the platform, causing platforms to be very noisy.

15:15 Optical Lever work completes.

15:20 ETMY SUS and SEI fully recovered 
      Begin charge measurements  at EY to confirm SUS health
      Keita begins IMC WFS plant measurement for new DOF5 to reduce 200-300 Hz intensity noise

16:00 Charge on ETMY done -- but discover drive is saturating. (Found out later it was because of the move of the "sumComp" filter from DRIVEALIGN to COILOUTF that was performed yesterday evening [[no aLOG]])
      Begin Initial alignment

17:05 Found SNR for AS 45 Q (during initial alignment of SRY) was really low, even with high-power into the IFO.
            << LESSON LEARNED Discovered there's a -160 dB filter that's ON for full IFO low-noise state, which we *don't* want on during initial alignment. This is *not* included in any Guardian's state request. This *should* be included in both the IFO_ALIGN guardian and the IFO DOWN state.

17:40 Initial alignment complete, beginning full IFO lock acquisition attempt  (Need some hand tweaking of the BS by Evan)

17:45 We had *just* reached some stages of the CARM offset reduction for the first time and then
      Magnitude 5.9 Earthquake - 29km S of Acandi, Colombia
      
17:50 Charge measurements resume (we figure out the saturation issue mentioned above)
      Matt and Hang do some L2P / L2Y measurements on ITMY

18:40 Resume locking

19:10 Make it up to DARM_WFS (not even to RESONANCE where the FULL IFO is at least RF locked on ETMX)
      Found ETMY M0 Bounce mode damping filters were not set correctly, ringing up EY's Bounce Mode terribly
          -- blamed filters disappearing (debunked)
          -- blamed improperly engaged guardian state (debunked)
      It was really, that the last global BURT restore was done *before* the EY BIOS upgrade was finished. So we didn't BURT enough!
          << LESSON LEARNED These filters should *also* be forced into the right configuration, 
             since they are NOT monitored by the SDF system (because there are other parts of 
             the filter module that ARE controled by guardian)

19:45 While having resolved the Bounce Mode damping problem, 
      Magnitude 6.3 Earthquake - 71km SSW of Redoubt Volcano, Alaska


Goodnight everybody. "Things will *definitely* be better in ULTRA LIGO."

Daniel gave me the test rig for the AM stabilized EOM drivers that we should be receiving from Caltech this week.  This allowed me to test that the Beckhoff controls and readbacks work as expected.  I also made a summary screen (ISC_CUST_EOMDRIVER.adl) of these readbacks and controls, which is accessible from the LSC dropdown menu on the sitemap. 

The chassis is labeled "Corner 6", and has 2 unconnected connectors labeled "EOM Driver A" and EOM Driver B". 

The "A" connector controls the 45 MHz channels, and the "B" connector controls the 9 MHz channels. 

The controls and readbacks performed the same for both channels, so I'll only write out the list once.

• "Power ok" readback works
• "Excitation enable" switch functions, although the LED on the test rig seems backwards (I don't know if the test rig is wired backwards, or if the EPICS controls are backwards).
  • When the EPICS switch says "off", the LED is on.
  • When the EPICS switch says "on", the LED is off.
• "Excitation switch" readback works
• "Remote control" readback works
• "RF on" readback works
• "RF control" readback works
  • The readback reports directly the number of Volts.
  • The 45 MHz channel reads high by less than 1% (ex. 6 V input with voltage calibrator read back as 6.04 V)
  • The 9 MHz channel reads low by less than 1% (ex. 6 V input with voltage calibrator read back as 5.99 V)
• "RF set mon" readback works
  • Table of selected values:

  • Voltage input with voltage calibrator	RF Set Mon readback value
    0.1 V	-13
    1 V	7
    3 V	16.6
    5 V	21.0
    7 V	24.0

• "RF set" input works
  • Minimum acceptable slider value is 4.0.  This corresponds to the word 00101000 (LEDs for 32 and 16 on, all others off).
  • Each slider value increment of +0.1 adds 1 (in binary) to the word.
  • Slider value 25.5 corresponds to the word 11111111 (all LEDs on)
  • Slider value 25.6 corresponds to the word 00000000 (all LEDs off)
  • Values larger than 25.6 continue counting up by 1 bit per 0.1 increment
  • Largest acceptable slider value is 27.0, which corresponds to 00001110 (LEDs for 8, 4, 2 on, all others off)
• "RF out" readback works
  • Table of values is identical to the RF Set Mon.

I need to investigate the situation with the "Excitation Enable" switch, but other than that we should be ready to go when the EOM driver arrives, if we decide to install it.

[Jenne, StefanB, Cheryl]

We were having some trouble getting the IMC WFS to converge (WFS came on, looked okay for a while, then started dragging the MC transmitted power down), so we implemented and tested the new global burt restore state, accessible from the ISC_LOCK guardian. 

In the end, the specific problem with the IMC WFS was that the ISC input filters on the M3 stage of the MC mirrors (definitely MC1, maybe others) had gain of zero, so the WFS signals weren't getting through to the optics' outputs. 

The solution we created, which should solve all kinds of problems, is a guardian state that does a global burt restore to a recent set of autoburt snapshots. This state has a date and time hard-coded in the guardian script, although it is easily change-able if we find a more preferable time.  Currently, it is restoring to the autoburts of 28 July 2015, 07:10.  To select this state, you need to go to "manual", since there are no edges to get there from any other state.  When it has finished the restores, it will immediately go to and run the Down state. 

The list of snapshots that are restored is:

• lsc
• asc
• omc
• alsex
• alsey
• iscex
• iscey
• pslfss
• psliss
• pslpmc
• ascimc
• susbs
• susetmx
• susetmy
• susitmx
• susitmy
• suspr3
• suspr2
• susprm
• sussr3
• sussr2
• sussrm
• susim
• susmc1
• susmc2
• susmc3
• sustmsx
• sustmsy
• ecatc1plc1
• ecatc1plc2
• ecatc1plc3
• ecatx1plc1
• ecatx1plc2
• ecatx1plc3
• ecaty1plc1
• ecaty1plc2
• ecaty1plc3

Perhaps though, we should just restore *every* snapshot that is captured by autoburt?

Added 429 channels. Removed 240 channels.

For Maintenance this morning I needed to break the IFO lock, so recorded the steps used today.

• ISC_LOCK  - setting ISC_LOCK to DOWN did not break the IFO lock
• requested IMC_LOCK to OFFLINE, but while IMC_LOCK is managed by ISC_LOCK, ISC_LOCK continues to relock the IMC
• setting ISC_LOCK to PAUSE allowed the change of IMC_LOCK to OFFLINE
• IMC_LOCK in EXC was still trying to put MC2 to MISALIGNED, instead of the requested state of SAFE, so was feeding signals to MC2 until IMC_LOCK was set to PAUSE

- Besty, Jeff, TJ, Cheryl

Serial cables and new software infrastructure was installed to read the status information of the GPS clocks in the end stations. The CNS II clocks are using the Motorola binary format at 9600 baud. We read the @@Ha message once a second. The GPS receiver is reported as a Synergy SSR-M8T. This seems to be an u-blox LEA-M8T in an M12+ form factor and Motorola protocol emulation. Medm screens are available from the auto-generated list.

The duplicated PSL_ERROR channels were also eliminated.

Some charge measurements was done today to check the ESD's after today's changes (see 19994). 
All quadrants of ETMX ESD seems working Ok. (Note, that charge measurements use Hi voltage driver).
We have ETMY measurements corrupted due to changing of the bias gain. The saturation was founded in ETMY ESD during the measurements. Jenne found that gain of bias is 1 while they reduced it last night (see 19978). Seems like this gain was somehow changed back to 1 right before the charge measurements. I don't think it was done by the charge measurement script, nevertheless it is possible. So we see that ESD works Ok, but we have no charge measurements data for ETMY today.
Updated ETMX charge plot is available in the attachment.

IMC WFS inputs have an aggressive 60Hz comb filters. These are not desirable when we want to use these WFSs to control the PSL periscope PZT at around 300Hz to suppress some noise in DARM.

Though it's not clear if these filters are doing anything good for us, just to be on a safe side I moved all comb60 filters in IMC WFS inputs to IMC DOF filters.

Correct setting from now on:

comb60 in FM6 in H1:IMC-WFS_[AB]_[IQ][1234] (e.g. H1:IMC-WFS_A_Q3) are all OFF (they used to be ON).

comb60 in FM6 in H1:IMC-DOF_[123]_[PY] (e.g. H1:IMC-DOF_2_Y) are all ON (there used to be no filter in these).

DOF4 and DOF5 do NOT have comb60.

Maintenance started with prep work before 8am by Jeff, Betsy, & Cheryl getting everything ready to take H1 down in a controlled way and then taking all guardians to a preferred states.

• 7:57:  Not clear how to break lock (ISC Lock Guardian both "DOWNs" did nothing); Hit Offline on IMC Lock & this took the IMC down (but since it was already being managed it immediately went back UP)
• 8:02:  Sprague on site (escorted by Joe)
• 8:05 - :  Cleanroom curtains (hugh)
• 8:09:  Richard checking state of EX global ESD
• 8:10 - 1:00:  RCG Upgrade
• 8:22 - 2:45:  Regenerate EY NEG pump (Kyle)
• 8:24:  Vinny to LVEA to look at tiltmeter & MY to check seismometer
• 8:30:  Bubba to check on EX instrument Air/compressor (had been getting instrument air alarms and trends showed clear drop in signal)
• 8:39:  Staging for PR3 work (Jason and Ed)
• 8:39:  Looking for parts in LVEA a few times (Travis/Betsy)
• 8:45:  Gerardo heading to EY to help Kyle
• 8:58:  King Soft on site to take water samples
• 10:00-11:00:  Magnetometer calibration & PEM photos (Katie B)
• 10:43-11:15:  Installing tubing for 3IFO long term storage containers in LVEA (Bubba)
• 11:15:  GPS/Ethercat cable at EY (Daniel)
• 11:25:  LVEA cleaning (Karen & Christina)
• 11:38:  Praxair will be here for CP4 (#372)
• 12:30:  RCG recovery starting
• 1:00:  ETMx LowVoltage/Low Noise (LVLN) driver work (Richard)
• 1:30 - 2:17:  PR3 Oplev laser swap (Jason & Ed)
• 2:23:  ETMy down for BIOS upgrade reboot
• 3:52:  Andres to MY
• 3:54:  Patrick turned vacuum gauge OFF (it was turned ON earlier today for NEG Pump work).

The all-encompassing RCG upgrade ended up taking up more time than planned, but it's looking like we can start seeing light at the end of the tunnel.  Despite valiant efforts Maintenance Day extended past noon.  There are a few remaining measurements which are going on now, but H1 looks close to being back at a state where it can be returned to locking.

J. Oberling, E. Merilh

We replaced the PR3 oplev laser with one that has been stabilized in the lab, as well as installing the cooler for laser thermal stabilization and an armored 3m fiber; this is in accordance with ECR E1500224.  The serial number of the new PR3 laser is 198-1; the old SN is 191.  The new laser will be left to thermally stabilize for the next 4-6 hours, then we will start looking at spectrograms of the SUM signal during quiet times to determine if we need to adjust the laser power (since the thermal environment of the LVEA differs from the LSB optics lab).  The old laser, since it was showing no signs of failure, just glitching, will be taken to the LSB optics lab to be stabilized.

After talking with the control room we determined the PR3 optic is in a good alignment, so we realigned the PR3 optical lever.

Modified BIOS settings for new faster h1susey computer to more closely match the original computer BIOS settings.  This makes the new computer a bit slower, but it is hoped that the IOP CPU usage will be stable with the new settings.  We should know the results in a few days.

This work required restart of dolphin IPC connected models at EY, which is all except the h1susauxey models.

Kyle, Gerardo

0900 hrs. local 

Added 1.5" O-ring valve in series with existing 1.5" metal valve at Y-end RGA pump port -> Valved-in pump cart to  RGA -> Valved-in Nitrogen calibration bottle into RGA -> Energized RGA filament -> Valved-out and removed pump cart from RGA -> Valved-in RGA to Y-end 

???? hrs. local 

Began faraday analog continuous scan of Y-end 

1140 hrs. local 

Isolated NEG pump from Y-end -> Began NEG pump regeneration (30 min. ramp up to 250C, 90 min. soak, stop heat and let cool to 150C) 

1410 hrs. local 

Valved-in NEG pump to Y-end 

1430 hrs. local 

Valved-out Nitrogen cal-gas from Y-end

1440 hrs. local 

Valved-in Nitrogen to Y-end -> Stop scanning

Rana, Matt, Evan

Back on 2015-07-20, we accidentally rang up the butterfly mode of the EY test mass. We notched this mode out of the DARM actuation path, so that it is not excited by DARM feedback. Additionally, I installed a bandpass filter in mode 9 of the EY L2 BLRMS monitors so that we could monitor the ringdown of the mode.

We looked at 2 hours of data starting at 2015-07-20 11:01:30 Z. Looking at the DARM spectrum from that time, the frequency of the mode is 6053.805(5) Hz. From the BLRMs, we see a nice ringdown in the amplitude. Fitting an exponential to this ringdown, the time constant is 1300 s. This gives the Q of the mode as 2×10⁷ or so. Uncertainty TBD.

Refsignal was brought to -2.10V

In an attempt to make both end stations function the same we installed the Low Voltage Low Noise  ESD driver below the existing HV ESD driver.  Finished pulling the cables and hooked everything up.  I verified that I could get Low Voltage out but could not proceed to HV testing do to site computer work.  I have left the controls for the system running through the LVLN system but have the High voltage cable still driving from the ESD driver directly.  Re testing will continue after software issues are fixed.

Image attached - two DOWN states, two NONE states, two CHECK_IR states - why?  When did this happen?  Who will fix it?

One clue to the issue is that the "top" DOWN when selected will show up in the guardian log, but the "bottom" DOWN does not produce any guardian log entry.

Check out my first entry about this... here.

I tried to turn the gauge back on through the Beckhoff interface by writing '01 02' to the binary field of the FB44:01 parameter but kept getting a response code corresponding to 'Emission ON / OFF failed (unspecified reason)' in FB44:03. Gerardo power cycled it and it came back on.