Jenne found that some of ALSX related signals goes funny.

I found that there are more spikes/glitches in ALSX laser than ALSY laser (1st attachment, the scale of the plots are set to show center +-1.5%).

However, this was always the case from the start of O3 (2nd attachment). ALSX still looks kind of fishy but we need to study it more.

TITLE: 08/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
    Wind: 17mph Gusts, 13mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY: IFO recovery is ongoing

TITLE: 08/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ed
SHIFT SUMMARY:  Commissioning all shift attempting to recover locking.  Still in progress.
LOG:

16:17 Still unable to get past DRMI ASC

16:18 Commissioners to LVEA to measure IMC open loop gain

16:29 Commissioners out

17:26 Jason to optics lab

17:57 Jason out

18:36 Niko and Ethan to PCal lab

18:59 Niko, Ethan out

21:36 Niko, Ethan to PCal lab

21:42 Adrian, Matt to CER moving PEM cables

21:43 NIko, Ethan out

21:47 Adrian, Matt out

Betsy, Rahul

We have been trying to look through and analyze the spectrum of the ETMX coil drivers from 1 -100Hz, to help investigate today's lock loss at LHO. The attached files shows the DTT screenshots of the coil drivers for the ETMX and ITMY (which had a template from 2018). ETMX did not have any slow channel template (DQ), due to which we were unable to look into the history, for the 16k channels we cannot go back in time and look for trends. Now we have created a fresh DTT template to see the spectrum of the ETMX coil drivers, the location of which is given below,

/ligo/svncommon/Sussvn/sus/trunk/QUAD/2019-08-15_2102_H1SUSETMX_OSEM_ASDS.xml

After looking through the spectrum of the coil drivers and comparing through the past data, we could not find anything out of order over here.

As part of the ongoing investigation in H1 locking issues I took a look at the PSL.  I noticed two things:

1) After a lockloss and the resulting disabling of the ISS Second Loop, the Inner Loop Percentage Diffraction was very low, <1%.  This was causing the ISS Inner Loop to in turn oscillate, unlock, relock, and repeat.  I changed the ISS Ref Signal from -2.05 V to -2.02 V.  This brought the Percentage Diffraction back to ~2.4%; we want it between 2% and 3%.  I accepted this change in both the safe.snap and OBSERVE.snap SDF files.  This had no effect on the locking behavior of H1.  *add red herring pic here*

2) As noted previously, the FSS RefCav transmission is low (TPD is ~2.2 V).  I was unable to improve it much during this last Tuesday maintenance; was hoping that some more stable weather would improve it, as the TPD took a dive late last week and over the weekend as the storm system moved through.  So far the TPD is holding steady at ~2.2V, so an enclosure incursion will likely be necessary to recover back to our usual RefCav transmission levels.

J. Kissel, B. Weaver

While down in the weeds of FSS oscillations and ALS issues, Betsy and I quickly looked at all of the Binary IO states of suspension which are toggled in the LOWNOISE_COIL_DRIVERS ISC_LOCK state -- i.e. the high-level acquisition state at which Corey reports regularly losing lock last night (see LHO aLOG 51288). We looked at the read-backs (e.g. H1:SUS-ITMX_BIO_L2_MON), and also toggled the state from its regular acquisition state to its low noise state, which are the following:
in LOWNOISE_COIL_DRIVERS:
            optics   = ['PRM', 'PR2', 'SRM', 'SR2', 'BS', 'ETMY', 'ETMX', 'ITMY', 'ITMX']
            stage    = ['M3',  'M3',  'M3',  'M3',  'M2', 'L2',   'L2',   'L2',   'L2']
            newState = [ 3,     3,     3,     3,     3,    1,      1,      1,      1]

At times in the past, we've seen that either
    - passively, the readbacks don't match the requested state for one or two of the coil drivers
    - actively, when switched, the readbacks don't respond, or don't respond as expected.
(see e.g. LHO aLOGs 40517, 39880, 39539)

We see neither of these things -- all readbacks and switching behave as expected.

J. Kissel for the entire staff

One of the suspicions of one of the things going wrong (we suspect 3 things), is that the coil drivers are glitching. 
The suspicion arises because one of Corey's consistent places of lock loss last night was the transition to LOWNOISE_COIL_DRIVERS. 
As such, I've take a time when the FSS was freaking out for hundreds of seconds (a sadly common bunch of lemons that inhibits iteration on problems higher in the acquisition sequence a few times a day) -- so we know there is no ISC drive to the suspensions -- and searched for glitches in the analog voltage / current monitors for all the QUADs.

I find no evidence for QUAD coil or esd driver glitching for 300 seconds surrounding 2019-08-15 17:38 UTC.

Attached are NDSCOPE trends of the UIM (row 0), PUM (row 1), and ESD (row 2) driver monitors, against (row 3) 
    - (col 0) the transmitted power in ALS X (whatever is glitching is showing up quite clearly in all X arm sensors and in ALS DIFF sensors, which kills ALS while we're going through DRMI acquisition -- this we think is problem 1 going wrong), 
    - (col 1) the L2 coil driver state (always going to the acquisition state 2 in these plots)
    - (col 2) the PSL FSS FAST (PZT) MON indicating that the FSS has railed (railed is +13 V, OK is ~0 V)
    - (col 3) the ISC LOCK guardian lock state (to show that we are DOWN)

S. Dwyer, J. Kissel, K. Merfeld

Exploring one avenue of why we might be having so much trouble with lock acquisition starting last night, we've measured the IMC open loop gain with the IFO held in DOWN (with the IMC locked). It's fine -- crappy picture of transfer function attached.

J. Driggers, S. Dwyer, J. Kissel, B. Weaver

In our continued broad sweep of understanding what's going wrong with the IFO, we found that SR3 was unexpectedly moving by ~14 urad (measured at the top mass), ~6 urad (measured by the optical lever) just after lock loss, and then restoring its position after the completion of the ISC_LOCK guardian's READY state. 

We traced this down to the Lockloss Trigger (see initial release aLOGs in LHO aLOG 44838 LHO aLOG 45061, and ) being errantly enabled on all stages of the SR3 -- which normally does NOT receive any ISC control, so there's no need. The move was because the SR3 CAGE SERVO (unique to SR3) uses the M1 DITHER PIT OFFSET (H1:SUS-SR3_M1_DITHER_P_OFFSET) as its actuator, which is upstream of the M1 ASC trigger. That cage servo is typically outputting 32 DAC counts ("should be" 85 urad in slider units ... but apparently it's not that much), so halting that output via the lock-loss trigger moved the suspension, and then it was virtually immediately restored upon clearing the trigger in the DOWN state.

Turns out this had been happening since the development of the trigger circa Fall 2018, we just never noticed.

We've now disabled (set of OFF, or 0.0) all of the following channels:
    H1:SUS-SR3_${M1,M2,M3}_TRIG_${LSC,ASC}_ENABLE
and accepted in the SR3's safe and OBSERVE.snaps, so this should not happen again.

#RedHerring

Bubba is helping us raise the Pcal lab (room #166) temperature by ~9F. Our plan is to lower the temperature again and measure the responsivity ratio of our Working Standards during the transition. As far as we can tell, this temp change should only affect the Pcal Lab, but if you notice the temperature rising in other areas of the site, please let either Bubba or myself know.

TITLE: 08/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 2mph Gusts, 0mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:  Attempting to relock after Corey gave up mid-last night.

TITLE: 08/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Travis
SHIFT SUMMARY:

Frustrating shift with H1 down the entire shift.  Left in the READY state at 12:10utc/5:10amPDT & with Observatory Mode = CORRECTIVE MAINTENANCE.

• Alignment looked OK. 
• ASC (for most part) was not much of an issue as long as you babysat it at certain steps. 
• Main issue was H1 could never get past LOWNOISE_COIL_DRIVERS.
• Ran an alignment.
• Later in the shift, the Green X-arm had trouble staying locked half of the time (eventhough its alignment as been looking good before/after an INITIAL ALIGNMENT).

LOG:

• 7:15 Continuing Locking with this my first Lock#1
  • Hung out at ENGAGE DRMI ASC & OFFLOAD DRMI ASC for a while.
  • Lockloss at while waiting for signals to converge at ENGAGE SOFT LOOPS.  
    • Did not see an ERROR signal ring up & so I was not able to run the Gain Reduction Script (from alog#51282).
    • I did see CONTROL and ERROR signals get bigger on the wall Strip Tool
    • Not really sure how to 
• 7:50 Lock#2
  • Lockloss OFFLOAD_DRMI_ASC
• 8:02 LOCK#3
  • lockloss at LOWNOISE COIL DRIVERS
• 8:42 LOCK #4
  • paused at LOWNOISE ASC, and then
  • lockloss (once again) at LOWNOISE COIL DRIVERS, and this definitely happened during one of the steps (around PR3 
• 9:17 LOCK#4
  • lockloss at LOCKING ALS
• 9:19 LOCK#5
  • lockloss at LOCKING ALS
• 9:23 LOCK #6
  • lockloss while trying to go through the LOWNOISE COIL DRIVERS steps via command line (not sure I did this correctly).
• 10:02 Lock #7
  • Actually, there were multiple locklosses at LOCKING ALS. 
  • Problems for X-arm at LOCKING ARMS GREEN:  Flashes were above 1, but X-ARM would lock, but had a noticeable oscillation which would eventually break arm lock.  Unlocked and tweaked...and this seemed to calm X-arm down and then be able to move on.
  • Lockloss at ENGAGE SOFTLOOPS
• 11:33-11:58  INITIAL ALIGNMENT
  • X-arm has issues remaining locked, which makes alignment frustrating.  It looks fine with transmitted nicely over 1.0, but it would randomly drop lock after a few minutes while waiting for signals to converge.
• 12:00 Locking post Alignment:  Continue to see X-arm drop out of lock randomly even with a nice alignment, and then multiple locklosses around LOCKING_ALS/FIND_IR
• 12:09 Leaving H1 in the READY state for the Day reinforcements.

Observatory Mode = CORRECTIVE MAINTENANCE due to H1 having problems.

• Have been able to get through ASC steps past LOWNOISE ASC, but repeatedly (3x) had a lockloss at LOWNOISE_COIL_DRIVERS.  
• Since the above state is after getting to 37W, I'm guessing alignment should be OK.  
• Tried running through LOWNOISE_COIL_DRIVERS by hand, but not sure if I did it correctly (had a lockloss nonetheless).
• At any rate, will try this one more time (each attempt takes ~45-90min to get there), and if there's another lockloss, I'm inclined to try an alignment (with hesitation), or wait for the day crew.

Have had a 3 attempts dashed by locklosses before getting through various ASC steps.....   Will keep making more attempts and see if I can try to employ the script (alog #51282) to stamp down CONTROL signals when they get ugly (have not tried that yet).  I reckon I will if I keep running into a wall, I'll run an alignment (although the arms & DRMI look fine).

Here are notes on the 1st lock:

7:15 Continuing Locking with this my first Lock#1

• Hung out at ENGAGE DRMI ASC & OFFLOAD DRMI ASC for a while.
• Lockloss at while waiting for signals to converge at ENGAGE SOFT LOOPS.  
  • Did not see an ERROR signal ring up & so I was not able to run the Gain Reduction Script (from alog#51282).
  • I did see CONTROL (attachment#1) and ERROR (attachement #2) signals get bigger on the wall Strip Tools (unfortunately, I didn't get a snap of the ASC ndscope....
  • But on the ASC ndscope I didn't see anything getting big/noisier.

TITLE: 08/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 18mph Gusts, 15mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:  H1 DOWN & trying to lock.

Since we have been down for 90-min I am going to mark this time on Observatory Mode as CORRECTIVE MAINTENANCE.

Ed had been trying to reacquire, but has been dropping out at various spots (green arms look good, PRMI locks well, from what I've seen since he left). 

He mentioned if I try to run an Alignment, this could be an involved process.....so I'm going to read alogs and keep on acquiring with what we've had for the for the last 2hrs (since 5:21utc).

NOTE:  While waiting here at ENGAGE_DRMI_ASC, received IMC_LOCK notification for "IMC WFS not centered".  Odd...and just something I wanted to note.

I measured the output power of the recently-removed SQZ NPRO laser SN 8552.  The laser settings were found as last set by the SQZ team:

• NPRO Crystal Temp: 31.68 °C
• Diode Current: 2.025 A
• Diode Temp: 20.21 °C

I set the laser settings to those specified in the original data sheet; the set value for the NPRO Crystal temp is slightly different from the actual value, and the power supply reports both numbers so I recorded the actual crystal temp as well.

• NPRO Crystal Temp: 25.10 °C
  • Actual Temp: 25.11 °C
• Diode Current: 2.15 A
• Diode Temp: 20.20 °C

I allowed the laser to warm up for ~5 minutes before taking the first power measurement.  For the measurement I used an Ophir VEGA power meter with an Ophir PD300-3W head.  After the intial reading, the power was slowly increasing, so I waited until it stabilized, noting the power measured a few times over the duration:

• Intial measurement, t0: 0.925 W
• t0 + 10min: 1.015 W
• t0 + 25min: 1.054 W
• t0 + 35min: 1.055 W

The data sheet reports an output power of 1.05 W using the initial settings, so this laser seems fine from an output power standpoint.  I took one last reading of the laser settings to see if things had changed since I started; they had, albeight slightly:

• NPRO Crystal Temp, actual: 25.12 °C
• Diode Current: 2.149 A
• Diode Temp: 20.18 °C

Since the NPRO only had a couple hours to acclimatize to the optics lab environment, I'll let it sit overnight and repeat these measurements tomorrow to make sure everything is unchanged.  As of right now though, the power out of the NPRO matches the original data sheet.

We noticed some curious jumps in the CLF power as measured by the CLF REFL PD. These jumps are also visible in CLF launch and rejected monitors. They are not visible in any of the other PDs on ISCT6 that are not associated with the CLF. See attached plot 1. We see coherent jumps in the RF output power readback of the 200MHz AOM driver. See plot 2.

The jumps in the CLF power are as large as 20%, whereas the RF power to the AOM only jumps by 5% or less.

Starting at ~9:31 utc have noticed elevated levels for DARM BLRMS bands:

• 500-1kHz
• 1kHz-2kHz
• And higher as seen on DARM spectrum

The noise went off and on for 45 min (so far) and looks to start/stop suddenly and then last on the order of a minute and over 10mins (attachment#1 is the DARM blrms where you can see the steps up in noise over most of the 45min.  Time periods for some of these are about:

• 9:50-10:02utc  Attached is a DARM spectra (w/ quiet reference)
• 10:05-10:06 this was quieter than above and not quite as broadband, see attachment

NOTE:  H1 range did not take a hit during this time.  It's been almost 2hrs since this noisy period last occurred.

Following on from last week's Round 2, at Keita's request I took a series of beam profiles between ALS_M11 and ALS_M12 (ALS_M12 is the bottom periscope mirror, so this is the green beam as it goes into WBSC10 (as close as I can get to it, at least)).  Using the ALS_M11 mirror as the reference point, I took 4 profiles along the path; the profiler sensor was at 0° for these measurements, and the distances are from ALS_M11 to the profiler's sensor:

I've attached a beam profile of the beam at the farthest extent from ALS_M11 that I could get the profiler without bumping ALS_M12 (this is the final data point in the above table).  All of the profiles taken had this shape to it.  To my eye it looks like the top half of the profile is compressed versus the bottom half.

In addition, I rotated the profiler sensor by +45° at each data point to check for any gross off-axis ellipticity.  I've attached an example of this, taken at the same location as the first attachment.  The beam gets more round when the sensor is rotated, showing no gross off-axis ellipticity; all sensor-rotated profiles showed this shape.