J. Kissel, B. Weaver

While down in the weeds of FSS oscillations and ALS issues, Betsy and I quickly looked at all of the Binary IO states of suspension which are toggled in the LOWNOISE_COIL_DRIVERS ISC_LOCK state -- i.e. the high-level acquisition state at which Corey reports regularly losing lock last night (see LHO aLOG 51288). We looked at the read-backs (e.g. H1:SUS-ITMX_BIO_L2_MON), and also toggled the state from its regular acquisition state to its low noise state, which are the following:
in LOWNOISE_COIL_DRIVERS:
            optics   = ['PRM', 'PR2', 'SRM', 'SR2', 'BS', 'ETMY', 'ETMX', 'ITMY', 'ITMX']
            stage    = ['M3',  'M3',  'M3',  'M3',  'M2', 'L2',   'L2',   'L2',   'L2']
            newState = [ 3,     3,     3,     3,     3,    1,      1,      1,      1]

At times in the past, we've seen that either
    - passively, the readbacks don't match the requested state for one or two of the coil drivers
    - actively, when switched, the readbacks don't respond, or don't respond as expected.
(see e.g. LHO aLOGs 40517, 39880, 39539)

We see neither of these things -- all readbacks and switching behave as expected.

J. Kissel for the entire staff

One of the suspicions of one of the things going wrong (we suspect 3 things), is that the coil drivers are glitching. 
The suspicion arises because one of Corey's consistent places of lock loss last night was the transition to LOWNOISE_COIL_DRIVERS. 
As such, I've take a time when the FSS was freaking out for hundreds of seconds (a sadly common bunch of lemons that inhibits iteration on problems higher in the acquisition sequence a few times a day) -- so we know there is no ISC drive to the suspensions -- and searched for glitches in the analog voltage / current monitors for all the QUADs.

I find no evidence for QUAD coil or esd driver glitching for 300 seconds surrounding 2019-08-15 17:38 UTC.

Attached are NDSCOPE trends of the UIM (row 0), PUM (row 1), and ESD (row 2) driver monitors, against (row 3) 
    - (col 0) the transmitted power in ALS X (whatever is glitching is showing up quite clearly in all X arm sensors and in ALS DIFF sensors, which kills ALS while we're going through DRMI acquisition -- this we think is problem 1 going wrong), 
    - (col 1) the L2 coil driver state (always going to the acquisition state 2 in these plots)
    - (col 2) the PSL FSS FAST (PZT) MON indicating that the FSS has railed (railed is +13 V, OK is ~0 V)
    - (col 3) the ISC LOCK guardian lock state (to show that we are DOWN)

S. Dwyer, J. Kissel, K. Merfeld

Exploring one avenue of why we might be having so much trouble with lock acquisition starting last night, we've measured the IMC open loop gain with the IFO held in DOWN (with the IMC locked). It's fine -- crappy picture of transfer function attached.

J. Driggers, S. Dwyer, J. Kissel, B. Weaver

In our continued broad sweep of understanding what's going wrong with the IFO, we found that SR3 was unexpectedly moving by ~14 urad (measured at the top mass), ~6 urad (measured by the optical lever) just after lock loss, and then restoring its position after the completion of the ISC_LOCK guardian's READY state. 

We traced this down to the Lockloss Trigger (see initial release aLOGs in LHO aLOG 44838 LHO aLOG 45061, and ) being errantly enabled on all stages of the SR3 -- which normally does NOT receive any ISC control, so there's no need. The move was because the SR3 CAGE SERVO (unique to SR3) uses the M1 DITHER PIT OFFSET (H1:SUS-SR3_M1_DITHER_P_OFFSET) as its actuator, which is upstream of the M1 ASC trigger. That cage servo is typically outputting 32 DAC counts ("should be" 85 urad in slider units ... but apparently it's not that much), so halting that output via the lock-loss trigger moved the suspension, and then it was virtually immediately restored upon clearing the trigger in the DOWN state.

Turns out this had been happening since the development of the trigger circa Fall 2018, we just never noticed.

We've now disabled (set of OFF, or 0.0) all of the following channels:
    H1:SUS-SR3_${M1,M2,M3}_TRIG_${LSC,ASC}_ENABLE
and accepted in the SR3's safe and OBSERVE.snaps, so this should not happen again.

#RedHerring

Bubba is helping us raise the Pcal lab (room #166) temperature by ~9F. Our plan is to lower the temperature again and measure the responsivity ratio of our Working Standards during the transition. As far as we can tell, this temp change should only affect the Pcal Lab, but if you notice the temperature rising in other areas of the site, please let either Bubba or myself know.

TITLE: 08/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 2mph Gusts, 0mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:  Attempting to relock after Corey gave up mid-last night.

TITLE: 08/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Travis
SHIFT SUMMARY:

Frustrating shift with H1 down the entire shift.  Left in the READY state at 12:10utc/5:10amPDT & with Observatory Mode = CORRECTIVE MAINTENANCE.

• Alignment looked OK. 
• ASC (for most part) was not much of an issue as long as you babysat it at certain steps. 
• Main issue was H1 could never get past LOWNOISE_COIL_DRIVERS.
• Ran an alignment.
• Later in the shift, the Green X-arm had trouble staying locked half of the time (eventhough its alignment as been looking good before/after an INITIAL ALIGNMENT).

LOG:

• 7:15 Continuing Locking with this my first Lock#1
  • Hung out at ENGAGE DRMI ASC & OFFLOAD DRMI ASC for a while.
  • Lockloss at while waiting for signals to converge at ENGAGE SOFT LOOPS.  
    • Did not see an ERROR signal ring up & so I was not able to run the Gain Reduction Script (from alog#51282).
    • I did see CONTROL and ERROR signals get bigger on the wall Strip Tool
    • Not really sure how to 
• 7:50 Lock#2
  • Lockloss OFFLOAD_DRMI_ASC
• 8:02 LOCK#3
  • lockloss at LOWNOISE COIL DRIVERS
• 8:42 LOCK #4
  • paused at LOWNOISE ASC, and then
  • lockloss (once again) at LOWNOISE COIL DRIVERS, and this definitely happened during one of the steps (around PR3 
• 9:17 LOCK#4
  • lockloss at LOCKING ALS
• 9:19 LOCK#5
  • lockloss at LOCKING ALS
• 9:23 LOCK #6
  • lockloss while trying to go through the LOWNOISE COIL DRIVERS steps via command line (not sure I did this correctly).
• 10:02 Lock #7
  • Actually, there were multiple locklosses at LOCKING ALS. 
  • Problems for X-arm at LOCKING ARMS GREEN:  Flashes were above 1, but X-ARM would lock, but had a noticeable oscillation which would eventually break arm lock.  Unlocked and tweaked...and this seemed to calm X-arm down and then be able to move on.
  • Lockloss at ENGAGE SOFTLOOPS
• 11:33-11:58  INITIAL ALIGNMENT
  • X-arm has issues remaining locked, which makes alignment frustrating.  It looks fine with transmitted nicely over 1.0, but it would randomly drop lock after a few minutes while waiting for signals to converge.
• 12:00 Locking post Alignment:  Continue to see X-arm drop out of lock randomly even with a nice alignment, and then multiple locklosses around LOCKING_ALS/FIND_IR
• 12:09 Leaving H1 in the READY state for the Day reinforcements.

Observatory Mode = CORRECTIVE MAINTENANCE due to H1 having problems.

• Have been able to get through ASC steps past LOWNOISE ASC, but repeatedly (3x) had a lockloss at LOWNOISE_COIL_DRIVERS.  
• Since the above state is after getting to 37W, I'm guessing alignment should be OK.  
• Tried running through LOWNOISE_COIL_DRIVERS by hand, but not sure if I did it correctly (had a lockloss nonetheless).
• At any rate, will try this one more time (each attempt takes ~45-90min to get there), and if there's another lockloss, I'm inclined to try an alignment (with hesitation), or wait for the day crew.

Have had a 3 attempts dashed by locklosses before getting through various ASC steps.....   Will keep making more attempts and see if I can try to employ the script (alog #51282) to stamp down CONTROL signals when they get ugly (have not tried that yet).  I reckon I will if I keep running into a wall, I'll run an alignment (although the arms & DRMI look fine).

Here are notes on the 1st lock:

7:15 Continuing Locking with this my first Lock#1

• Hung out at ENGAGE DRMI ASC & OFFLOAD DRMI ASC for a while.
• Lockloss at while waiting for signals to converge at ENGAGE SOFT LOOPS.  
  • Did not see an ERROR signal ring up & so I was not able to run the Gain Reduction Script (from alog#51282).
  • I did see CONTROL (attachment#1) and ERROR (attachement #2) signals get bigger on the wall Strip Tools (unfortunately, I didn't get a snap of the ASC ndscope....
  • But on the ASC ndscope I didn't see anything getting big/noisier.

TITLE: 08/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 18mph Gusts, 15mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:  H1 DOWN & trying to lock.

Since we have been down for 90-min I am going to mark this time on Observatory Mode as CORRECTIVE MAINTENANCE.

Ed had been trying to reacquire, but has been dropping out at various spots (green arms look good, PRMI locks well, from what I've seen since he left). 

He mentioned if I try to run an Alignment, this could be an involved process.....so I'm going to read alogs and keep on acquiring with what we've had for the for the last 2hrs (since 5:21utc).

NOTE:  While waiting here at ENGAGE_DRMI_ASC, received IMC_LOCK notification for "IMC WFS not centered".  Odd...and just something I wanted to note.

TITLE: 08/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
LOG:

23:45 Initial Alignment

1:05 Kara and Sheila out to LVEA  - aligning DIFF beatnote on ICST1

1:43 first DRMI lock since adjusting the beatnote

2:36 H1 back to Observing

5:21 Lockloss

• Multiple attemps and some manual PRMI to get levels high enough for ASC to engage non-destructively - Still no success
  • lockloss at PREP_ASC_FOR_FULL_IFO
  • CARM_TO_TR
    • failures occurring at different stages. No pattern to speak of.

7:03 Handing off to Corey

116 Mpc

DARM looks good. The usual, rumbly, 10-20 Hz business going on.

J. Driggers, J. Kissel

I've been noticing that we've had to use this script more often because of our new global alignment position, so I want post an explicit aLOG about it (essentiall a repost of LHO aLOG 45688):

To move an arm, or another collection of optics in our angular control system basis to reduce that ASC degree of freedom's error signal (because you're losing lock when you turn on any of the FULL IFO ASC in "PREP_ASC_FOR_FULL_IFO" or "ENGAGE_SOFT_LOOPS"), use the following script (from a command line terminal):
 
/opt/rtcds/userapps/release/asc/h1/scripts/move_ARM_dev.py ${ARMDOF} ${ANGLEDOF} +/-${STEPSIZE}
So the arguments are 
    ${ARMDOF} = CH, DH, CS, DS, XH, XS, YH, YS
    ${ANGLEDOF} = P, Y (you *can* use "PIT" and "YAW")
    ${STEPSIZE} = slider units of the optics, so urads ish.
with "D" differential, "C" for common, "H" for hard, "S" for soft, and if you want to only move an arm, "X" for X arm, "Y" for y-arm.

An example that we used tonight to reduce the CHARD Pitch error signal:
 
    $ cd /opt/rtcds/userapps/release/asc/h1/scripts/
    $ python move_ARM_dev.py CH P +0.03

You're going to want to do the step sizes in small increments (i.e. run the script many times, pausing for 10 seconds in between each run), and as always it's a guess and check as far as the direction (sign) of the step. The goal is to make the error signal go towards zero. 

Remember to use the "ASC Signals" ndscope template from Jenne's locking screen to quickly pull up all ASC error signals.

J. Driggers, S. Dwyer, J. Kissel, E. Merilh, K. Merfeld 

Given the afternoon's earthquake and apparently "new" trouble with ALS DIFF beatnote being too low (LHO aLOG 51280), we ran out of time to today to complete the plan I laid out in Part 1's LHO aLOG 51275.

Instead, we
    (a) *Did* run an initial alignment, because initially after the earthquake our PRMI alignment looked poor and we started suspecting the DIFF beatnote and thought an initial alignment would help the arms / beam splitter into an alignment where the beatnote was good (it didn't).
    (b) Aligned the beatnote on ISCT1
    (c) reverted the spot positions to the Post-July 31 values, and are now
    (d) just pushing to get back to nominal low noise at 37W.
In other words -- reverted all of our changes (except for the DIFF beat note improvement, so we'll have to remember to look at it again once we do successfully move the spots and save it as an alignment reference), and are just getting back to how we've been the past few weeks.

Operators have no new notes, and should just carry on as we have been with no fear of performing an initial alignment if they deem it necessary, and they should expect the same lock acquisition success rate as we've had over the past few weeks.

We'll formulate a modified plan tomorrow (though the goals will remain the same -- we just ran out of time with a functional, robustly locking IFO today).
    On the list of things to remember to adjust once we get the the global alignment position changed successfully:
    (1) Initial alignment references
    (2) OMC ASC alignment offsets
    (3) On table ISCT1 DIFF beatnote alignment

I measured the output power of the recently-removed SQZ NPRO laser SN 8552.  The laser settings were found as last set by the SQZ team:

• NPRO Crystal Temp: 31.68 °C
• Diode Current: 2.025 A
• Diode Temp: 20.21 °C

I set the laser settings to those specified in the original data sheet; the set value for the NPRO Crystal temp is slightly different from the actual value, and the power supply reports both numbers so I recorded the actual crystal temp as well.

• NPRO Crystal Temp: 25.10 °C
  • Actual Temp: 25.11 °C
• Diode Current: 2.15 A
• Diode Temp: 20.20 °C

I allowed the laser to warm up for ~5 minutes before taking the first power measurement.  For the measurement I used an Ophir VEGA power meter with an Ophir PD300-3W head.  After the intial reading, the power was slowly increasing, so I waited until it stabilized, noting the power measured a few times over the duration:

• Intial measurement, t0: 0.925 W
• t0 + 10min: 1.015 W
• t0 + 25min: 1.054 W
• t0 + 35min: 1.055 W

The data sheet reports an output power of 1.05 W using the initial settings, so this laser seems fine from an output power standpoint.  I took one last reading of the laser settings to see if things had changed since I started; they had, albeight slightly:

• NPRO Crystal Temp, actual: 25.12 °C
• Diode Current: 2.149 A
• Diode Temp: 20.18 °C

Since the NPRO only had a couple hours to acclimatize to the optics lab environment, I'll let it sit overnight and repeat these measurements tomorrow to make sure everything is unchanged.  As of right now though, the power out of the NPRO matches the original data sheet.

We noticed some curious jumps in the CLF power as measured by the CLF REFL PD. These jumps are also visible in CLF launch and rejected monitors. They are not visible in any of the other PDs on ISCT6 that are not associated with the CLF. See attached plot 1. We see coherent jumps in the RF output power readback of the 200MHz AOM driver. See plot 2.

The jumps in the CLF power are as large as 20%, whereas the RF power to the AOM only jumps by 5% or less.

Starting at ~9:31 utc have noticed elevated levels for DARM BLRMS bands:

• 500-1kHz
• 1kHz-2kHz
• And higher as seen on DARM spectrum

The noise went off and on for 45 min (so far) and looks to start/stop suddenly and then last on the order of a minute and over 10mins (attachment#1 is the DARM blrms where you can see the steps up in noise over most of the 45min.  Time periods for some of these are about:

• 9:50-10:02utc  Attached is a DARM spectra (w/ quiet reference)
• 10:05-10:06 this was quieter than above and not quite as broadband, see attachment

NOTE:  H1 range did not take a hit during this time.  It's been almost 2hrs since this noisy period last occurred.