J. Driggers, J. Kissel, S. Dwyer

We were unsuccessful at measuring the two things we needed in yesterday's attempt (LHO aLOG 49541) at checking whether the ETMX IFO beam's spot position (i.e. determined by ETMX PUM A2L gains) impacts the sensing function measurement [where the hypothesis is that the confusing, apparently a-causal detuned spring response -- see e.g. LHO aLOG 48083 -- is an artifact of the DARM EXC through all stages of ETMX (including the PUM, which we suspect has L2A2L problems because of the spot position) during the open loop gain transfer function half of the two-part sensing function measurement suite].

Here "unsuccessful" means that we were able to get one half of the measurement -- the swept-sine DARM Open Loop Gain TF, but apparently lost like right at the start of the swept PCAL2DARM transfer function.
However, upon further investigation (details below), we believe the lockloss was NOT caused by starting the PCAL excitation, but by a rather fast run-away (within 5 seconds) of the SRC1 Pitch loop which in turn cross-coupled to the ARM ASC DOFs, and saturated ETMX. This is after ~1000 seconds of being happy in the moved-spot configuration. 

As such, we suggest the following when we try again:
   (1) Pay closer attention to the SRC1 P loop:
        (a) either rephase the AS72 WFS after spots have moved, or
        (b) open the SRC1 P loop during the measurement, and
   (2) Be faster with the measurement.

(2) will be a challenge, given that -- assuming all things the same, we'll only have ~1000 secs, and we *need* a swept sign measurement downs to ~5 Hz in order to really resolve changes in the funky detuned spring. Our suggestion is to lop off data above ~100 Hz, and maybe relax the integration time by a factor of 2.

But at least we now have a path forward to try again.

Investigation Details:
(1) I ran "lockloss select" in a terminal, and selected the appropriate lockloss time from the list. This opened the lockloss tool webpage for GPS time 1243206548.
(2) Browsing through the signals, at the 30 sec and 5 sec prior to lock loss, we immediate saw three suspicious things:
    (a) The ETMX L2 UL and UR (i.e. the PUM stage of ETMX) was the first to saturate, and did so in a rather sudden, but still slow (~5 sec time scale) and in a "runaway" fashion, not in a "oscillation" fashion, nor in a "sudden huge burst." Because the PUM is involved, which is the only stage the ASC system uses for control, this puts ASC signals rather high on the suspect list.
    (b) Scrolling down to the ASC section, we find the CHARD P, (and the remaining arm DOFS in P) show a 0.15 Hz oscillation, which apparently ramps up within the last 5 seconds, and
    (c) SRC1 P shows an interesting "runaway" feature that looks similar to the ETMX saturation.

(3) Upon pulling up these channels in NDSSCOPE -- such that we can investigate the time scale, whether these 30 seconds was similar to the previous many minutes that we had been stably functional in the spot, and to confirm that it wasn't the PCAL measurement's fault -- we confirm that
    (a) The PCAL excitation had started *extremely coincidentally* with the lock loss, but was not the cause. The DARM OLGTF excitation had stopped at 15 seconds before the lock loss, and the PCAL measurement start 1 second after. Neither correlated with the sudden change in character of the ASC signals 5 seconds prior.
    (b) The spot position, driven by the ADS system and triggered by the change in ETMX PUM P2L gain change, had settled (the ADS signal had converged), by ~1000 seconds before the lock loss, so the ~5 sec time-scale of the last-minute runaway or oscillations is not a function of the new spot position.
    (c) The apparent runaway "oscillation" of arm ASC DOFs, and specifically CHARD is a selection bias -- these kind of 0.15 Hz periodic wiggle in the last 5 seconds of the lock are at roughly the same amplitude as similar wiggles within the past ~1000 seconds when we arrive at the new spot position.

From this, we conclude that the SRC1 P runaway was the cause of the lock loss. Thus, we should NOT need to bother to adjust any ASC loop gains, because the lockloss was NOT caused by loop oscillating. We also don't need to worry about running the same templates for the CAL measurements. Thus we suggest the above ideas for trying again.

Accepting the following sdf differences to SUSPRM. Sheila noted that the values we are accepting match the values that were being used during the previous lock, so it seems that the SUSPRM setpoints might be changing unexpectedly between losing lock and reaching NLN.

Jenne, Georgia, Sheila

Lockloss while trying to AC couple ISS:1243269492

~ 2 hours to relock because of other relocking issues:

• rainy day glitch, 49556.
• added TMS to green WFS 49557 (didn't take much time at all but should help with some of the arm alignment issues)
• one lockloss where we moved through the convergence checker for the dither loops too quickly, and many ASC signals weren't converged before we increased gains 1243273097  To address this we have made a tighter threshold on the ADS error signals before we increase the gains.

1 attempt at increasing input power:  We requested 38W but the rotation stage calibration was off so we got 40W out of the PSL.  Things seemed OK, Georgia tried to increase CO2X.  We checked the REFL wfs phases with a frequency noise injection, they seemed OK.  While we were attempting to measure an ASC sensing matrix for the REFL WFS, we noticed that the ADS YAW3 (ITMY to PRM dither loop) was running away, and something started to become unstable.  We powered down to 35W out of the PSL, where we checked the phasing of ADS YAW3 which looks fine at 35W.  We turned the loop back on and it seems OK here, so this is something that we need to investigate more.  

We went back to low noise at about noon.  Georgia is now investigating why the ISS AC coupling was a problem. 

Laurence Datrier, Sheila Dwyer

We looked at O2 data for H1 to get an estimate of down time due to wind. We used the maximum wind speed of the three max. 1min and 30min trends for the H1 EX, EY, CS wind channels, for all of O2.

The first histograms and trend line show duty cycle vs wind speed percentiles, for wind speed vs lock status and previous wind speed (max. of previous 1 or 30 min) vs lock status. Looking at the previous wind speed shows more down time due to high winds. Percentiles and wind speed values are (in mph):

The following histograms are the normalised distributions for wind speed at:

•  Instant of lock loss (1min and 30min trend)
•  Instant before lock loss (1min and 30min trend)
•  Time leading up to lock loss (1min trend: 5mins up to lock loss, 30min trend:2.5hrs up to lock loss)

Compared to the normalised distribution at all times (blue). A significant tail appears in the distribution for wind speed at time of lock loss for winds >~35mph.

The Xarm green WFS were struggling a bit during re-acquisition this morning, so Sheila tried enabling the TMS WFS as well as the usual ETM WFS in the LOCKING_ARMS_GREEN state.  This worked well, and she has previously also had it work for the Yarm, and has made a guardian state to do this in the ALS_ARMs. 

I have now changed the ALS_ARM guardians to use the 2 WFS version (ETM and TMS) in the main path, and the ETM-only version is now on the side branch.  This also required a change in ISC_LOCK guardian to request the 2 WFS state. 

So far, we're still on the first acquisition with both green arms using both ETM and TMS WFS automatically through the guardian, but it seems to be working so far.

While trying to relock after loosing lock due to AC coupling the ISS, we had a re-occurance of the "rainy day" glitches which were addressed by burying the ALS Y arm fiber.  1243271468

Just noting this because we hadn't noticed them recently.  It is not rainy right now.

Ops Shift Transition: 05/30/2019, Day Shift 15:00 – 23:00 (08:00-16:00) - UTC (PT)

State of H1: Locked

Intent Bit: Observing

Weather: Low wind, overcast

Primary 0.03 – 0.1Hz: 0.01 um/s

Secondary 0.1 – 0.3Hz: 0.1 um/s

Outgoing Operator: Jeff

Quick Summary: Locked for ~3 hours, Observing for 2.5. SQZ MANAGER has a notification that I haven’t seen before, but it doesn’t seem to be affecting range.

   Accepted the following SDF differences upon relocking after the El Salvador earthquake. 

    Working on relocking after El Salvador earthquake.  

TITLE: 05/29 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 110Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Lock loss during calibration work. Back to observing after initial alignment and help from Sheila and Georgia. Have remained locked and in observing since. Dust alarms for PSL and optics lab.
LOG:

23:09 UTC Lock loss
00:04 UTC Lock loss from ENGAGE_SOFT_LOOPS
00:14 UTC Starting initial alignment
FSS lost lock transitioning to PRC_ALIGN
01:04 UTC Initial alignment done
Sheila and Georgia helping with relocking, changing guardian code
02:12 UTC Observing
04:05 UTC Restarted video4

Lock loss during calibration work. Back to observing after initial alignment and help from Sheila and Georgia.

Screenshotting some SDF diffs associated with the ASC ADS. These were caused by the MICH_BRIGHT_ALIGN part of TJ's fast initial alignment, and I think they won't be touched by anything else.

Jenne, Sheila, Georgia, Patrick

We found a couple of problems with the way the DRMI guardian graph was.  

Several years ago we created a branch in the DRMI guardian that would allow the DRMI ASC to be offloaded when loccked on 1F, this was to make things easier at a time when 3F locking wasn't working.  This wasn't part of the usual procedure, but just an optional branch path in the guardian.  At some point the graph for this guardian was changed so that we were offloading the ASC before transitioning to 3F every single lock, which does sometimes cause locklosses.  

Additionally, earlier today Jenne found that there was a counter error in the ISC_LOCK OFFLOAD_DRMI_ASC state that meant that we weren't actually offloading the DRMI ASC there, where we intended to be.  This is why Georgia's change to the graph last night caused problems.  

Now we have edited the guardian so that the things that we need to do after running initial alignment are done in the prep state (not down, so there shouldn't be a need to rerun down), so that the DRMI ASC is only offloaded after we have transitioned to 3F.  

In the last lock, while the calibration measurements were happening and while the IFO was thermalizing before the calibration measurements, we took the squeezer offline to check some things that would ideally be checked after the laser current is changed.

The SHG temperature didn't need to be changed, but its conversion efficiency has dropped, so I reset the minimum conversion efficency threshold. 

I adjusted the half wave plate to bring the green power into the fiber closer to 20mW, according to the launch diode.  I would like to double check the calibration of the two SHG power monitor diodes.  I added a parameter file for the squeezer guardians, so hat we would not have the TEC temperature hard coded in multiple places and can update it more easily.  (I also changed the temperature to 33.32 degrees).

I then tried to measure the nonlinear gain at a couple of different green powers, to make this easier I also added parameters that would change with green power to the parameter file.  I will plot and post the data soon. 

We still need to double check the squeezing angle next time we are locked. 

J. Driggers, J. Kissel

We've gathered some new data on the DARM loop both in support of regular check-ins on the calibration as well as in prep for a potential increase in IFO power tomorrow, and further investigations in to "L to A to L" coupling in the ISC control scheme:

Standard sensing function suite:
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
        2019-05-29_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml
        2019-05-29_H1_OMCDCPDSUM_to_DARMIN1.xml
        2019-05-29_H1_PCAL2DARMTF_BB.xml
        2019-05-29_H1_PCAL2DARMTF_LF_SS_5t1100Hz_10min.xml

Some incomplete data during the spot position move:
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
        2019-05-29_H1_PCAL2DARMTF_BB_postspotmove.xml
        2019-05-29_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min_postspotmove.xml

Data analysis and results to come later.

Sheila, Georgia, Jenne

Today we took some commissioning time to try to increase the input power.  The increase in input power went fairly well, but we had many locklosses due to problems unrelated to power up, more likely related to Tuesday maintence.  In total we spent about 3-4 hours today commisioning, although the observatory mode was set to commisioning the entire time. 

Increasing power:

• We were able to increase the power to 40W requested input in the adjust power state without many problems, although we haven't had a stable low noise lock yet with the increased power.  
• We made some effort to make sure that it would be easy to revert the changes we made if necessary, and make it easy to come back to this effort another day just by changing the requested power for low noise in lscparams.
• As we increase the input power from 35W to 40W, the POP18 build up drops from ~50 cnts to eventually settle just below 40 counts.  To keep the PRCL ugf at around 50Hz, we need to increase the PRCL digital gain from 16 to 30.  We added a decorator to ISC_library that watches the value of POP18 and adjusts this gain in two steps as we thermalize, and we added this to several of the states that immediately follow the power increase.  This seems very similar to the expirence of increasing the power from 25W to 30-35W in December.  After some changes to TCS and spot positions we were able to prevent the PRCL gain and the POP18 from dropping as dramatically then.  For now it seems OK to digitially compensate for the change in optical gain until we have a chance to improve the build ups. 
• The CARM ugf stays pretty close to 7kHz in these steps (It doesn't get increased to 14kHz until the LASER_NOISE_SUPPRESSION state). 
• Georgia changed the POWER_30W state to MAXIMUM_POWER, and it is written so that all we need to do to change the requested power from 35W to 40W is to change the NLN_POWER in lscparams.
• We made some slight adjustments to the MICH and SRCL gains in LOWNOISE_LENGTH control based on ugf measurements: MICH gain is increased by 10%, SRCL gain 55%.   These are set based on the power request in lscparams.
• We had no difficulties with the noise reduction states after powering up, until we tried to reduce the 9MHz modulation depth.  This seems to have created an ASC instability, which caused enough noise that we started to saturate the ASC REFL45 sensors.  We reduced their whitening gain by 6dB and added 6dB digital filters, but we have not been able to relock since then because of other prolems.  Because we have been having unrelated ASC problems, we reverted the change to be sure that nothing we did today could be the cause of the locking issues. 
• Our rough plan for the next time that we are able to try going to 40W would be to sit and let things thermalize without reducing the 9MHz modulation depth. There we could check the phasing of the REFL WFS, measure the ASC sensing matrix, and perhaps try tuning CO2s to improve the POP18 build ups.

Other problems:

most of our locklosses today were unrelated to powering up, we lost lock twice for reasons that were related to our commissioning efforts, the rest of our many locklosses seem to be related to other problems.  We have had MC1+3 T2 osem glitches 3 times today, 49257 the first two times caused locklosses, and the third time we were able to stay locked.  

We also have had many similar locklosses in the states where we engage ASC and transition to DC readout before powering up.  We don't understand this, and the problems seem to have started yesterday.  

• We saw that the M3 picomotor which was moved yesterday (49238) wasn't set quite back to the original location according to the counter (the counter isn't a reliable way of knowing where the pico is, but it is the best thing we have).  We tried to undo this move by running the green WFS and camera, then opening the loops and moving the pico back to its original counter position (moving it by about 130 counts down).  After that we relocked the arm in green, and ran a complete initial alignment.  According to the sliders and top mass drift monitors this brought our alignment closer to what it was before maintenance day yesterday, but not all the way back. 
• We noticed that both SRC2P and INP1P were not bringing their error signals to 0 when we are engaging the ASC, so we have increased their gains, SRC2 will be reset latter in the guardian but is left with a factor of 3 higher gain for the ASC engagement and power ups.  For INP1 we have increase the gain by a factor of 10, and this will stay increased in full lock.  
• In our most recent locking attempt we phased AS72 after the ASC came on but before the soft loops were engaged.  We opened the SRC1 loops and adjusted the SRM alignment by hand, then adjusted the phases of AS72A to zero the Q signal in each quadrant.  This resulted in better POP18 and POP90 build ups, and we were able to get through the following steps.  We will see if this has fixed the problem or if we just got lucky this time.