[Keita, Jenne]

Keita and I have been trying to figure out what is going on with the ISS that is causing the diffracted power to oscillate so largely the past few days.  When the IMC is locked, the ISS first loop, with the slow part of the second loop board but not the "second loop" as you would normally think of it, is causing the diffracted power to oscillate more than 1.5%.  This is very large compared to what it ought to be. 

In the attached plot, I have the peak-to-peak values of the diffracted power, from August 1st 2018 through yesterday.  I'm using the max and min of minute trends, so this won't catch if some short period of time has a small pk-pk value, but will see larger trends.  Data is only plotted here if the first loop is closed, the second loop is open, but the second loop is sending signal from the slow loop over to the first loop (yes, the ISS is confusing...).  That is, this should represent times when the IMC is locked, or we're trying to acquire IFO lock, but not times when the second loop is actually feeding back using IMC transmitted power (this is engaged in guardian once DRMI is locked). 

You can see that there was a time in ~September that the diffracted power oscillations were high, but then they got better.  Since the beginning of December, the peak-to-peak value hasn't been near it's normal small value. 

It seems like the second loop board is picking up more noise somehow, somewhere, and is feeding that through the slow portion of the board to the AOM.  When we turn off the output of the second loop board, we immediately see that the diffracted power becomes nice and smooth and quiet (we tried this briefly yesterday).  Still under investigation, but something is definitely not right.

• Alignment really bad, ran initial alignment. 
  • MICH_DARK_Align worked today (tried it on a whim, since it hadn't been working).
  • PRC_Align needs to turn off the input limiter.
• ISS Second Loop offsets need to be adjusted.  Trying to close second loop makes first loop lose lock.  Commented out ISS in DRMI_CHECK_ASC for now.
• ALS was having trouble locking.  Looked like DIFF glitches, of the style that happen when COMM and DIFF's frequencies cross.  I by-hand requested COMM to go to HANDOFF_PART1 and the DIFF glitches seemed to go away.  So maybe we want to at least partly lock COMM when we're working on DIFF, so that we don't get frequency crossings (if that's what is going on)?
  • Can semi-repeatably lock ALS by requesting COMM to HANDOFF_PART1 and then requesting main guardian to LOCKING_ALS.  But, at some point COMM and DIFF both see an oscillation at about 765 Hz (on a different lock it was more like 360 Hz - why??).  This coincides in time with the Xarm green power dropping, which makes things worse, and then we lose lock.  The COMM crossover with MC2 (measured by taking an In1/In2 TF of LSC-MCL) is nice and stable.  The COMM OLG as measured by Craig also looks nice and stable, having a little more than 90 degrees of phase at about 750 Hz.
  • Have also a few locks been okay just requesting LOCKING_ALS.
  • Sheila is reverting to LOCKING_ALS setup back to a situation where we lock both COMM and DIFF at the same time.
• Trying to hold on to the arms in green stronger, Craig made the Yarm green PDH settings the same as the Yarm green PDH settings.  This means that right now Yarm has 3dB less slider gain, but is using both boosts.  The Yarm control signal doesn't seem drastically changed, but maybe the transmitted green power is a bit more steady.
• Sometimes we are able to lock ALS, but the Xarm gets misaligned and we lose lock.  Georgia and Sheila point out that because of the high ground motion, we seem to be pushing on the ETMX L1 stage pretty hard to hold the lock (fine).  But, we have bad L2A coupling, so this large actuation on L1 is moving ETMX in angle.
  • It looks, according to the oplev, like the L2P decoupling for L1 is fine, and ETMX isn't pitching too badly.  But, the optic is yaw-ing a lot.  We don't have any L2Y decoupling engaged for the L1 stage right now, but maybe we need to put that in, in order to handle these high ground motion times.
  • It looks like the data for ETMX L1 was most recently measured in alog 43150. In the comments, Hang has done the fits to JeffK's data for L2P, but not for L2Y.
• Between ETMX L2Y tuning and the wind dying down, we are now again able to lock DRMI. 
• ISS_ON state in IMC_LOCK guardian is killing the lock.  There is definitely something wrong with the ISS second loop.  For now, it's commented out of the guardian.  (alogs forthcoming from me and Keita about the second loop acting weird).
• MICH ASC was also causing locklosses again.  The AS36 phase needed to be tuned to match the new TCS settings.  I added +60 deg of phase to each quadrant of AS36_A, and at least for this lock, MICH ASC hasn't caused any more trouple.
• Lockloss at the DC readout transition.  Sheila says that she saw this at least once yesterday as well.
  • Next lock the transition just worked on its own.
• Based on a plot from the 16.5 hour lock Wednesday night / Thursday morning, I have made the IFO lockloss trigger more strict. 
  • Once ASC is on and converged, normalized POP_A_DC never falls below about 950 counts.  So, I have set in PREP_DC_READOUT_TRANSITION that the overall IFO trigger should shut things down if POP_A_DC falls below 850.  This should leave enough margin in case the power dips for some unknown reason.
  • Note: the long lock was at 20W injected, but now we're at 30W injected, and normalized POP_A_DC is closer to 910 counts, so the threshold of 850 should be okay, and actually be a little tighter.
  • The attached plot is the lockloss from the 16.5 hour lock.  The first T-cursor on the left side of each plot is at the time that we *really* lost lock, as witnessed by a glitch in ASC-AS_C (when you zoom in a lot).  The next T-cursor is at the time that the IFO trigger starts ramping signals down (when LSC-POP_A_LF_OUT_DQ falls below 850*19.78 = 16821).  The top right trace in blue is the guardian state, so when it falls below 600 is when the guardian begins to turn things off.  So, the IFO trigger is doing better than the guardian, but clearly needs to be turned off even sooner.  We have a delay of about 0.22 seconds between true lockloss and when the trigger went, and a whole 1 second between the lockloss and the guardian recognizing it. 
  • The trick, of course, is to not make the trigger threshold so tight that if there is a slight fluctuation we kill the lock ourselves. In the case of the 16.5 hour lock, POP_A_LF_OUT_DQ fluctuates down to a value of 19230 at some point.  If the trigger threshold were set to be just barely below this value (972 in the normalized units, since the PSL was injecting 19.78W at that time), we would have triggered off at 0.14 seconds after the true lockloss.  So, maybe it is difficult to catch a lock much faster than that.

Guardian changes summarized:

• Commented out ISS_ON request in DRMI_CHECK_ASC.
• PRC1 input limiter is disabled in PREP_PRC_ALIGN in the ALOGN_IFO guardian.
• Reverting LOCKING_ALS state to lock both COMM and DIFF at the same time.
• Made lockloss trigger thresholds more strict, to catch locklosses earlier. 

Other changes:

• Yarm green PDH gain down, second boost engaged.

The 16-hour lock from Jan 3rd hovered around 78 Mpc BNS range but had very strong scattering as described in 46222. This is a response to Jenne's request in that alog for detchar help on where the scattering might come from. This scattering is very broad and reaches to frequencies as high as 80Hz in strain, while it is seen at mostly 10-20Hz in auxiliary channels. The most strongly related channels to this scattering are Y-arm related, and the motion of ETMY's upper stage matches the fringes qualitatively. 

Fig 1 shows a normalized spectrogram of (cleaned) strain with about 10 scattering events in 2 minutes. Hveto identified a strong correlation with the channel ASC-Y_TR_B_PIT_OUT_DQ. Glitches at 10-20Hz in that ASC channel are coincident with glitches 40-70Hz in strain. Note that this channel only sees the very loud/high frequency scattering. 

Fig 2 shows the characteristic SNR vs frequency behavior of the scatter, where scatter that reaches higher frequency has higher SNR (as the detector noise drops to higher frequency).

Fig 3 shows a montage of three omega scans of scattering events, a large one (the same as the first one in fig 1), a medium one, and a weak one. The fringes are very broad in h(t), and reach ~80Hz in strain. The fringes are at lower frequency, narrower and stronger (higher SNR) in ASC-Y_TR. There are also fringes in ETMY Oplev Pitch channels with similar period at even lower frequency, but as you can see, these don't really line up. 

Figs 4,5 show spectrograms of ASC-Y_TR_B_PIT_OUT_DQ with fringe frequencies calculated from the long motion of the upper mass* of ETMY and SRM suspensions. Out of all H1 suspensions, ETMY M0/R0 and SRM line up best. Fig 5 is an alternative look at a different time from Guillermo. 

Finally, if I understood Jenne's alog correctly, they were intentionally moving the compensation plate from roughly 17:30-19:30 UTC, so I ignored those times when doing this work. 

* As Rana pointed out, there are no laser beams on the upper mass. We're using these for convenience because the long signals there are calibrated to um, but we'd love to switch to an estimated lower mass displacement w/r/t something (to calculate fringe frequency) and are very open for suggestions. 

TITLE: 01/04 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY:  Struggled most of the day to get to/past DRMI due to high microseism and wind and ALS issues.  Just getting close to NLN at the end of the shift.
LOG:

16:28 Chris clearing tumbleweeds along X arm

16:35 Ed to EY to tweak OpLev laser

17:10 Ed back

18:10 Kyle to LVEA retreiving pump carts

18:25 Kyle out

19:23 Kyle to both mid stations

20:40 Kyle back

20:44 Fil to CER mezzanine

20:55 Fil out

This is a quick companion to my post for LLO 42583. Please look through or see the associated https://dcc.ligo.org/G1900011 that for context on these plots

I ran the same pipeline for H1 at 2019-01-03T08:00:00 UTC averaged for 700 seconds. I also looked at O2 data, but the machine did not appear to be sufficiently shotnoise limited around the UGF to compare with LLOs O2 in the presentation. Now it is.

darm_loop20190103T080000.png : the DARM loop I get from the RCG calibration system. I do not know yet why it is a poor estimator at low frequencies given A: the coherence, B: that the same thing works fine at LLO. May be from more cal lines in LHO, but those shouldn't have a broadband effect. In any case it works OK above 70Hz. I'll dig in to figure out why this is.

strain20190103T080000.png : The strain averaged over this time, from both cal systems.

DCPDCL20190103T080000.png : DCPDs in closed loop

DCPDOL20190103T080000.png : DCPDs in open loop, zoomed way out

spec20190103T080000.png : The PSD of the open loop DCPDs, calibrated to shot noise. as well as the noise-subtracted version.

I might say that the same loop-injected excess I'm curious about at LLO is also occurring at LHO. Curiously, the same .95 correction factor to the ASD as I used for the absolute calibration at LLO is also necessary for LHO, suggesting that the power/transimpedance calibration is 10% off.

loop_adjusted.png : If I try to virtually retune the loop correction, I do not get a flattening of the noise curve the way I can in the LLO O2 data shown in the presentation.

Once we see a sufficient segment of squeezing (or antisqueezing), I will analyze further.

Keita, Sheila, Georgia, Jenne, Craig

We noticed that with the ground motion a little high, our X arm green transmissions sometimes dip low enough to cause ALS to have trouble locking, because the drive to the UIM was causing the test mass to yaw.  The first attachment shows a time series, the dips in the transmitted green power seem to correlate well with excursions of the optical lever in yaw, which are very well correlated with the length drive to the UIM.  

We looked back a little at the history of this stage's coil balancing and decoupling since the vent: Two of the magnets were found to be flipped in June 42723 The coils were balanced in July 42740, measurements were made for the  frequency dependent decoupling at the end of July 43150 and at that time Hang did the fitting for the pitch measurements and installed a L2P filter, but the yaw measurements weren't fit.  

Based on the first attachment, the L2P decoupling is working well.  We noticed while doing driven measurements that the pitch mode of the suspension rings with any transient for a long time, but things seem to be OK when we are just looking at the ambient ground motion. 

Since we are concerned with motions that are at 40 mHz or so today, we tuned the DC gain of the L2Y filter without engaging any frequency dependent filter.  We reduced the coupling seen by the optical lever by a factor of 4 or more by setting the DC gain to -0.00075

SQZ locking is now automated from locking OPO to SQZ angle. The automation will also bring things back up to nominal states when they're kicked out of place. SQZ guardians now have a manager called SQZ_LOCK (Haocun wrote this a while back, now it's ready). SQZ_LOCK manages OPO guardian, PLL (beat note locking), CLF, and LO. SHG guardian is still a stand-alone guardian (I see no advantage of it being managed at the moment).

When the nominal state is requested, SQZ_LOCK will lock OPO on CLF double resonance. Once the node is arrived SQZ_LOCK takes PLL to CHECK_BEATNOTE_2 which put SQZ pump laser frequency close to the IFO carrier and lock CLF. Although SQZ_LOCK manages LO guardian, LO lock still has to be requested by hand at the moment (note that LO_LOCKED_OMC doesn't work yet, so don't go there, the current nominal state is LO_LOCKED_HD).

LO looks at average RF3 beatnote value to determine if it's okay to close the loop. Once it's locked it uses stddev to determined whether or not it's actually locked. LO guardian also looks for rails. If any output rails it will disengage the loop and turn off boosts. In addition it also looks at OPO, CLF, and IMC status.

PLL looks at both OPO and IMC guardian states. If either is not nominal it will go to a state where it waits for both of them to be nominal again. IMC when loses lock usually kicks the reference cavity which is where our PSL LO sample comes from. When PLL is disengage LO will also rail and disengage. This has been tested with several IMC lock losses. All of them successfully came back to nominal state on their own.

Intensity stabilization servo is now a function lives in SQZ_OPO guardian.

The repeated cause of TTFSS EOM railing is LO output rails. So OPO CHECK_EOM state will now close EOM loop if LO outputs are okay and automatically return to nominal state. That will also allow PLL and LO to go back to their nominal states.

Will need some time with the IFO to get the LO OMC guardian to work. And will continue to fine tune the guardian.

We had difficulty locking most of the day today, but we were able to get to low noise with the 30W input power, which seems stable.

Difficulty locking

This morning we attempted to inject squeezing into the interferometer, which caused a lockloss.  We've had a tough time re-locking all day since then, it seems that the arms are moving a lot at 50mHz (up to 5um pp).  While it has been windy today it doesn't seem like the ground motion and tilt are large enough to explain the difficulty we've had.  We have had the seismic in the "WINDY", we have had max wind speeds around 30mph for most of the day but periods when it was gusting up to 40 mph. There have also been a few rather small earthquakes.  

ALS changes reverted

We spent some time having difficulty engaging ALS DIFF, we would loose lock reliably when it first ramped on the gain.  We have reverted a series of changes that were made recently, and things seem better now.  Yesterday we added a 120Hz notch to the 60Hz notch we use with ALS because we had bad ground loop problems at the PSL, today I removed this (46213).  A couple of months ago people made ALS DIFF and COMM lock simultaneously to save time, this doesn't save us much time and make it confusing to debug things, so we have gone back to locking DIFF then locking COMM.  Lastly, we went back to ramping the DIFF gain on first to 1/10th of its full bandwidth slowly, then ramping on to the full gain (this was removed over the break 46146).  We also reduced teh gain back to 400 (rather than 600, the 3dB increase Rana mentioned in his alog). 

Squeezer automation

We took some time to work on the automation of the squeezer while we were having trouble locking.  Nutsinee spent some time working on the squeezer guardians, and has a working guardian to close the LO loop which she tested on the homodyne.  We also spent some time testing some of the code that Haocun wrote for doing initial alignment of the squeezer, which is in the ALIGN_IFO guardian.  We don't know if we will really need to do this each time we do initial alignment, but since we have been doing it each time we inject squeezing it will be useful to have it automated.  We also added another branch to the ALIGN_IFO guardian that lets us engage the AS centering loops with a single bounce PSL beam, and offload the OM sliders, so that we know we are aligning the squeezer to the right alignment when we inject it.  

ISS problems

Keita and Jenne spent some time looking at problems with the ISS, I don't know all the details but when the second loop is either off or AC coupled the first loop diffracted power is fluctuating a lot.  Keita and I tried DC coupling the ISS with the interferometer at 30W input power, and DC coupling it caused the power to drop by about a Watt.  The first attachment shows that the problem is when the AC coupling output gets held it is not at the mean of the perviously noisy value, which we think is why the diffracted power changed.  We also saw that the diffracted power servo wasn't really doing anything, Keita increased the gain from 0.05 to 5.  This was tuned for 2W, so it makes sense that we need to adjust it for 30W input power.  This gain is now adjusted in the IMC_LOCK guardian before it turns on the ISS at 2W and before it gets DC coupled.  The ISS second loop is left commented out of the guardian, because DC coupling it could still cause a transient in the input power while the first loop is so noisy. 

30Watts input (~90kW circulating power) low noise 

Following the instructions given in 46190 we were able to get to 30W without an issues, and then go to low noise.  I moved ADJUST_POWER to after CHARD blends and it will not set the power to 25 W in one step, wait 5 minutes then set the power to 30W and wait 10 minutes.  I didn't try to shorten this time, and I haven't tested the code this way, but think it should work. I also lowered the amplitudes of the HARD yaw dithers another factor of 10 since those lines are very large in DARM once we are in low noise. 

We were locked in low noise (the only thing I skipped was reducing the 9 MHz modulation depth) from about 5:27 UTC to about 5:48 UTC.  We had a lot of non stationary noise at low frequencies, similar to last night.  I tried to drive ITMX CP to see if I could find a better alignment for it, but that unlocked the IFO.  DARM has coherence with MICH below 100 Hz (attached plot), we haven't re-tuned the feed-forwards yet for this these settings.  

TITLE: 01/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY:
LOG:

17:29 Kyle to MX

17:59 Kyle to MY then back to MX

18:31 Kyle back

21:20 Ed to EY tweaking OpLev laser

21:49 Ed back

21:49 Betsy inspected TCS tables in LVEA for the suspected water leak.  No water was found on or around the tables.

The current-monitor voltage used as a metric to adjust power was adjusted from 924mv to 914mv in a second attempt to quiet the glitching. OpLev damping is not being thus far on the ITMs and ETMs, however it is being considered for use in the not-so-distant future.

in other news...there is promise for the newest addition to the laser family being installed in the ITMX position by next Tuesday as it is passing stabilization testing in the LSB optics lab. Currently, the ITMX OpLev is a glitchy mess without a temperature stabilization enclosure (modified ice chest), the new standard of supply power, or armored fiber.

At 13:11 PST h1nds1 died. The console output was showing a kernel problem in the network stack. The only recourse was to reboot the computer using the front panel reset button.

This is a 2.6.35 kernel machine, but had only been running for 192 days so this does not look like a 208.5 day bug crash.

h1nds1 is the default nds server for the control room, so some nds clients may need to reconnect to the NDS server.

While we were locked (16.5 hours!), we noticed that especially with the large wind (and / or anthropogenic ground motion?) we were seeing big scatter shelves in the DARM spectrum. 

As a first pretty passive check, we moved the ITM compensation plates to see if a different position would give different scattering structure.  The test positions (slider values shown in time series attachment) are perhaps slightly worse than the nominal, but it's pretty hard to say. 

In the attached 5000 second spectrogram, I have drawn colored lines along the top of the x-axis to indicate the state of the CPs at various times.  The plot includes 1000 seconds at the CPs' nominal positions (yellow line), a test position (turquoise line), again the original positions (yellow) and again the test positions (turquoise).  During the times of the black lines the CPs are being moved. You can see that the first time we were in the test positions things were clearly worse, but that could perhaps have corresponded to some higher wind speeds (we've been getting sustained winds of more than 20 mph this morning).  When we repeated the test back to the test positions, the scattering is maybe a teeny bit worse than the nominal positions, but it's really very marginal. 

We moved on to some squeezing tests after this, so we haven't tried any more CP positions.  For now, we're leaving them at their nominal positions. 

If DetChar has time, it would be helpful to run the scattering identification tools on some of the data in the last ~5 hours, to help us identify what might be the cause of this scattering.

• BRSx is approaching it's negative Red Line (but probably has a few weeks).
• BRSy is looking good and is pretty flat.

Looks good.  Pitch & Yaw are within acceptable range (i.e. no centering needed).

Note:  Script would not run from the OPS/WEEKLIES medm.  Get message:  "bash: /opt/rtcds/userapps/trunk/sys/h1/scripts/oplev_trends.py: Permission denied".  So ran script manually.

The mount for the PZT in the IO path on the PSL, IO_MB_M4, was swapped from the 2" mirror and the damped mount to the new PZT mount.

Aligning with the new PZT mount is challenged by having to remove the entire mount from the table, to access the bolts underneath, to adjust the +/-X position.  There was also a change of the beam alignment when securing the bottm plate to the table.  The beam was carefully aligned to IO GigE camera 2 and camera 3, and when the bottom plate of the new mount was secured to the table, the beam was gone from both cameras, and an offset was introduced into the IMC_IN beam.  With the PZT it was possible to restore a beam to IO GigE 2, the camera that is on the bottom periscope transmitted beam.  It was not possible to restore both camera images with the PZT.  This indicates that the change in IMC_IN position and angle are due to the PZT swap.  It was past noon, so the decision was to leave this change as is, lock the IMC, and evaluate how it and the beam downstream was affected, instead of using M3 and the PZT on the PSL, to restore an image to both cameras.  The change in IMC_IN on the iris at the bottom periscope was identifiable as yaw with an IR viewer.

The reference beam from the shutter (between the PSL and HAM1) was recorded before and after the PZT swap.  Those pictures show the difference is between the before and after beams is 1mm and 2mm in pitch and yaw (image attached).

Sheila and JeffK moved IMC DOF, Corey and JeffK started an initial alignment, and now the effort is to lock.

BEFORE reference time 12/11 18:00 UTC, AFTER reference time 12/12 1:24 UTC.  A snapshot showing IMC WFS, IM4 Trans, and ISS Second Loop QPD signals is attached.

- Cheryl, Jason, Keita