Loss of lock probably due to my work at EY on ESD drive.  

Following up on last nights report I went down to EY to investigate problems with the ESD.
Much to my surprise aside from the unit being in the off state I could not find a problem.  The HV supplies were on and at nominal current draw for no drive (3.2mA).  Went in to check the ESD Chassis and all the lights were green. The only indication of a problem was the main indicating light for the unit showed it to be in the off state.  I had disabled all of the DAC inputs to the driver so I turned it on.  Went over to computer and enabled outputs and immediately saw a response on the OpLev.  I am not sure what interlock could have shut the unit down but further investigation is warranted.  I have left the unit on for now.  please keep me Richard McCarthy informed if this or more problems arise.

Evan, Lisa

So, we leave the interferometer with:

• DARM on ETMX (high noise, ETMY ESD not working, please check it tomorrow), DARM boost on (engaed by hand, not in the Guardian)
• PRM, PR2, PR3, SR2, SR3, ITMX, ITMY in their final low noise state for all stages (robust, in the Guardian)
• ETMX and ETMY L2 in low noise (low noise transition not included the Guardian, mode switched back to acq in the downstate)
• SRCL gain reduced from -25 to -17 to give a UGF of 20 Hz, plus ELP100 cut off added (not included in the Guardian, so this would go back to nominal tomorrow, we just want to test if it is stable)

After all of this we saw that the range was around 20 Mpc, even without the low noise ETMY ESD, and without any MICH/SRCL feedforward, so we might indeed have made some improvement.

As far as we can tell right now, the locking sequence tomorrow should work as usual.

Evan, Dan, Lisa

Dan and I wanted to leave, but Evan forced us to stay...so, here we are.

Dan and Evan, the experts in roll and bounce damping science present in the Hanford control room at mid-night, eventually succeeded in damping.

So we relocked to DC readout, right before the transition to low noise ETMY, and switched the coil drivers for PRM-PR2-SR2 (tested at least twice, now in the Guardian) and BS in their final low noise state (for BS we used the script which switches one coil at the time, so this transition is not included in the Guardian yet). We also switched ETMX, ETMY, ITMX, ITMY L2 to their final low noise state (state 3), but we are not including this step in the Guardian yet as we tested it only once (we modified the down state to go back to the acquisition mode for the quads).

We also investigated why the transition to ETMY stopped working by measuring the DARM open loop while actuating on both ETMX and ETMY at the same time.

As far as we can tell, the transition is not working simply because the ETMY ESD is not doing anything at all.

Please go to EY tomorrow morning and check the ESD driver. 

Evan, Kiwamu, Sheila, Dan, Lisa

Summary

We still haven't quite finished our list of "low hanging fruits", but we are getting close. The OMC is not shaking anymore after engaging the DARM boost and reducing the RIN in transmission of the OMC by nearly a factor of 10. Also, we switched the vertex control from POP to POPAIR, which has ~5 times more light, and in doing so we reduced the noise coupling from MICH to DARM. We could reach ~ 28 Mpc again without engaging the MICH feed-forward.

Sadly, we don't have any improvement in Mpc units because a few "technical problems" slowed us down, and we didn't have a chance to finish the coil driver transitions to low noise, improve the vertex length DOF loop cut-offs, and tune the MICH/SRCL feed-forward subtraction while on POPAIR.

In particular, a typo in the ISC_LOCK Guardian (switching of a filter in the "DAMP_M0" filter bank, instead of "DARM_DAMP_MO") badly tripped ETMY (thanks to Jeff who helped us recovering by phone), and caused both roll and bounce mode to run up insanely, and we can't lock. Our damping techniques with ifo unlocked are very (very) slow, so we are giving up.

Also, the transition to ETMY with low noise ESD worked well once today, but later attempts failed (not totally clear why, the only known difference in later attempts is that we engaged the DARM boost before making the transition).

The full list of things we did/tried is below.

Fixed the "shaking" OMC (in the Guardian)

We reduced the DARM motion between 1 and 6 Hz, as Dan reported here, and now the OMC is happy (and we are happy with him).

Transitioned to POPAIR (in the Guardian)

POPAIR has ~5 times more light than POP, so we decided to transition from the in vac diode to the in air one, to reduce the noise coupling of the vertex length DOF to DARM.  Indeed, we could reach the same sensitivity as the other day (~28 Mpc) without any MICH feed-forward.

Close ALS shutters on ISCTEX/EY (in the Guardian)

Not clear improvement, but this is the beginning of the "closing beam diverter/shutter" campaign.

Test of SRCL cut-off  (not in the Guardian yet)

The SRCL loop had 60 degrees of phase at 30 Hz, and a very modest cut off filter. We tested a more aggressive cut-off, which removed all the coherence we see between DARM and SRCL above 100 Hz. We tested this filter only once, so we are not including it yet in the locking sequence.

Test of BS coil driver switch in DRMI  (not in the Guardian yet)

Kiwamu and Evan imported and adapted the script that Den wrote to do the BS coil driver switching one by one. It was tested in DRMI - 1f, and it worked. We haven't tested it in full lock yet.

Dan, Evan, Kiwamu, Lisa, Sheila (and earlier advice from Keita)

Today we turned on the DARM boost filter described here, to increase the suppression in the 1-6Hz band.  The boost itself was no problem, but we realized that changing the phase of the open-loop gain around 10Hz changes the sign of the ETMY and ITMX bounce modes in the DARM error signal, which means the sign of the damping loops need to change as well.  This led to a few hours of bounce mode excitement, but the new signs and phases are now in the Guardian.

The first plot is a comparison of the OMC TRANS intensity noise before and after the boost filter was engaged.  The RMS before the boost was 2e-2 /rt[Hz], now it is 3e-3 /rt[Hz].  The OMC TRANS camera is much calmer.

The boost filter also reduced the coherence between the OMC length error signal and DARM.  Yesterday, Keita pointed out that the OMC length loop was far, far too coherent with DARM between 1-6 Hz, and this noise was being upconverted around the dither line at 3.3kHz.  The upconversion mechanism is not clear; if the OMC length is properly servoed the dither should not generate any RIN (or, very little) and should not upconvert any low-frequency RIN noise to sidebands around the dither frequency.  But, that's what's happening.  The new boost filter reduces the coherence at low frequency (second plot) and reduces the sidebands at 3.3kHz (third plot), but the coherence at low frequency and at 3.3kHz is not zero (second, fourth plots).

As a precaution against OMC LSC --> DARM coupling, we have reduced the gain of the OMC length loop even further: the OMC LSC boost is now off, and there is a 30Hz rolloff filter (FM3) in the OMC length loop.  The UGF is a little more than 10Hz, as described by the green trace from the first figure in this entry.

Keita also recommended changing the dither frequency and changing the dither amplitude and OMC LSC loop gain.  We haven't had a chance to do these things tonight.  We need to investigate why this noise is being upconverted - note that the amplitude of the 3.3kHz dither line in the OMC RIN increased after the OMC LSC boost was turned off (third plot).

The good ETMY and ITMX bounce mode damping settings with the DARM boost ON are (in M0_DARM_DAMP_V):

The good settings with the boost OFF are:

This switching is handled by the Guardian.

By the way, an ongoing mystery is the source of a 300Hz line in DARM that is very large at the beginning of a lock but steadily decays to the level of the noise floor over a timescale of ~10-20min.  The rate of decay is somewhat matched to the BS butterfly mode and the earlier twin peaks noise, but the line is present when we're locked on RF readout, so it doesn't seem to be coming from the OMC.  We've checked the IOP inputs that are sampled at 65k and we don't see a line at 16384+/-300Hz that could be aliased, but we should check again at the very start of a lock when the 300Hz line is very prominent.

J. Kissel
WP #5118

After building the MEDM infrastructure for the calibration lines and optical gain compensation (a.k.a. "gamma"; see ) I realized a large fraction of calibration line path from its generation deep inside the GAMMA_LINE1_DEMOD block to its injection into the DARM loop has no readbacks or test points. Similarly, the calculation of each optical gain compensation coefficient has a ton of steps with out EPICs records to show the progress along the way.

As such, I've installed several new EPICs records in the h1calcs.mdl (or, more accurately in the CAL_CS_MASTER.mdl and CAL_GAMMA_CALC_MASTER.mdl library parts) and installed an EPICs record / Test Point pair in the h1omc.mdl (again, more accurately in the omclsc.mdl library part), right before its injected into the DARM loop. See attached screen shots. 

I've chosen to store the new test point at the full 16 [kHz] in the frames, because I expect we'll want to make comparisons of the CALCS to OMC model delay over long stretches of time in the future to ensure the phase lag between generation and injection remains stable against, say, computer reboots. The new test point channel in the OMC will be
${IFO}:OMC-LSC_CAL_LINE_SUM_DQ
where the already existing channel in the CALCS model is
${IFO}:CAL-CS_LINE_SUM_DQ
remember that this is *sum* of all calbiration lines. The test point for each individual line,
${IFO}:CAL-CS_GAMMA_LINE${lineNum}_DEMOD_LO
 is *not* stored in the frames. We expect that the lines will be sufficiently far apart that a simple band-pass on the noise-free digital signal should be adequate to isolate each line if necessary.

These new models have been compiled and committed to the userapps repo, and I'll install them tomorrow morning and restart the DAQ.

= Ops Summary =

7:00 - 8:00 Cris and Karen to LVEA

~9:00 Filiberto to LVEA near HAM4 (grab some stuff)

9:26 Karen to Mid Y and Cris to Mid X

11:19 Karen leaving Mid Y

11:50 Kyle to Mid X

12:30 - 13:00 ITMX and ITMY cable swap (Jeff K. Refer to alog17466 ) 

16:24 Jeff K. to log himself out at CDS highbay

It's been a quiet day today....

Scott L. Ed P. Chris S.

Relocated lights and equipment to next section of tube, single door to X-1-7 double doors. Refilled D I water tank and cleaned 15 meters of tube this afternoon.
Tube pressures monitored by control room operator during cleaning operations.

I added a simple script to /opt/rtcds/userapps/release/isi/h1/scripts/ called BRSswitch.py

You can use this to turn ON or OFF the BRS sensor correction. All this does is change the X2 and X3 matrix values in the STS2CART matrix so that when it is ON the the BRS signal is subtracted, and when it is OFF the signal stops there.

To run this script, navigate to /opt/rtcds/userapps/release/isi/h1/scripts/

Then:

python BRSswitch.py ON

or

python BRSswitch.py OFF

The purpose of this script is to eliminate confusion over how to turn it on or off and to allow for a method that can adapt to changes in the BRS system. As of now this script is specific to this BRS at EX, but can easily be made more general when/if there are more here or at LLO.

J. Kissel, F. Clara, E. Hall

The commissioning vanguard had discovered that a portion of the ITMX M0 Top Driver's Binary Input/Output (BIO), which is control of the low-pass and test/coil enable switches, was malfunctioning a few days ago (see LHO aLOG 17449). This afternoon, Fil, Evan, and I ventured into the CDS highbay to diagnose the problem. After a cable-to-cable swap at the coil driver between ITMX M0 F1F2F3SD BIO cables and the ITMY R0F1F2F3SD BIO cables, and cycling both suspensions' top mass digital control and readbacks, we identified that *one* bit (the F1 bit) of the ITMX M0 F1F2F3SD BIO remained stuck (with the otherwise functional ITMY cables). This made us suspicious of the BIO chassis. However, due to time constraints, we couldn't diagnose further. Instead, we restored the nominal configuration -- and this FIXED the problem. Thus, happily, the problem is gone and all of ITMX driver channels can switch as expected, but, sadly, we don't know why our cable Chinese fire drill fixed the problem.

-----------
Other details:
Follow along using pgs 8 through 14 of D1100022 where the BIO connections happen.
Remember that the top mass cabling -- even through the SUS have 6 OSEMs on the main and reaction chain -- is grouped in groups of four OSEMs. For a given QUAD, the break down is
Set 1: M0F1 M0F2 M0F3 M0SD
Set 2: M0LF M0RT R0LF R0RT
Set 3: R0F1 R0F2 R0F3 R0SD
thanks to the mechanical arrangement of the cabling in vacuum (see T1100327).

What we saw initially:
When we requested to switch the M0 "coil driver" (the main chain uses one-and-a-half top drivers) filter state request to go from State 1 to 2, it switched only the M0F1F2F3SD set, and it switched the test/coil enable bits instead of the filter state.
This was reproducible over many attempts. All other functionality -- switching ITMX R0 state and test/coil enable was functional -- indicating a problem only with this cable/OSEM set.

We confirmed that all cables were connected and secured screwed in as designed in D1100022 before doing anything.

We swapped the cables at the coil driver, namely (on pg 9 of D1100022) we exchanged cables CAB_L1:SUS_ITMX-102 and CAB_L1:SUS_ITMX-103 in the SUS-C5-37C top driver position/chassis with the CAB_L1:SUS_ITMY-100 and CAB_L1:SUS_ITMY-101 in the SUS-C5-39C top driver position/chassis. With all cables connected in this fashion, the ITMY M0 and R0 BIO was entirely functional bhaving as expected. The ITMX M0 and R0 BIO was *more* functional, in that switching the filter state request actually switched the filter, EXCEPT that the F1 LP bit remained stuck. We then quickly assumed that this series of events was symptomatic of a problem with the ITMX BIO chassis, but we ran out of time to create enough test scenarios that we're confident.

Running out of time, we re-exchanged the cabling, restoring everything to how it was, and to how its laid out in D1100022. However, after toggling the ITMX and ITMY BIO switches again, to confirm that functionality had been restored to ITMY and the same broken behavior was present on ITMX, we found that all problems had been fixed. YUCK. Well, YAY, but... YUCK.

We'll keep an eye on this BIO chassis to see if it goes screwy again, but for now we consider the problem (dis-satisfyingly) fixed.

The BRS software was restarted at ~12:30PM.  The device is currently disabled. The INMON signal is very clipped/saturated. The wind speed at EX is currently 20mph+.  We still think there is a fault with the equipment. Krishna should be contacted.

Here is my contribution to the range worsening during the last long lock, following up on Lisa's and Sheila's entries.

Basically, my conclusion is that high wind affects the angular stability of the interferometer, the noise gets very non stationary (as visible from the summary page spectrogram and as it was observed in the past). 

I loaded all 4 hours of lock data, computed the band-limited RMS of DARM in the 100-200 hz region and correlate it with angular signals (both local sensors and WFS/QPD). For the first couples of hours the noise is lower and more stationary (still, there is some fluctuation that is correlated with angular motion). The first attached plot shows the trend of the BLRMS during the whole lock, together with the reconstruction using the WFS and QPD signals. The second and third plots are zooms into a quiet and a not so quiet time periods. 

The final attachment is the usual ranking of the signals that contribute most to the fit. AS_B_45_YAW is the clear winner, both I and Q.

P.S. The second attachment is broken and I can't remove it, ignore it.

Looking at the DARM spectrogram, there are few lines that wander around in frequency over long periods of time:

• 166 Hz, showing high coherence with SRCL
• 235 Hz, showing coherence with PSL Periscope (not sure if this is the real origin of the line)
• 250 Hz, showing coherence with PRCL
• 416 Hz, showing coherence with SRCL/PRCL and moving in frequency like the 166 Hz.
• 970 Hz, showing coherence with PRCL

The attache spectrograms show all these lines. I wrote I simple peak identification script (attached) to extract the frequency as a function of time. It's not perfect, so the traces are kind of noisy, but they should be good enough to try to correlate them with some slow channels (power, temperature, angular drifts, etc.). Unfortunately it's too painful to access remote data for me right now, so I'll let this analysis to somebody else...

Attached a txt file with the frequency of the lines a function of time (firt column) measured in seconds from GPS time 1111224616

Now the whole initial alingment procedure can be done using ALIGN_IFO, which will hopefully reduce confusion and the need to init different guardians.  We have replaced the ISC DOF with the new diagnostic guardian on the ISC overview screen.  

Also, the states which used to be called SRM ALIGN and OFFLOAD SRM ALIGN are now SRC ALIGN ect since they are really aligning both SR2 and SRM.  Nutsinee and Ed are working on updating the procedure for operators.  

Laura, Josh, Andy, Joe, Detchar

We took a look at the 28 Mpc lock on 24th March to see if there was any evidence of arches that we have seen previously at LLO (for example LLO alog 17090).

The first attachement shows a histogram of the rate of glitches (omicron triggers) binned by the value of IMC-F at the same time. The plot looks pretty Gaussian, except for a few bins specifically between IMC-F values of -37.5 to -41.7 and -45.8 and -50. Between these values we see evidence of arches in DARM. The second attachment is a pdf file showing the arches in DARM lining up with values of IMC-F in the second bin. Here are links to many spectograms confirming the arches in the IMC-F bins outlined (spectrograms for IMC-F bin -37.5 to -41.7 / spectrograms for IMC-F bin -45.8 to -50). Each spectrogram time corresponds to a glitch as seen by omicron.

LLO has dramatically reduced the effect of these arches, see LLO alog 17191.

Also the value IMC-F has when these arches appear in DARM seems to drift, i.e. there does not appear to be a specific value of IMC-F which produces the arches (this is evident in the second attachement). This is not what we have seen at LLO, does anyone know what could be causing this?

Time Code Translator work at end stations, WP5116

Dave, Jim:

We racked up the missing TCT in EY, then connected it to one of the unused fiber pairs running between the VEA rack and the computer rack. We verified it was using ports 19 and 20 in the MSR (same as EX). Once connected the TCT synced to the time and the error LED on the MSR sender was extinguished.

We then ran 30 foot 50-Ohm coax, BNC terminated, between the TCT in the computer rack, through the wall, to the first port on the CER comparator. This was done at both EX and EY. The 1PPS port on the back of the TCT is a TNC connector, so we used a TNC-BNC adapter on that port for EX. We need to do the same for EY.

At EX, the comparator is now displaying a time of -22.2uS on the TCT port, roughly the time of flight between the buildings. We will revisit the TCT internal settings, they are most probably what was left from iLIGO.

Rebuild of ASC model

Daniel, Dave:

Daniel though h1asc would benefit from a clean build, install and restart. This was done, I'll see if this helped in the IPC errors.

PCAL filter modules, port from PEM file

Sudarshan, Dave:

Sudarshan discovered that I had forgotten to migrate the PCAL filters from when the CAL system shared a core with the PEM. He copied them from H1PEME[X,Y].txt to H1CALE[X,Y].txt, making a CAL_PCAL to PCAL name change in the process. The filters were then loaded into the PCAL models at both end stations.

Since we are frequently running out of range on the end station VCOs, when there is high wind, we switched the drive of the fiber AOM back to the IMC VCO. This was the original scheme. It has the disadvantage that the end station green locking now depends on the IMC to be locked. On the other hand, this is probably not a big deal, when working with the fully locked interferometer.