Many people,

(Anyone, please add comments if I am missing something or inaccurate)

The OMC does not seem to resonate a 00 mode any more.

One hypothesis is the OMC with damaged cavity mirrors.

[Time line]

The interferometer was locked with a 50 W PSL last night (28670) with the DC readout. At around 8:13 UTC (1:13 local), the interferometer was unlocked due to an human error where an integrator of the OMC LSC servo in the digital system (FM2 of OMC-LSC-SERVO) was accidentally disengaged. 20-30 msec after the disengagement of the integrator, the laser power in HAM6, according to ASAIR_A_LF, went up to at least 150 W for a short duration of roughly 50 msec. Since ASAIR_A saturated, this power is a lower limit of the actual laser power in HAM6. In terms of energy, it is about (50 msec) * (150 W) =   7.5 J at least. According to OMC-LSC_SERVO_OUT, the OMC seemed to have escaped the resonance before the laser pulse arrived. Therefore it is unclear how much energy was actually deposited to the cavity mirrors of the OMC from this particular lockloss. 

No locking attempt was made until 16:00 UTC (9:00 AM local) in this morning. Later, the interferometer was locked with a 2W PSL with the RF readout. We noticed that the OMC were unable to acquire a 00 carrier mode at all. After one hour or so of investigation, the interferomter was intentionally unlocked. We started investigating the OMC with a single bounce configuration.

[The symptom]

No matter how we changed the length offset, the OMC did not show a visible 00 mode in the OMC trans camera. Instead, resonance the OMC went across appeared to be higher order modes with some airy disk-type halos around. In fact, we could not get a visible 01 or 10 mode either. Keita studied the effect of the OMC SUS and OM tip-tilts alignment and he was able to get a visible TEM11 mode though.

We do not think this symptom is due to some kind of misalignment --- we steered the OM mirrors and OMC suspension around by more than several 100 urad typically, but were never able to get visible 00, 01 or 10 mode in the camera. The PZT2 DC voltage monitor told us that the PZT2 was getting correct voltage.

The beam shape of OMC REFL at ISCT6 visually looked fine -- it appeared to be a gaussian beam. We steered the input optics back to where they used to be (28670) before Jenne moved them.

[Shutters were not functioning]

Daniel discovered that neither mechanical shutter nor PZT shutter had been working in the past months after the HAM6 vent on April. Richard and Daniel found that the shutter trigger box had a wrong cabling. So for the reason, we believe that the OMC and the DCPDs have been exposed to high intensity light at every lockloss. They fixed the cable and now the shutters should be triggered as intended.

We are going to try going forward with high power work tonight using RF instead of DC readout.  There is a new value in lscparams.py, "use_dc_readout".  It is currently set to zero, so guardian will not try to transition to DC readout.  When we're ready, we should just have to flip this to 1.

The plot shows that the shutters were not triggered since Apr 4, 2016.

(Stefan was working on this but I extended it to look at the other lock losses)

Plots of ASAIR_B and DCPD_SUM for last 4 lock loss

Jul 27, 2016
lockloss1: 3:48
lockloss2: 5:38
lockloss3: 6:15
lockloss4: 8:15 (Last one)

These tell us that the last one was not particular lock loss. We regularly had the similar level lock losses.

The mode which give us ~10% of transmitted light thru the OMC doesn't look like a mode of a misaligned cavity. There are multiple concentric rings around the center spot, more reminiscence of a fringe pattern with a central aperture.

This would be compatible with a worst case scenario where we have an OMC optics with a damaged coating. The DCPDs look healthy without any indication of elevated dark current. This counters our intuition where the DCPDs are most vulnerable.

We tried mode scan using a single shot beam with QPD alignment and no sensible mode was visible at all. The maximum transmission measured by DCPD_SUM was about 0.7mA or so when we expect O(100mA) for 00.

Later I found that when I misalign the OMC enough, I recover some of the sensible-looking higher order modes, but only the ones with the node at the center. We were never able to visibly identify any mode that doesn't have the node at the center.

In the attached, OMC suspension was YAWed considerably, OMC automatic alignment was disabled, and PZT was scanned a bit more than the FSR.  X axis is the PZT2 voltage, Y axis is DCPD_SUM.

Two modes visibly identified were plus-shaped HG11 type mode (i.e. 2nd order, about 8mA) and LG3 type mode (i.e. 3rd order, 6 bright spots, about 6.5mA), these both have a node at the center. These are both O(10%) of the power coming to the OMC.

We were also able to see what is arguably HG10-type mode, but one of the two bright spots was more like an ugly blob with a lot of structures in it. And this HG10-type thing is very broad compared with HG11 and LG3 type peaks.

Everything else was kind of hard to identify, but the transverse mode spacing tells us the positions of 00, 4th and 5th HOM.

It seems like 00 peak is tiny, and  even broader than the first order mode.

Attached is a trace of ASAIR_B_LF_OUT, calibrated in Watt out of HAM6. The top panel is the fatal lock-loss, the bottom one is the one before.

For the OMC REFL light; we have realigned the gigE camera and took some pictures to quantitatively assess how Gaussian the beam is.

• 1st attachment: OMC REFL camera image.
  • white dashed line is the intersection where I checked the beam profile (see the 3rd attachment).
• 2nd attachment: ASAIR camera image.
• 3rd attachment: beam profile of the OMC REFL image.
  • The fitting function is Gaussian + offset
• 4th attachment: beam profile of the ASAIR image.

The measurement was done with a 2 W PSL in a single bounce configuration (with ITMY misaligned). The OMC was in a non-resonant state where I see almost no light in the OMC trans camera. The OMs and OMC SUS was initially servoed to the nominal operating points using the ASC DC loops and the OMC SUS QPD loop.

Clearly, the OMC REFL showed some discrepancy from a pure Gaussian, but not a lot. It is unclear what optic introduced the distortion form the image. Moving the OMC REFL camera around did not improve the beam quality in the camera.

The last attachement is a tar.gz of the images in numpy npz format.

The loose cables across the walkway shown in the pictures here have now been taped to the floor for entire length of the cable in walking areas

And a different trend pattern yet lowering from 19% to 18%.

Just curious -- it's my impression that the point of "upgrading the timing system to the new V3 firmware" was to reprogram all timing system hardware's LED lights so as to not blink every second or two, because we suspect that those LEDs are somehow coupling into the IFO and causing 1 to 2 Hz combs in the interferometer response. 

The I/O chassis, IRIG-B, comparators, and RF amplifiers are a huge chunk of the timing system. Do we think that this majority will be enough to reduce the problem to negligible, or do we think that because the timing master and fanouts -- which are the primary and secondary distributors of the timing signal -- are still at the previous version that we'll still have problems?

With the I/O chassis timing upgrade we removed the separate power supply form the timing slaves on the LSC in the corner and both EX and EY ISC chassis.  Hopefully the timing work will eliminate the need for the separate supplies.

Could you clarify that last comment? Was yesterday's test of changing the LED blinking pattern
done in parallel with removal of separate power supplies for timing and other nearby electronics?



 

Ansel has been working with Richard and Robert of the past few months testing out separate power supplies for the LEDs in several I/O chassis (regrettably, there are no findable aLOGs showing results about this). Those investigations were apparently enough to push us over the edge of going forward with this upgrade of the timing system. 

Indeed, as Richard says, those separate power supplies were removed yesterday, in addition to upgrading the firmware (to keep the LEDs constantly ON instead of blinking) on the majority of the timing system. 

To clarify Jeff's comment: testing on separate power supplies was done by Brynley Pearlstone, and information on that can be found in his alog entries. Per his work, there was significant evidence that the blinking LEDs were related to the DARM comb, but changing power supplies on individual timing cards did not remove the comb. This motivated changing the LED logic overall to remove blinking.

I'm not sure whether the upgrades done so far will be sufficient to fix the problem. Maybe Robert or others have a better sense of this?

Notable alog entries from Bryn:
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=25772
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=25861
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27202

I have gone through and manually compared FScan spectrograms and
normalized spectra for the 27 magnetocometer channels that are
processed daily: https://ldas-jobs.ligo-wa.caltech.edu/~pulsar/fscan/H1_DUAL_ARM/H1_PEM/fscanNavigation.html,
to look for changes following Tuesday's timing system intervention,
focusing on the lowest 100 Hz, where DARM 1-Hz (etc.) combs are worst.

Because of substantial non-stationarity that seems to be typical,
it's not as straightforward as I hoped it would be to spot a change
in the character of the spectra. I compared today's generated FScans (July 20-21)
to an arbitrary choice two weeks ago (July 6-7).

But these six channels seemed to improve w.r.t. narrow line proliferation:

H1_PEM-CS_MAG_EBAY_LSCRACK_X_DQ
H1_PEM-EX_MAG_EBAY_SUSRACK_X_DQ
H1_PEM-EX_MAG_EBAY_SUSRACK_Y_DQ
H1_PEM-EX_MAG_EBAY_SUSRACK_Z_DQ
H1_PEM-EY_MAG_EBAY_SUSRACK_X_DQ
H1_PEM-EY_MAG_VEA_FLOOR_X_DQ  (before & after figures attached)

while these four channels seemed to get worse w.r.t. narrow lines:

H1_PEM-EX_MAG_VEA_FLOOR_Z_DQ
H1_PEM-EY_MAG_EBAY_SEIRACK_X_DQ
H1_PEM-EY_MAG_EBAY_SEIRACK_Y_DQ
H1_PEM-EY_MAG_EBAY_SEIRACK_Z_DQ

In addition, many of today's spectrograms show evidence of broad
wandering lines and a broad disturbance in the 40-70 Hz band
(including in the 2nd attached figure).

Weigang Liu has results in for folded magnetometer channels for UTC days July 18 (before changes), July 19-20 (overlapping with changes) and July 21 (after changes):

Compare 1st and 4th columns of plots for each link below.

CS_MAG_EBAY_SUSRACK_X - looks slightly worse than before the changes
CS_MAG_EBAY_SUSRACK_Y - periodic glitches higher than before
CS_MAG_EBAY_SUSRACK_Z - periodicity more pronounced as than before

CS_MAG_LVEA_VERTEX_X -  periodic glitches higher than before
CS_MAG_LVEA_VERTEX_Y -  periodic glitches higher than before
CS_MAG_LVEA_VERTEX_Z -  periodic glitches higher than before

EX_MAG_EBAY_SUSRACK_X - looks better than before
EX_MAG_EBAY_SUSRACK_Y - looks better than before
EX_MAG_EBAY_SUSRACK_Z - looks slightly worse than before

EY_MAG_EBAY_SUSRACK_Y  - looks slightly better after changes
EY_MAG_EBAY_SUSRACK_Z - looks the same after changes
(Weigang ran into a technical problem reading July 21 data for EY_MAG_EBAY_SUSRACK_X)

A summary of links for these channels from ER9 and from this July 18-21 period can be found here.