We now have a problem we haven't encoutnered before. The ISC_LOCK guardian has SPM DIFFs errors, because of dead channels.  There are ten channels related to the ALS_COMM guardian listed as the SPM diffs, I don't know if these are all of them or just the first ten.  Attached screen shot shows the list, and also that we can caget these channels.  Kiwamu and I restarted the guardians, which made no difference, then we tried destroying them as well, with the same result.  

This morning Elli and I retuned the dark offsets for the transmon QPDs, after doing this we were able to do the entire CARM offset reduction without using the in air transmission PDs.  After this succeeded we closed the beam diverters, redid the initial alignment, and attempted to lock.  We failed at the TR_CARM step three times in a row, so we opened the beam diverters and re did the initial alignment.  After that we were able to lock without using the In air PDs, which is what we have been doing all day.  

The current situation is that the End X beam diverter is closed, and End Y is open.  Kiwamu re did the initial alignment in this situation, and when we have ALS locked we can see that the spots are well centered on X end transmission QPDs.  We were hopping to try locking like this, but ran into difficulties with the mich lock acquisiton attempts breaking the lock of ALS DIFF.  We saw that the beat note power has slowly drifted down over the last few weeks, (from 0.5 dBm to -0.5 dBm), so we though it could be that a small alignment kick to the BS is enough to loose the beat note completely.  Its hard to verify this theory since we don't trust the timing between the Beckhoff and the RCG.(alog 17455)  I went to the table and touched up the beat note alingment, now it is about 0 dBm.

I tracked down the guardian communication problem to the following code in the /opt/rtcds/userapps/release/isc/h1/guardian/bs_m2_switch.py module:

import ezca
ezca = Ezca()

This is NOT OK to put in any module being imported by guardian code.  This creates a second, separate instance of the EPICS interaction object, that interferes with the one created and supplied by guardian itself.  It effectively breaks all EPICS interaction in the guardian node, which is what happened here.

If you think you need to do something like this please contact me and we'll figure out a better way.

Presumably this module was newly added to the ISC_LOCK node before the problem arose, and the reported breakage happened after it was reloaded.  After removing those lines from bs_m2_switch.py everything now appears to be working normally.

As an aside, please always provide FULL DISCLOSURE when reporting problems like this.  Usually things don't break spontaneously on their own.  Whatever was being done right before you saw the problem is probably what caused it.  (Kiwamu: if you didn't know about this change when we spoke on the phone then you're off the hook).

Evan, Dan

The Guardian issue was fixed when we returned from dinner, but then the winds came.  We switched the end station ISI's to the 90mHz blends and increased the tidal gains; the common UGFs are now 0.1Hz (were 0.05Hz) and the X-arm tidal UGF is 0.1Hz (was 0.06Hz).

Initially we observed the same problem that Sheila reported, the ALS would lose lock as soon as the DRMI started to drive the mirrors.  But after mucking with the DRMI alignment for a little while (not at all sure that this helped), I have been able to acquire lock with DRMI a couple of times.  A test of applying a large DC offset into the BS longitudinal drive does not break the ALS_DIFF lock, so maybe that problem is no longer a problem.

There is now a new problem, the DRMI Guardian isn't happy with the state DRMI_LOCKED_1F_ASC (which is the requested state from the ISC_LOCK Guardian).  Upon finishing this state is recalculates the path and jumps back to LOCK_DRMI_1F, see the attached log, and the attached graph.  This has happened three times now.

Also attached is a plot of our progress since Monday night.  I am leaving the IFO set to CHECK_IR to acquire ALS data during the windy conditions.

For MICH FF, the filters used are FM6 and FM7 (which are stopbands for the violin modes and harmonics), and FM9 (which inverts the BS M2→M3 plant). The gain is 0.040 ct/ct.

For SRCL FF, the filters used are FM6 and FM7 (as above). The gain is −2.5 ct/ct.

Both feed back only to ITMY L2.

Also attached is a text file and noise budget of a good spectrum from today. I would like to remark that

1. the GWINC model used here does not take into account the low recycling gain that we see in our current locks, and hence may underestimate the true shot noise; and
2. the actuation coefficient for the ESD in this budget is a canned value, and we have seen previously that it may be an overestimate.

Thank you so much Jamie.

I think bs_m2_switch.py is a code that TJ, Sheila and I have been preparing today. I was aware of the active coding effort that TJ has been putting today, but I simply did not know that bs_m2_switch had been already loaded in the ISC_LOCK guardian. My apologies that I did not collect and did not provide all information about the coding activities on ISC_LOCK at around the time.

Dan: the ISC_DRMI assert_drmi_locked GuardStateDecorator (defined in ISC_library.py) is wrapping the methods in DRMI_LOCKED_1F_ASC, and it returns 'LOCK_DRMI_1F' if DRMI_locked() returns False.  The obvious explanation for the behavior you saw is just that DRMI_locked() was False.  Here's what's currently defined in that function (at least from what's currently committed to the USERAPPS):

def DRMI_locked():
    #log('checking DRMI lock')
    return ezca['LSC-MICH_TRIG_MON'] and ezca['LSC-PRCL_TRIG_MON'] and ezca['LSC-SRCL_TRIG_MON']

The epics problem was my bad.  I imported bs_m2_switch but didn't think of it when I called Jamie since we weren't calling it I assumed it couldn't be doing much.  Apologies.  

As I think is known, there is a filter solution for the ring heater driver to eliminate the fan noise.  It is the same basic design as was used in the Seismic Coil Driver fans.  We just need to make these available for installation soon.  Added this to my to-do list.

The 160 and 420 Hz peaks are moving in frequency with a timescale of some minutes. The moves in a coherent way, in the sense that the frequencies seem pretty much to maintain a ratio of 2.5

See my previous log entry for an analysis of the line wandering.

It turns out the 55Hz line is actually a 57Hz line, and it's the Hartmann camera.   This was explored in detail at LLO a long time ago, I should have remembered.

I have turned off the HWS box in the EX electronics rack.  This eliminated the 57Hz line and harmonics from the rack magnetometer, but for some reason the overall noise floor has increased, see attached.  Probably turning off the camera is a better solution, as Aidan recommends in the LLO log.  We'll see if this affects the noise.

K. Izumi, J. Kissel

For the record: Kiwamu drove down to the EY end station to check: *this* lock loss did *not* cause the EY ESD Driver to trip. Huh.

Looking at spectrograms of DARM during the first hour of lock (first plot) and the last hour (second plot), it seems to me that the noise is more or less stationary, but there are huge glitches. They are so big that you can even see them easily in time domain (third plot). A zoom in is visible in the fourth plot. They look like bursts of oscillations at about 5.5 kHz.

We now believe that the glitches we had in the lock stretch from this morning was due to the ISS which repeatedly unlocked. This is a typical behavior of the ISS when the diffraction power is too low. Indeed the diffracted power had been 4% on average during the time when the interferometer was locked. There was clear correlation between arm cavities' power and the ISS diffracted power. See the attached trend of some relevant channels. Elli adjusted the diffracted power in this after noon so that the diffracted power is now at 8% with only the inner loop closed.

Looking at glitch-to-glitch coupling between auxiliary channels and DARM shows a large number of glitches in the > 1kHz range that are coincident with glitches in CARM, REFL 9 PIT/YAW, and REFL 45 PIT. Interestingly, CARM is highly correlated with high frequency glitches until about 12:00:00 UTC, at which point REFL 45 PIT becomes the stronger veto.

It looks like REFL 45 Q PIT was offset by about 2500 counts during the lock, is it possible that intensity fluctuations on an uncentered WFS are showing up as alignment glitches? I've attached a time series covering 2 hours of the lock from 11 UTC to 13 UTC.

We're currently running code to see if the lower frequency (50-200 Hz) glitches are caused by zero-crossings in the 18-bit DACs.

I've attached an Omicron glitchgram for the whole day, it seems as if the higher frequency glitches and the glitches populating the 50-200 Hz region are the two dominant populations right now. There are also a few high SNR glitches scattered around in the 100-400 Hz region that we'll follow up individually.

-2^16 Crossings in ETMY L3 ESD causing many of the glitches in this lock:

In addition to the arches reported in 17452 and 17506 we found DAC glitches in this lock when ETMY L3 ESD DAC outputs were crossing -2^16 counts. Attached is a PDF with a few examples that were lined up by hand. We will follow up more closely to see if other suspensions and penultimate stages also add glitches. Note: At Livingston, SUS MC2 M3 DACs were also a problem. 

If you'd like to see the primary culprits from this long lock, here is a tar file of omega scans (thanks to Joe Areeda) of the loudest 100 glitches between 30 and 200Hz. The vertical lines that repeat are DAC glitches, the crazy wandering features are the arches described in the pages linked above, those two mechanisms account for most of the glitches we see.   

There is some cabling work that needed to be completed on this particular channel (ISCT6_ACC). Hope to bring it online on Monday.

Same story. Channel 1 opens and closes immediately. Next thing to check are cabling and slow controls.

why use the error signal for bounce mode damping, when the DARM control signal is available?

the phase of G/(1+G) changes very little as the loop gain changes in the case that G>>1 (which should be the case with the Bounce/Roll RG filters on)

Rana,

This is a good point. We are planning to change the pick off point so that it takes the DARM signal from its output as you suggested.

Dave and I looked into this somewhat and we don't think that the timestamps of the EtherCAT systems are used at all by the frame builder. As a matter of fact the internal clock of these machines was about 3 seconds off. Since it is hard to believe that the EtherCAT systems violate causality, the most likely candidates are frame builder, data concentrator or maybe the EPICS gateway. This would also mean that all data transported through EPICS to the frame builder could be affected. More investigations are pending.