Betsy, J. Kissel, Nutsinee

Today we restored all the band-limitting and low-pass filters to the violin monitor (alog20660). The band-limitting filter gains have been restored to 100.

Look for a subsequent alog regarding sign change troubles in ASC from commissioners later.  However, because Stefan was lamenting that the alignment of the SRC seemed to be problematic of late, we trended the SR3 PIT pointing during OPLEV damping control epochs and CAGE servo control epochs.  We discovered that the "bad pointing" of SR3 corrected by commissioners on last Thur Aug 13 (20519) was restored somehow on the day the cage servo was implemented Sunday Aug 16 (20571).  So, today we turned the cage servo off, manually restored the SR3 PIT pointing, cleared the offset  and turned it back on.  Note - ON, OFF, and CLEAR were all done via the servo's Guardian.  Hopefully this will help with the unstable ASC matrix problems, but we'll see... 

See attached trend which represents the pointing history of SR3.

Continuing on the task of summarizing the SVN status of CDS code, here is the guardian python user code list:

M       /opt/rtcds/userapps/release/als/common/guardian/ALS_ARM.py
M       /opt/rtcds/userapps/release/als/common/guardian/ALS_GEN_STATES.py
M       /opt/rtcds/userapps/release/isc/h1/guardian/lscparams.py
?       /opt/rtcds/userapps/release/isc/h1/guardian/TEST_BOUNCE_ROLL_DECORATOR.py
M       /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/ISI_STAGE/edges.py
M       /opt/rtcds/userapps/release/omc/h1/guardian/omcparams.py
M       /opt/rtcds/userapps/release/sys/common/guardian/ifolib/CameraInterface.py
M       /opt/rtcds/userapps/release/sys/h1/guardian/IFO_NODE_LIST.py
M       /opt/rtcds/userapps/release/sys/h1/guardian/SYS_DIAG_tests.py

to get the list of python files I did the following:

looking at the archive of the guardian logs under /ligo/backups/guardian, I made a list of log files created in the past 19 days (for the month of August). For each log file I grepped for "user code:" to get the source py file. This gave a list of 79 files. For each file I checked its SVN status.

A brute force coherence report can be found here:

https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_1123740797/

I’m using data from Evan’s elog 

It took some time to process this since first data was not available on ldas-pcdev1.ligo-wa.caltech.edu due to some maintenance, and then my home folder was not available on ldas-pcdev1.ligo.caltech.edu due to some other maintenance. However, all’s well that ends well

• Below about 15 Hz there is coherence with DCPD_NULL, and this coherence is quite large in the 2-7 Hz region. I don’t know how to interpreter this, but I’m sure I’ve never seen it before (figure 1)
• There’s a bit of coherence below about 14 Hz with ASC-SRC2_Y (figure 2)
• Coherence with SRCL is at about 20%, so not limiting right now (figure 3)
• Coherence with MICH is somewhat higher, but still not limiting (figure 4). We are very close in the 35-45 Hz region.
• Between 15 and 30-40 Hz there is very large coherence with DHARD PITCH: this is limiting the sensitivity between 15 and 25-30 Hz! (figure 5) The same coherence is visible with ITMX and ITMY L2 coil signals (figure 6)
• Two “lines” at 27.25 and 28.25 Hz shows high coherence with PRCL
• In the regions 50-60 Hz there is some (not very high) coherence with ASC-X_TR_B_YAW. Is this beam jitter? (figure 7). Instead, between 77 and 78.75 Hz there is large coherence with ASC-X_TR_B_YAW_OUT (figure 7)
• Figure 8 shows the coherence with OMC-LSC_DITHER: there are few lines here an there that are coherent, even at low frequency (the lowest one is 64 Hz). Is this some kind of non-linear up/down conversion from the main dither lines? 
• At 64, 128 Hz and 256 Hz there is coherence with OMC-LSC_DITHER again but also with PEM_EY_MAG_EBAY_SUSRACK_X/Y/Z. Having coherence at 64, 128, 256 Hz is very suspicious!
• At 64 Hz there is coherence with a microphone (PEM-EY_MIC_VEA_PLUSY) and a magnetometer (PEM-EY_MAG_EBAY_SEIRACK_Z) and the end Y station
• Coherence with AS_A/B/C_SUM is still present at high frequency. This is our 45 MHz amplitude noise (figure 10). The same level of coherence is visible with ASC-OMC_A/B/C_SUM (figure 11)
• Good news is that the coherence with the periscope accelerometer is low (figure 12), although I still can guess the shape of the two bumps in the sensitivity…

Yesterday I completed installing the updated DMT software (gds-2.17.2) and its dependencies on the DMT Machines at both observatories. This includes (but is not limited to) new versions of:

 * h(t) calibration pipeline (gstlal-calibration-0.4.0)
 * SenseMonitor
 * Omega trigger generation from h(t) improved, configured and run continuously.
 * Improved monitoring of the DMT run status.

A few notes:

The latest version of SenseMonitor contains the improved anti-aliasing filtering using the decimation function written long ago by Peter for dtt. This fixes the problem that was noted at the start of ER7 where noise ner the Nyquist frequency was folded down to near DC due to inadequate anti-aliasing.

The version of the h(t) calibration pipeline was introduced because it is packaged with the new gds infrastructure. I am not qualified to comment on this verison of the pipeline, but from what I understand, it continues to run in a mode using the same calibration algorithm as the package it replaces. Further updates will be needed to start makeing the additional corrections under development by the calibration group.

The new monitoring functionality continues to generate the DMT Spi page showing the status of all monitors running. It now does a better job of checking that all monitor processes are reading data from one of the shared memory partitions. This is especially useful for monitors that were reading the calibrated h(t) data.

This non-event almost went unnoticed, but yesterday was the day h1ecatx1 should have done its every-fortnight-on-a-Tuesday crash until Carlos fixed the issue a week ago. Looks like this problem is now resolved.

Fil, Peter K., Nutsinee

Today I tried to hook up the spare IR sensor and the comparator box to the controller. Again it didn't work. I tried swapping the sensor, comparator, controller, I even tried swapping DB9 cable. A set of sensor and comparator box has been sent to EE shop for analysis (one left at the controller so the CO2X can lase). Peter and Fil found that the IR sensor works but the comparator box didn't seem to behave right. Fil has replaced the op-amp and the comparator but the output was still not what we were expecting (reduced set point but the comparator tripping point didn't get any lower). We are going to replace the switch and hope for the best. 

LVEA: Laser Hazard
IFO: Locked
Observation Bit: Commissioning  

All Times UTC

07:00 Take over from TJ.
07:00 IFO Locked at DC_READOUT
07:00 Go to INCREASE_POWER at 16w
07:21 Take IFO to 21w  
07:35 Take IFO to 23w
07:42 Go to LOWNOISE_ESD_ETMY 
07:42 Lockloss – Commissioners looking into 
07:56 Locked at DC_READOUT
09:05 Go to INCREASE_POWER at 15.8w
09:14 Take IFO to 23w
09:20 Go to COIL_DRIVERS
09:20 Lockloss – UE
09:40 Locked at DC_READOUT
09:45 Go to INCREASE_POWER at 23w
09:51 Go to NOMINAL_LOW_NOISE
09:53 Lockloss – 
10:10 Locked at DC_READOUT  
10:15 Go to LOWNOISE_ESD_ETMY
10:17 Lockloss – 
10:32 Locked at INCREASE_POWER at 23w
10:58 Locked at NOMINAL_LOW_NOISE
11:00 Set Undisturbed bit
11:37 Lockloss – 
12:01 Locked at NOMINAL_LOW_NOISE
13:30 Set Undisturbed bit

Matt asked if Detchar could check other OSEMs for problems like the ones seen in this alog. We're working on an automated solution, but for now Josh has suggested just plotting all the spectra. There are 270 channels in the attached PDF, with spectra taken in the middle of the most recent observation intent time (Aug 19 11:20 UTC). These are 60 seconds of data, 4 second FFTs, 75% overlap. We can easily make plots at other times, repeat this for L1, etc. I'll clean up the script and make it available.

Here's a quick list of channels with lines or other bad features in them. You can search the PDF to see the plots. The first two groups are the ones to be most concerned about. Detchar should follow these up, especially the 'bouncy' spectra (which may be time-domain glitching).

These channels have a distinct line, that may be aliased down like the 1821 Hz from the TMSX was:
All OM1_M1 - just below 120 Hz
All OM3_M1 - just above 90 Hz
SR3_M1 T1 - about 65 Hz
SR3_M2 LR - about 65 Hz
SRM_M1 LF, RT, and SD - about 70 Hz, and something weird at high frequencies
SRM_M2 UL - about 65 Hz
SRM_M3 UR - about 65 Hz

The following have 'bouncy' spectra, which usually means repeated glitches that are better seen in the time domain:
MC2_M1 SD
PR2_M1 T2
PR2_M3 LL and UL
SR2_M1 T1

The following all have a big 60 Hz line and the spectra above 60 Hz is not smooth. Maybe there's a forest of lines or wandering lines.
ITMX_M0 LF and RT
ITMX_R0 LF and RT
ITMY_L1 UR
ITMY_L2 UR
ITMY_R0 RT
PRM_M1 RT and SD
PR3_M1 T1 and T2
All PR3_M3
PR2_M3 UR 
MC2_M1 RT
All SR2 M1 and M2
ETMY_M0 F1, F2, and SD
All IM2_M1 

S. Dwyer, D. Hoak, J. Driggers, J. Bartlett, J. Kissel, C. Cahillane

We had trouble with the transition to ETMY, which turned out to be the same problem that we had with ETMX ESD driver this morning (20652) -- the Binary IO configuration was toggled such that high-voltage driver was disconnected, but we were still driving through the high voltage driver.  This was the source of several lock losses during this evening's maintenance recovery. We'd found it by comparing a conlog between now and during yesterday's DARM OLGTF measurements by the calibration group. Once we fixed it, we made sure to accept the configuration in the SDF (where we looked first, but it turns out that the wrong configuration had been stored there).

For the record, the attached screenshot shows the correct configuration for ETMY BIO. The good configuration is with have HI/LO Voltage OFF to run with the low voltage driver and the HI Voltage Disconnected (i..e the switch is OFF). In the same screenshot, we show what the DARM loop gain looks like with (yellow) locked on EX alone, (blue) a half-transition to EY when EY is in the bad configuration, (red) a half transition to EY with EY in the good configurtation.

I have taken the transfer function of the coil A, B, C, and D inputs to outputs on the Satellite Box (D0901284).
The measurement was taken with an SR785 sourcing the input through a Coil Driver Chassis to turn the input into a differential, and then fed through the Satellite Box.
We expected the Satellite Box to be have flat phase response and gain.  We have confirmed that to a tenth of a percent. (0.1%)
The Reference TF was taken of the Coil Driver Chassis by itself, and the residual calculation took our response from the Satellite Box Coil A over Reference TF.
The following was calculated using Satellite_Box_TF_Plotter.m located in /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER8/H1/Measurements/Satellite_Box/2015-08-17/.

Dan, Jeff, Jenne, Sheila, TJ

As part of the SUS model restarts, safe.snap restores, and other business, some of the TMs were driven with large broadband signals for a short time.  This has rung up two of the violin modes which we have, until now, never been able to damp.

Part of the issue with recovery today was due to safe.snap files that had recorded the wrong filter settings for the violin mode damping.  When the Guardian turned on the gains, the filters were in the wrong state.  This tripped the L2 watchdogs and led to a fun evening of team-building and shared struggle.

We followed Nutsinee's wiki page and restored the correct filters.  Most of the modes damped rapidly and we were able to transition to DC readout with one stage of whitening on the DCPDs.

However, the excess drive to the ETMs has excited two modes which we've never been able to damp (and, until today, were small enough that we didn't mind).  These are 504.805 and 507.195 Hz.  Right now they're not terribly high, but we'll need to damp them if we want to enable a second stage of DCPD whitening.  Probably damping these modes will be another painful story like 508.289.  Between the two, 504.805 is worse, by a factor of twenty.  Based on the accounting of modes so far I think we can be certain this is an ITMY line.

The attached plot shows that, compared to the black reference from earlier today, many of the modes have been damped, except for the two marked by the crosses.

The other two modes which we've not been able to damp, 501.811 and 507.992, are also excited, but these are slowly damping.  We'll have to see how they evolve over the next couple of days.  I think we can also be sure that 501.881 is an ITMY line.

While Travis was working on initial alignment earlier today, we found that we were once again having trouble turning on the ASC for the X green arm.  I think the trouble was that the DC centering loops' error signals (from the green ITMX camera) were a little bit too large.  This isn't something that happens too often, but it definitely caused the ASC to pull the arm away from resonance.  Engaging the ASC without the DC centering was fine, so it's definitely something with that pair of loops.

By hand, I turned the gain on the DOF 3 loop for both pitch and yaw (which handles the DC centering) to half of the nominal value.  Once the error signals were closer, I put the gain back up at the nominal value for both loops.

Since this is a particularly tricky problem to troubleshoot, I've tried to handle it in the ALS Arm guardians. The modifications were made in the "generator" states, so they are the same for each arm.  The actual gain values for each arm are stored in the alsconst.py file, which is already loaded by the generator states. 

• In the ENABLE_WFS state, if the DOF3 error signal is large for either pitch or yaw, it'll turn the gains for DOF3 on to half of nominal.  If the error signal is not too large, carry on as usual, with the nominal gain.
• In the LOCKED_W_SLOW_FEEDBACK state, if the gain is not at its nominal value (i.e., it has been turned on at half of nominal), it will wait in the run state until the DOF3 error signals are small-ish, then it will ramp the gain up to the nominal value, and allow the guardian to continue.

I have loaded this new code, but it has not been fully tested, since we have not done initial alignment since I loaded it (and that's where it'll get used).  I've tried to test the logic in the guardian shell and that all seems to be fine, but there's no test like doing it live.  If it is giving errors that seem insurmountable, I did an svn checkin of the as-found code before I started modifying it (as well as another checkin after my edits), so we can go back one svn version. 

Sudarshan, TJ, Sheila

Yesterday the ISS second loop caused 4 locklosses.  Times of the first three are 2015-08-17 22:28:26 21:52:13 and 20:54:05, plots are attached. 

The problem seems to be that the first loop diffracted power was large yesterday (13%, maybe a result of PMC misalignment? ).  The reference signal level was hard coded into the IMC guardian, which doesn't seem like a good idea since this does drift somewhat and we don't want to loose lock when it does. 

We have tried to edit the guardian so that iss_diffracted_power_target is now a dictionary, it gets set to the current value of the diffracted power before the second loop is closed.  If we're feeling brave and satisfied that we have recovered from our maintence day, we will try to engage it again.

I have checked the SVN status of the source files used to build the front end models.

To do this, from the /opt/rtcds/lho/h1/target directory I ran the script

for i in 'sort -u */src/src_locations.txt';do svn st $i;done

This produced the output:

M       /opt/rtcds/userapps/release/cal/h1/models/h1calex.mdl
M       /opt/rtcds/userapps/release/cal/h1/models/h1caley.mdl
M       /opt/rtcds/userapps/release/isc/h1/models/h1iscex.mdl
M       /opt/rtcds/userapps/release/isc/h1/models/h1oaf.mdl
?       /opt/rtcds/userapps/release/sus/common/models/DARMERR_VIOLIN_MODE_MONITOR_MASTER.mdl
M       /opt/rtcds/userapps/release/sus/common/models/FOUROSEM_DAMPED_STAGE_MASTER_WITH_DAMP_MODE.mdl

On first run through I saw that my h1iopsuse[x,y] changes to add the additional ADC card for PI were not committed, so I have taken care of that.

This entry is meant to survey the sensing noises of the OMC DCPDs before the EOM driver swap. However, other than the 45 MHz RFAM coupling, we have no reason to expect the couplings to change dramatically after the swap.

The DCPD sum and null data (and ISS intensity noise data) were collected from an undisturbed lock stretch on 2015-07-31.

Noise terms as follows:

• Shot noise
  • For 20.0 mA dc on the DCPD sum, we expect a shot noise of 8.0×10⁻⁸ mA/Hz^1/2.
• Dark noise
  • Dan and I measured this a few months ago. In full lock we use one stage of whitening, which amounts to a dark noise of 1.5×10⁻⁸ mA/Hz^1/2 or so above 100 Hz. We have at times experimented with two stages of whitening, but it requires some vigilance regarding the violin modes.
• Intensity noise
  • Coupling was measured à la Kiwamu by leaving the outer loop off and driving the inner-loop excitation point. For the noise estimation, I used the outer loop out-of-loop sensor. Since Sudarshan et al. previously found that this is artificially high, this estimation is an upper limit (though probably no more than a factor of 3, given Sudarshan's plot).
• Frequency noise [CARM sensing noise only]
  • Previously (i.e., when we locked CARM using the in-air REFL sensor), I estimated the coupling by looking at the TF in-vac REFL to DARM, and for the frequency noise estimation I used the noise of in-vac REFL as an upper limit for the frequency noise. Since we now lock with in-vac REFL, which has more light on it than in-air REFL does, I do not expect that the noise of in-air REFL is a particularly useful upper limit for the frequency noise.
  • Therefore, I have tried to instead estimate the sensing noise in CARM (from in-vac REFL, the summing-node board, the common-mode board, and PDH shot noise) in V/Hz^1/2 and propagate it (using the Izumi/Sigg expression) into a frequency noise in Hz/Hz^1/2. For the CARM optical TF I use 12.7 W/Hz at dc, with a pole at 0.48 Hz.
  • Then I have injected noise into CARM and looked at the TF from the residual of the excitation into DARM. This is the coupling TF needed to propagate the estimated sensing noise to DARM, assuming the CARM gain is high (this isn't quite true at high frequencies; the ugf is 17 kHz or so, and the gain does not exceed 20 dB until 3 kHz).
  • Since this includes sensing noise only (and not, for example, FSS noise), I have labeled it as a lower limit. In fact, the coherence between REFL 9Q and the DCPD sum is about 0.04 around 7 kHz, perhaps suggesting a frequency noise contribution of 2×10⁻⁸ mA/Hz^1/2 or so. More investigation required.
• Oscillator phase noise [not included here]
  • Stefan and I measured the coupling of the 9 MHz RF source into DARM, and using the expected performance of the OCXO estimated the noise in DARM should be 2.8×10⁻¹⁰ mA/Hz^1/2 at 1 kHz. This is quite low (and not shown on the plot).
  • This number should be replaced with Alexa's and Daniel's measurements of the phase noise. An even more satisfying number could be made by trying to measure the 9 MHz and 45 MHz phase noises in situ, closer to the EOM (e.g., at the output of the distribution system on the ISC rack).
• Oscillator amplitude noise [45 MHz only]
  • In this case we know that the 45 MHz RFAM is the dominant contributor (other than shot noise) to the DARM noise above a few hundred hertz. Here I use Stefan's estimate of the RFAM-induced photocurrent (1.8×10⁻⁸ mA/Hz^1/2), which (as he notes) is not enough to explain the excess between sum and null.
  • However, based on tonight's measurements using the new EOM driver readback, it seems that 45 MHz RFAM is enough to explain the excess. Perhaps Stefan's measurement is an underestimate, since it was done at the output of the distribution system and did not include the final rf amplifier before the EOM.
  • As for 9 MHz RFAM, Stefan and I made an estimate of the coupling from the 9 MHz source into DARM. Based on the specified performance of the OCXO, the expected noise in DARM is quite small. However, it would be better to look somewhere further downstream, as Stefan did (although I think he said the measurement was not that clean). This noise is not estimated in the plot.

The downward slope in the null at high frequencies is almost certainly some imperfect inversion of the AA filter, the uncompensated premap poles, or the downsampling filter.

There were eight separate locks during this shift, with typical inspiral ranges of 60 - 70 Mpc. Total observation time was 28.2 hours, with the longest continuous stretch 06:15 - 20:00 UTC on June 11. Lock losses were typically deliberate or due to maintenance activities.

The following features were investigated:

1 – Very loud (SNR > 200) glitches
Omicron picks up roughly 5-10 of these per day, coinciding with drops in range to 10 - 30 Mpc. They were not caught by Hveto and appear to all have a common origin due to their characteristic OmegaScan appearance and PCAT classification. Peak frequencies vary typically between 100 - 300 Hz (some up to 1 kHz), but two lines at 183.5 and 225.34 Hz are particularly strong. These glitches were previously thought to be due to beam tube cleaning, and this is supported by the coincidence of cleaning activities and glitches on June 11 at 16:30 UTC. However, they are also occurring in the middle of the night, when there should be no beam cleaning going on. Tentative conclusion: they all have a common origin that is somehow exacerbated by the cleaning team's activities.

2 – Quasi-periodic 60 Hz glitch every 75 min
Omicron picks up an SNR ~ 20 - 30 glitch at 60Hz which seems to happen periodically every 70 - 80 min. Hveto finds that SUS-ETMY_L2_WIT_L_DQ is an extremely efficient (use percentage 80-100%) veto, and that SUS-ETMY_L2_WIT_P_DQ and PEM-EY-MAG-EBAY-SEIRACK-X_DQ are also correlated. This effect is discussed in an alog post from June 6 (link): "the end-Y magnetometers witness EM glitches once every 75 minutes VERY strongly and that these couple into DARM".  Due to their regular appearance, it should be possible to predict a good time to visit EY to search for a cause. Robert Schofield is investigating.

3 – Non-stationary noise at 20 - 30Hz
This is visible as a cluster of SNR 10 - 30 glitches at 20 - 30 Hz, which became denser on June 11 and started showing up as short vertical lines in the spectrograms as well. The glitches are not caught by Hveto. Interestingly, they were absent completely from the first lock stretch on June 10, from 00:00 – 05:00 UTC. Daniel Hoak has concluded that this is scattering noise, likely from alignment drives sent to OMC suspension, and plans to reduce the OMC alignment gain by a factor of two to stop this (link to alog).

4 – Broadband spectrogram lines at 310 and 340 Hz
A pair of lines at 310 and 340 Hz are visible in the normalized spectrograms, strongest at the beginning of a lock and decaying over a timescale of ~1 hr as the locked interferometer settles into the nominal alignment state. According to Robert Schofield, these are resonances of the optic support on the PSL periscope. The coupling to DARM changes as the alignment drifts in time (peaks decay beacuse the alignment was tuned to minimize the peaks when the IFO is settled.) Alogs about this: link, link, link.

There are lines of Omicron triggers at these frequencies too, which interestingly are weakest when the spectrogram lines are strongest (probably due to a 'whitening' effect that washes them out when the surrounding noise rises). Robert suspects that the glitches are produced by variations in alignment of the interferometer (changes in coupling to the interferometer making the peaks suddenly bigger or smaller).

5 – Wandering 430 Hz line
Visible in the spectrograms as a thin and noisy line, seen to wander slightly in Fscan. It weakened over the course of the long (14h) lock on June 11. Origin unknown.

6 – h(t) calibration
Especially noisy throughout the shift, with the ASD ratio showing unusually high variance. May be related to odd broadband behavior visible in the spectrogram. Jeff Kissel and calibration group report that nothing changed in the GDS calibration at this time. Cause unknown.

Attached PDF shows some relevant plots.

More details can be found at the DQ shift wiki page.