Fil, Peter K., Nutsinee

Today I tried to hook up the spare IR sensor and the comparator box to the controller. Again it didn't work. I tried swapping the sensor, comparator, controller, I even tried swapping DB9 cable. A set of sensor and comparator box has been sent to EE shop for analysis (one left at the controller so the CO2X can lase). Peter and Fil found that the IR sensor works but the comparator box didn't seem to behave right. Fil has replaced the op-amp and the comparator but the output was still not what we were expecting (reduced set point but the comparator tripping point didn't get any lower). We are going to replace the switch and hope for the best. 

LVEA: Laser Hazard
IFO: Locked
Observation Bit: Commissioning  

All Times UTC

07:00 Take over from TJ.
07:00 IFO Locked at DC_READOUT
07:00 Go to INCREASE_POWER at 16w
07:21 Take IFO to 21w  
07:35 Take IFO to 23w
07:42 Go to LOWNOISE_ESD_ETMY 
07:42 Lockloss – Commissioners looking into 
07:56 Locked at DC_READOUT
09:05 Go to INCREASE_POWER at 15.8w
09:14 Take IFO to 23w
09:20 Go to COIL_DRIVERS
09:20 Lockloss – UE
09:40 Locked at DC_READOUT
09:45 Go to INCREASE_POWER at 23w
09:51 Go to NOMINAL_LOW_NOISE
09:53 Lockloss – 
10:10 Locked at DC_READOUT  
10:15 Go to LOWNOISE_ESD_ETMY
10:17 Lockloss – 
10:32 Locked at INCREASE_POWER at 23w
10:58 Locked at NOMINAL_LOW_NOISE
11:00 Set Undisturbed bit
11:37 Lockloss – 
12:01 Locked at NOMINAL_LOW_NOISE
13:30 Set Undisturbed bit

Matt asked if Detchar could check other OSEMs for problems like the ones seen in this alog. We're working on an automated solution, but for now Josh has suggested just plotting all the spectra. There are 270 channels in the attached PDF, with spectra taken in the middle of the most recent observation intent time (Aug 19 11:20 UTC). These are 60 seconds of data, 4 second FFTs, 75% overlap. We can easily make plots at other times, repeat this for L1, etc. I'll clean up the script and make it available.

Here's a quick list of channels with lines or other bad features in them. You can search the PDF to see the plots. The first two groups are the ones to be most concerned about. Detchar should follow these up, especially the 'bouncy' spectra (which may be time-domain glitching).

These channels have a distinct line, that may be aliased down like the 1821 Hz from the TMSX was:
All OM1_M1 - just below 120 Hz
All OM3_M1 - just above 90 Hz
SR3_M1 T1 - about 65 Hz
SR3_M2 LR - about 65 Hz
SRM_M1 LF, RT, and SD - about 70 Hz, and something weird at high frequencies
SRM_M2 UL - about 65 Hz
SRM_M3 UR - about 65 Hz

The following have 'bouncy' spectra, which usually means repeated glitches that are better seen in the time domain:
MC2_M1 SD
PR2_M1 T2
PR2_M3 LL and UL
SR2_M1 T1

The following all have a big 60 Hz line and the spectra above 60 Hz is not smooth. Maybe there's a forest of lines or wandering lines.
ITMX_M0 LF and RT
ITMX_R0 LF and RT
ITMY_L1 UR
ITMY_L2 UR
ITMY_R0 RT
PRM_M1 RT and SD
PR3_M1 T1 and T2
All PR3_M3
PR2_M3 UR 
MC2_M1 RT
All SR2 M1 and M2
ETMY_M0 F1, F2, and SD
All IM2_M1 

S. Dwyer, D. Hoak, J. Driggers, J. Bartlett, J. Kissel, C. Cahillane

We had trouble with the transition to ETMY, which turned out to be the same problem that we had with ETMX ESD driver this morning (20652) -- the Binary IO configuration was toggled such that high-voltage driver was disconnected, but we were still driving through the high voltage driver.  This was the source of several lock losses during this evening's maintenance recovery. We'd found it by comparing a conlog between now and during yesterday's DARM OLGTF measurements by the calibration group. Once we fixed it, we made sure to accept the configuration in the SDF (where we looked first, but it turns out that the wrong configuration had been stored there).

For the record, the attached screenshot shows the correct configuration for ETMY BIO. The good configuration is with have HI/LO Voltage OFF to run with the low voltage driver and the HI Voltage Disconnected (i..e the switch is OFF). In the same screenshot, we show what the DARM loop gain looks like with (yellow) locked on EX alone, (blue) a half-transition to EY when EY is in the bad configuration, (red) a half transition to EY with EY in the good configurtation.

I have taken the transfer function of the coil A, B, C, and D inputs to outputs on the Satellite Box (D0901284).
The measurement was taken with an SR785 sourcing the input through a Coil Driver Chassis to turn the input into a differential, and then fed through the Satellite Box.
We expected the Satellite Box to be have flat phase response and gain.  We have confirmed that to a tenth of a percent. (0.1%)
The Reference TF was taken of the Coil Driver Chassis by itself, and the residual calculation took our response from the Satellite Box Coil A over Reference TF.
The following was calculated using Satellite_Box_TF_Plotter.m located in /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER8/H1/Measurements/Satellite_Box/2015-08-17/.

Dan, Jeff, Jenne, Sheila, TJ

As part of the SUS model restarts, safe.snap restores, and other business, some of the TMs were driven with large broadband signals for a short time.  This has rung up two of the violin modes which we have, until now, never been able to damp.

Part of the issue with recovery today was due to safe.snap files that had recorded the wrong filter settings for the violin mode damping.  When the Guardian turned on the gains, the filters were in the wrong state.  This tripped the L2 watchdogs and led to a fun evening of team-building and shared struggle.

We followed Nutsinee's wiki page and restored the correct filters.  Most of the modes damped rapidly and we were able to transition to DC readout with one stage of whitening on the DCPDs.

However, the excess drive to the ETMs has excited two modes which we've never been able to damp (and, until today, were small enough that we didn't mind).  These are 504.805 and 507.195 Hz.  Right now they're not terribly high, but we'll need to damp them if we want to enable a second stage of DCPD whitening.  Probably damping these modes will be another painful story like 508.289.  Between the two, 504.805 is worse, by a factor of twenty.  Based on the accounting of modes so far I think we can be certain this is an ITMY line.

The attached plot shows that, compared to the black reference from earlier today, many of the modes have been damped, except for the two marked by the crosses.

The other two modes which we've not been able to damp, 501.811 and 507.992, are also excited, but these are slowly damping.  We'll have to see how they evolve over the next couple of days.  I think we can also be sure that 501.881 is an ITMY line.

While Travis was working on initial alignment earlier today, we found that we were once again having trouble turning on the ASC for the X green arm.  I think the trouble was that the DC centering loops' error signals (from the green ITMX camera) were a little bit too large.  This isn't something that happens too often, but it definitely caused the ASC to pull the arm away from resonance.  Engaging the ASC without the DC centering was fine, so it's definitely something with that pair of loops.

By hand, I turned the gain on the DOF 3 loop for both pitch and yaw (which handles the DC centering) to half of the nominal value.  Once the error signals were closer, I put the gain back up at the nominal value for both loops.

Since this is a particularly tricky problem to troubleshoot, I've tried to handle it in the ALS Arm guardians. The modifications were made in the "generator" states, so they are the same for each arm.  The actual gain values for each arm are stored in the alsconst.py file, which is already loaded by the generator states. 

• In the ENABLE_WFS state, if the DOF3 error signal is large for either pitch or yaw, it'll turn the gains for DOF3 on to half of nominal.  If the error signal is not too large, carry on as usual, with the nominal gain.
• In the LOCKED_W_SLOW_FEEDBACK state, if the gain is not at its nominal value (i.e., it has been turned on at half of nominal), it will wait in the run state until the DOF3 error signals are small-ish, then it will ramp the gain up to the nominal value, and allow the guardian to continue.

I have loaded this new code, but it has not been fully tested, since we have not done initial alignment since I loaded it (and that's where it'll get used).  I've tried to test the logic in the guardian shell and that all seems to be fine, but there's no test like doing it live.  If it is giving errors that seem insurmountable, I did an svn checkin of the as-found code before I started modifying it (as well as another checkin after my edits), so we can go back one svn version. 

Jeff K, Jeff B, Dan, Jenne, Sheila

 -3.4 AS36 A I and -1 AS36 B I.  Note that the sign has flipped for AS36B.  

We can not use this matrix at 3 Watts, but it has been better consistently tonight at 15 Watts and above than the original, which was -3 ASA36I +-0.5 AS36BI.  I've added a few lines to the increase power state that changes this matrix once the power is above 15 Watts. 

Right now we are locked stably, but there is large noise that is coherent with ASC.  In the morning if inital alingment is redone it this matrix might not be usefull anymore, in that case you can comment out lines 2175 through 2177 of the ISC_LOCK guardian.

I started the shift taking over for Travis trying to lock DRMI. Sheila suggested another Initial Alignment, so we did and watched for the control signals to settle down between each step. This helped significantly. This eventually brought us up past DRMI but we had a few instances of the PUM watchdogs tripping (see Jeff's alog).

When the PUMs stopped tripping we then saw that the violin modes where very rung up. Dan worked his magic on those while showing me how it works. We got them down to a managable level and then rang into a problem in COIL_DRIVERS. Lost lock several times here, and the commissioners are working on a fix.

J. Kissel, S. Dwyer

Since we always seem to give Master Warner a hard time when he's just trying to do what we've asked, Shiela asked me to prove that commissioning HAM1 HPI, in attempts improve CHARD performance, is worth our time. As such, I attach a bunch of plots that supplement Jim's comparison of LLO vs. LHO HAM1 HEPI performance (see LHO aLOG 20538) taken while we're still trying to recover. I compare (regrettably uncalibrated) pitch and yaw ASDs of the REFL WFS against both the L4Cs in HAM1 and the GS13s in HAM3. We see that HAM1 is coherent with the features seen in the REFL WFS at 5-7 Hz and around the microseism -- which is a region which we can improve with HEPI inertial isolation -- and we see that our good ol' friend, the 0.6 [Hz] in HAM3, (see Integration Issue 1005) is coherent at ... 0.6 Hz, and indeed dominates a good fraction of total the RMS.

So, while the CHARD problem is complicated, because the REFL WFS contain CHARD motion, PR3 / PR2 motion, Input Beam Motion, some CSOFT, and their own motion, we can at least see that some protions of these signals can be improved if SEI were improved, which means we might be able to increase the bandwith of CHARD which would yield a more robust and stationary IFO. 

Sudarshan, TJ, Sheila

Yesterday the ISS second loop caused 4 locklosses.  Times of the first three are 2015-08-17 22:28:26 21:52:13 and 20:54:05, plots are attached. 

The problem seems to be that the first loop diffracted power was large yesterday (13%, maybe a result of PMC misalignment? ).  The reference signal level was hard coded into the IMC guardian, which doesn't seem like a good idea since this does drift somewhat and we don't want to loose lock when it does. 

We have tried to edit the guardian so that iss_diffracted_power_target is now a dictionary, it gets set to the current value of the diffracted power before the second loop is closed.  If we're feeling brave and satisfied that we have recovered from our maintence day, we will try to engage it again.

J. Kissel, T. Shaffer, J. Driggers, S. Dwyer

In addition to the one big blast of the PUM's Analog RMS Current Watchdog Trips this morning, at ~16:50 UTC (mentioned briefly in LHO aLOG 20654), we had three more instances of this silly watchdog tripping, which required drives to the end station.

Attached is a trend of the past 7 hours. The sad thing, is that there is no pattern. 
(1) The ITMs (Blue and Green) appear to *have* tripped when we've sent a blast to the PUM, but we've sent similar blasts since and they do not trip.
(2) ETMX (Black) has tripped when there is no signal sent to it.
This is why we always get into the blame game when we get a rash of these trips: 
- CDS says "What're you doing differently with the drive? It's gotta be you!" and 
- Users say "We're not doing anything different. It's a Tuesday! What did you do near these drivers?"
and nothing gets resolved.

Will someone please write an ECR to remove these Analog PUM WDs? 
The first action in said ECR should be to find what components this which dog is supposedly protecting, and to determine if it is actually protecting anything and/or if that component is actually in danger.
If first action deems that these are worthless, we should just rip them out. If-and-only-if they're not worthless and they're actually protecting some sensitive component that is unique to the PUM driver, then the second action would be to really study these circuits to understand what trips them, if the threshold is set correctly, and why there are so many false alarms. The third action should be, then, to fix them.

The Message: When these watchdogs trip -- especially when just one quadrant trips, and we send any number of large amounts of drive to the L2 stage, we ring up violin modes, and we suffer even more down time.

J. Kissel, B. Weaver
WP #5444
ECR E1500346
II 1097

We've completed the removal of the the violin mode monitoring from the QUAD models this morning. See attached model change screen shots. 

The good news: It has reduced the end-station's QUAD model's CPU clock-cycle turn-around-time from a range of (57 - 67) to a range of (53 - 61); see attached trend (DCUID 88 = ETMX, DCUID 98 = ETMY), which puts these models back under the 1/16384 [hz] = 61 [us] limit, so we should see a drastic reduction in clock-cycle overage timing errors.

The bad news, in the copy over, we've "lost" all of the band-limiting and low pass filters. They're not truely lost, because they're in the filter archive, but because the filter banks no longer exist in the QUAD models, they don't exist in the current foton file, so it's going to be an arduous, by-hand copy and paste between files. Nutsinee and Betsy will work on restoring these tomorrow.

The new front-end library part that is now a part for the violin mode damping that now lives in the OAF model:
/opt/rtcds/userapps/release/sus/common/models/DARMERR_VIOLIN_MODE_MONITOR_MASTER.mdl
which has been committed to the repo. I've also committed the new local version of the top-level model,
/opt/rtcds/userapps/release/isc/h1/models/h1oaf.mdl

They've been removed from this sub-library part of the QUAD_MASTER library:
/opt/rtcds/userapps/release/sus/common/models/FOUROSEM_DAMPED_STAGE_MASTER_WITH_DAMP_MODE.mdl

This closes out the above mentioned ECR and Integration Issue for LHO. 
For LLO to update:
(1) Update the above mention library SUS common library folder
(2) Update the LLO copy of the h1oaf.mdl model for demonstrative purposes
(3) Install SUS block as shown into l1oaf.mdl
(4) The new(ish) screens (see LHO aLOG 20378
for these can be found here:
/opt/rtcds/userapps/release/sus/common/medm/quad/
SUS_CUST_QUAD_L2_VIOLIN_MON_ALL.adl
SUS_CUST_QUAD_L2_VIOLIN_MON.adl

I've also saved new SDF tables such that the channels moved from the QUAD are no longer NOT FOUND.

Here is the current SVN status of the front end filter file's svn status.

A reminder, in the /opt/rtcds/lho/h1/chans/ directory, all the filter files should be symbolic links to the appropriate userapps filterfiles area.

I found that the following chans directory files are not symbolic links:

H1CAL[CS,EX,EY].txt, H1SUSETM[X,Y]PI.txt

I found that all symbolic links used the absolute path of /opt/rtcds/userapps/release/ except for:

../../../userapps/release/sus/h1/filterfiles/H1SUSBS.txt

Here is the list of the svn status:

M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM1.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM6.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM2.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM3.txt
M       /opt/rtcds/userapps/release/isi/h1/filterfiles/H1ISIHAM3.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM4.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM5.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIITMY.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIBS.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIITMX.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPRM.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPR3.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSRM.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSR3.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSOMC.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSITMY.txt
M       ../../../userapps/release/sus/h1/filterfiles/H1SUSBS.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSITMX.txt
M       /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMCS.txt
M       /opt/rtcds/userapps/release/isc/h1/filterfiles/H1OAF.txt
M       /opt/rtcds/userapps/release/lsc/h1/filterfiles/H1LSC.txt
M       /opt/rtcds/userapps/release/omc/h1/filterfiles/H1OMC.txt
M       /opt/rtcds/userapps/release/lsc/h1/filterfiles/H1LSCAUX.txt
M       /opt/rtcds/userapps/release/asc/h1/filterfiles/H1ASC.txt
M       /opt/rtcds/userapps/release/asc/h1/filterfiles/H1ASCIMC.txt
M       /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMMX.txt
M       /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMMY.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMY.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIETMY.txt
M       /opt/rtcds/userapps/release/isc/h1/filterfiles/H1ISCEY.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMX.txt
M       /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIETMX.txt

and here are the scripts which generated it (running in the chans directory):

for i in 'awk '{print $1}' ../cds/model_info.txt';do echo $i|tr [a-z] [A-Z]|awk '{print $i".txt"}'>>/tmp/filtfilelist.txt;done

for i in 'cat /tmp/filtfilelist.txt' ;do ls -al $i >> /tmp/filelonglisting.txt;done

for i in 'grep  "->" /tmp/filelonglisting.txt |awk 'BEGIN{FS=" -> "}{print $2}'';do svn st $i;done

I have checked the SVN status of the source files used to build the front end models.

To do this, from the /opt/rtcds/lho/h1/target directory I ran the script

for i in 'sort -u */src/src_locations.txt';do svn st $i;done

This produced the output:

M       /opt/rtcds/userapps/release/cal/h1/models/h1calex.mdl
M       /opt/rtcds/userapps/release/cal/h1/models/h1caley.mdl
M       /opt/rtcds/userapps/release/isc/h1/models/h1iscex.mdl
M       /opt/rtcds/userapps/release/isc/h1/models/h1oaf.mdl
?       /opt/rtcds/userapps/release/sus/common/models/DARMERR_VIOLIN_MODE_MONITOR_MASTER.mdl
M       /opt/rtcds/userapps/release/sus/common/models/FOUROSEM_DAMPED_STAGE_MASTER_WITH_DAMP_MODE.mdl

On first run through I saw that my h1iopsuse[x,y] changes to add the additional ADC card for PI were not committed, so I have taken care of that.

There were eight separate locks during this shift, with typical inspiral ranges of 60 - 70 Mpc. Total observation time was 28.2 hours, with the longest continuous stretch 06:15 - 20:00 UTC on June 11. Lock losses were typically deliberate or due to maintenance activities.

The following features were investigated:

1 – Very loud (SNR > 200) glitches
Omicron picks up roughly 5-10 of these per day, coinciding with drops in range to 10 - 30 Mpc. They were not caught by Hveto and appear to all have a common origin due to their characteristic OmegaScan appearance and PCAT classification. Peak frequencies vary typically between 100 - 300 Hz (some up to 1 kHz), but two lines at 183.5 and 225.34 Hz are particularly strong. These glitches were previously thought to be due to beam tube cleaning, and this is supported by the coincidence of cleaning activities and glitches on June 11 at 16:30 UTC. However, they are also occurring in the middle of the night, when there should be no beam cleaning going on. Tentative conclusion: they all have a common origin that is somehow exacerbated by the cleaning team's activities.

2 – Quasi-periodic 60 Hz glitch every 75 min
Omicron picks up an SNR ~ 20 - 30 glitch at 60Hz which seems to happen periodically every 70 - 80 min. Hveto finds that SUS-ETMY_L2_WIT_L_DQ is an extremely efficient (use percentage 80-100%) veto, and that SUS-ETMY_L2_WIT_P_DQ and PEM-EY-MAG-EBAY-SEIRACK-X_DQ are also correlated. This effect is discussed in an alog post from June 6 (link): "the end-Y magnetometers witness EM glitches once every 75 minutes VERY strongly and that these couple into DARM".  Due to their regular appearance, it should be possible to predict a good time to visit EY to search for a cause. Robert Schofield is investigating.

3 – Non-stationary noise at 20 - 30Hz
This is visible as a cluster of SNR 10 - 30 glitches at 20 - 30 Hz, which became denser on June 11 and started showing up as short vertical lines in the spectrograms as well. The glitches are not caught by Hveto. Interestingly, they were absent completely from the first lock stretch on June 10, from 00:00 – 05:00 UTC. Daniel Hoak has concluded that this is scattering noise, likely from alignment drives sent to OMC suspension, and plans to reduce the OMC alignment gain by a factor of two to stop this (link to alog).

4 – Broadband spectrogram lines at 310 and 340 Hz
A pair of lines at 310 and 340 Hz are visible in the normalized spectrograms, strongest at the beginning of a lock and decaying over a timescale of ~1 hr as the locked interferometer settles into the nominal alignment state. According to Robert Schofield, these are resonances of the optic support on the PSL periscope. The coupling to DARM changes as the alignment drifts in time (peaks decay beacuse the alignment was tuned to minimize the peaks when the IFO is settled.) Alogs about this: link, link, link.

There are lines of Omicron triggers at these frequencies too, which interestingly are weakest when the spectrogram lines are strongest (probably due to a 'whitening' effect that washes them out when the surrounding noise rises). Robert suspects that the glitches are produced by variations in alignment of the interferometer (changes in coupling to the interferometer making the peaks suddenly bigger or smaller).

5 – Wandering 430 Hz line
Visible in the spectrograms as a thin and noisy line, seen to wander slightly in Fscan. It weakened over the course of the long (14h) lock on June 11. Origin unknown.

6 – h(t) calibration
Especially noisy throughout the shift, with the ASD ratio showing unusually high variance. May be related to odd broadband behavior visible in the spectrogram. Jeff Kissel and calibration group report that nothing changed in the GDS calibration at this time. Cause unknown.

Attached PDF shows some relevant plots.

More details can be found at the DQ shift wiki page.