guardian hardware injection node development tests

Chris B., Jamie R.

This aLog is documenting the work that has been done so far to get the guardian hardware injection node running at LHO. Documenting the work over the next few days could ease the installation at LLO, after we have sorted out everything at LHO.

It also includes some tidbits about the development that's been done, since several members of the hardware injection subgroup wanted to be kept in the loop.

Installations

There are only a few things we need on the guardian script machines that were not there are already. Things we have done:
 (1) Updated userapps SVN at /opt/rtcds/userapps/release/cal/common/guardian
 (2) Checked that we can instantiated guardian node with: guardctrl create INJ
 (3) Installed awg on the guardian script machine

Things we have yet to install on the guardian machine:
 * glue
 * ligo-gracedb
 * grid credentials

Codebase Development

This afternoon was mostly spent implementing several new things in the codebase. I have attached a new graph of the node to this aLog since there a number of new states, eg. new failure states, renamed the active injection states (formerly called CBC, BURST, etc.), renamed the IDLE state, and the renamed GraceDB state. And as always, the code lives in the SVN here: https://redoubt.ligo-wa.caltech.edu/svn/cds_user_apps/trunk/cal/common/guardian/

Some changes of note:
 * Changed jump transitions in successful injection pathway to edges. This changes how the node should be run. The model now is that the requested state should be INJECT_SUCCESS while running the node.
 * The modules (eg. inj_det, inj_types, etc.) have been moved to a new injtools subpackage.
 * Added success/failure messages for GraceDB events after injection is complete.
 * Added guardian decorators for a few tasks that are often repeated, eg. checking for external alerts.
 * Process managing for the awg call.

These changes have made the schedule validation script (https://redoubt.ligo-wa.caltech.edu/svn/cds_user_apps/trunk/cal/common/scripts/guardian_inj_schedule_validation.py) out-of-date, and that will need to be updated.

Also the GraceDB portions of the code have been commented out for now, since we're holding off on that for now until we have the grid credentials/how we will run the guardian process sorted out.

Tests

As I was developing I did a couple tests with the guardian node today. I did three hardware injections of constant amplitude (1e-26) into H1:CAL-PINJX_TRANSIENT_EXC. Each injection has a duration of 1 second. GPS start times are:
 * 1145497100
 * 1145497850
 * 1145500600

These tests were mostly to check that the call to awg was working properly.

The PINJX_HARDWARE filterbank has already been turned off (aLog 26748) so the signal will only appear in the PINJX_TRANSIENT and PINJX_HARDWARE channels.

In the attached plot below I shows the injection at 1145500600.

controls@h1guardian0:~$ cat /etc/apt/sources.list.d/lscsoft.list
deb http://software.ligo.org/lscsoft/debian wheezy contrib
deb-src http://software.ligo.org/lscsoft/debian wheezy contrib
controls@h1guardian0:~$ 

controls@h1guardian0:~$ grep gds /etc/guardian/local-env
. /ligo/apps/linux-x86_64/gds/etc/gds-user-env.sh
controls@h1guardian0:~$ 

PSL high frequency glitches, a little locking

Matt, Sheila

We locked and saw that we have 3 orders of mognitude or so to much noise at DC readout.  It seems this is related to PSL problems, the atached screenshot shows coherence between the ISS second loop PDs and DARM up to 100 Hz, as well as coherence between frequency noise and DARM.  The pink trace shows the coherence between intensity and frequency noise.  With this noise, we were able to power up to 10 Watts, but saturated the DC PDs.  

We looked a little but at impementing a dither loop for SRM control.  The alingment dither system currently would allow us to demodulate POP18, but POP90 has much better signal, as you would expect.  The IPC for adding this is already in the models, so I've just added it to the matrix but didn't do the model restart yet (WP5840).  

We then gave up on locking and went to the PSL racks to look at some signals at high frequencies.  We saw a glitch that happens at a repetition rate of 37 kHz, and has frequency content of nearly a MHz, which shows up in the laser intensity noise.  Matt has a picture of this in the ISS PD and the ref cav transmission.  If we turn off the ISS this is still there (as expected since its above the bandwidth), but it is harder to see on top of low frequency intensity noise.  

When the ISS is off, the ISS PDs wander from rail to rail, and oscillate only when they are near the upper rail.  

Turned off INJ-PCALX_HARDWARE filterbank

Chris B., Matt E.

Turned off the INJ-PCALX_HARDWARE filterbank output at 16:07 UTC.

From Today (April 23) to April 27, I plan to test the guardian hardware injection node. Turning off the output of INJ-PCALX_HARDWARE means that no signal will go into the detector.

OPS Eve Shift Summary

TITLE: 04/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: After battling ISS oscillations and rotation stage issues all night, I was unable to get past INPUT_ALIGN in initial alignment.  I think the IFO is tired of getting beat on this week.
LOG: See previous 2 aLogs from Sheila for details of most of the issues.
 

Locking today

Today, many of us worked to relock H1.  We've been able to get to DC readout, and had trouble powering up because of rotation stage problems we think we've solved, and as people have noted, there is something wrong with the ISS.  

Here is a list of things that were small problems:

• Wrong settings on ETMX SUS model left over from a model restart. 
• Alingment onto AS air diodes needed to be realinged
• OMs were not damped, although the guardian indicated that they were alinged.  
• We had one lockloss trying to engage ASC (part2), but that has not happened since
• We had high winds all afternoon and are now using BRS for tilt corrected sensor correction at both end stations.
• We also had a lockloss when the triple coil drivers switch, so I've moved this to the coil drivers state (later in the sequence, which doesn't solve the problem but means we won't have to worry about it for now)
• OMC PZT HV was off, also the PZT driver chassis was powered off with the rocker switch which took us a while to figure out. 
• Rotation stage issues: 26741 and comments

We spent some time trying to diagnose what was going on with the intensity noise of the laser. 

a remaining rotation stage bug

Craig, Sheila, Jenne, Travis

There is still at least one problem with the rotation stage, that sometimes it goes the wrong way when it first starts moving. 

The attached screenshot shows our first attempt at increasing the power since the vent, the rotation stage velocity and request are correctly set to 22 Watts and a velocity of 10, before the rotation stage starts moving.  When the rotation stage starts moving, you can see a drop in the input power (which happens faster than the normal rotation stage motion), the normalization (PSL-POWER_SCALE_OFFSET) is lowered by the guardian to adjust for this, but we loose lock anyway.  After the lock is lost the rotation stage proceeds at a reasonable pace to 20Watts, and the normalization follows correctly. 

This is not a new bug, we've had random locklosses like this for some time, but it is a bug that remains after last week's rotation stage fix. 

Manually over-filled CP3 at 23:01 utc

1/2 open LLCV bypass valve, and the exhaust bypass valve fully open. Flow was noted after 53 seconds, closed LLCV valve, and 3 minutes later the exhaust bypass valve was closed

changes in IMC alignment from April 4th to April 22:

Changes as measured by OSEM values: last IMC lock before laser work to stable IMC lock today

	April 4th	April 22	diff (urad)
MC1 p	-27	-13.5	+13.5
MC1 y	-1044	-1036	+8
MC2 p	500	504	+4
MC2 y	-688	-686	+2
MC3 p	-895	-895	0
MC3 y	-1027	-1018	+9
IM4 Trans p	-0.425	-0.490	-0.065
IM4 Trans y	0.103	0.081	-0.022

I worked on the IMs (IM1-4) this morning, and have verified that they are aligned to the April 4th values, so cannot account for the change in IM4 Trans.

The alignments of MC1-3 follow the alignment of the IMC input beam, so while the alignment changes for these optics are bigger than what we typically see, after the laser work they are not unexpected.

Power stabilisation work

This morning I measured the AOM drive voltage as a function of "offset slider %", and also the amount
of diffracted light both with and without the digital support loop.  All measurements were taken with
the HEPA fans running.

    With no diffracted light, the maximum output voltage on PDA was (-9.6,-9.3) V and on PDB was
(-9.65,-9.4) V.  No saturations were observed with the incident power at this level.

    Attached are the spectra and transfer function measured.  The relative power noise is about a
factor of 10 higher than it should be.  In the spectrum is a peak at ~1 kHz, the source of which
is unknown, but is probably related to the problems mentioned by Keita.  Pushing the gain higher
seems to increase the noise beyond 10 kHz.  The noise at frequencies below 100 Hz is significantly
higher than with the low power mode.

    At low frequencies the transfer function is not as clean as the measured result in the low power
mode.  I am not sure but this might be related to the linearisation of the AOM drive.

Attached are the plots for the AOM drive voltage and diffracted power as a function of
the offset slider.

    With the current settings with the offset slider at 6%, we are diffracting ~4.5 W.
The plotted and reported diffraction percentage on the MEDM screen needs to be recalibrated.

PSL ISS 3rd loop filter (for high power alignment stability)

Following up on Kiwamu's LSC model change, and in preparation for closing the ISS 3rd loop, I made a loop filter.

The idea is to make an ISS loop which sends the arm power signals (TRX and TRY) to the ISS 1st loop error point to prevent radiation pressure driven instabilities.  This loop only need to have gain around 0.5Hz (near the suspention pitch mode), so the open loop TF can be a peak around 0.5Hz with upper and lower UGFs a factor of 5 or so away (see figure 1).  The assumed plant transfer function is the interferometer coupled cavity pole aroud 0.7Hz, and the (yet to be realized) anti-aliasing filter (SR560 AC-coupled and with a pole at 10Hz, see figure 2).  The control CDS filter is shown in figure 3.

ISS issue (Sheila, Kiwamu, Jenne, Jim, Keita)

During relocking effort, for no apparent reason ISS went crazy several times (attached left), oscillating at about 1.6kHz (attached right). This is not unlike 2.5kHz oscillation reported in   alog 26666, but the frequency is different.

This is not FSS though FSS is affected by this, as the refcav was once knocked out of lock during the oscillation but the oscillation in ISS continued. It seems that FSS is affected because there's simply too much intensity noise (attached right top cyan and right bottom shows the refcav transmission DC).

HPO output itself seems to be fine (right top, brown trace), so this really sounds like something that happens in ISS.

Anyway, when this happens, do the following.

1. Open PSL_ISS MEDM.
2. Press AUTOLOCK OFF.
3. Press Manual Mode. Manual mode MEDM screen will open.
4. Press red OFF button in Manual mode MEDM.
5. If Refcav is unlocked, open the PSL_FSS MEDM, press AUTOLOCK OFF, then AUTOLOCK ON, and wait for the FSS to acquire lock.
6. Go back to Manual mode MEDM screen and press green ON button.
7. At first the red average trace in "DIFFRACTED POWER in %" on the ISS screen will be off the screen. Wait until the red trace becomes visible.
8. Press green ON+INT button.

Optical Lever 7 Day Trends

With the main laser down, it looks like some work effecting the oplevs must have happend over the last week, because some signals sit at 0 and then recover to typical levels.

I zoomed in on the Y axis to the sections of data that have reasonable signals.

Attached:

• oplev SUM
• oplev PIT
• oplev YAW

HAM11/HAM12 leak hunting

Kyle, Gerardo 

0900 - 1100 hrs. local -> In and out of LVEA, used Main Crane to "fly" leak detector over YBM.  

Dissatisfied with the slow recovery of the HAM11/HAM12 annulus system, we decided to do some unscheduled (no work permit) helium leak testing -> We found a minor leak on a conflat which was mitigated by torquing the flange fasteners but were surprised to discover several very loose chamber flange bolts, i.e. we could visually see the bolt shanks between the flanges!  In total we found (4) adjacent bolts on the flange connecting HAM11 to the OMC adapter spool to be very loose and (9) adjacent flange bolts that we could spin easily with our fingers on the flange connecting HAM12/septum/HAM11 -> these were torqued using twentieth century "old school" wrenches after we were unable to find a cell phone app.  

There are (3) pump carts running in the LVEA over the weekend, (2) at HAM11/HAM12 and (1) on the SW corner of BSC4.

Good catch! We visually inspected the CFFs on diagonal before pumping it down but did not physically touch each bolt.