IMC frequency and intensity signals, before and after HPO turn-on

The attachment shows the frequency control signal (IMC-F) and the transmitted intensity (IM4 trans) before and after turning on the HPO. 2 W into the modecleaner, with the interferometer unlocked.

Although Sheila's most recent DARM spectrum shows coherence with both frequency and intensity noise on the laser, I would hazard a guess that the frequency noise is not really the issue here, since it is only a factor of (at most) a few worse than before.

checks on guardian hardware injection node logging and plus some more development

I did a set of tests with the guardian node. The codebase is in a state that should be ready for Jamie and I to set it up tomorrow on the guardian script machine. Going forward things to do are:
 * Update docstrings
 * Install glue, gracedb, and grid credentials on guardian machine
 * Plan out how to run the gracedb process and get robot certificate
 * Do series of injections with guardian node on guardian machine - test full injection pathway, test killing active injection, test reloading schedule, test multiple injections in a row, etc.

Below I outline the tests I did.

How to do command line tests with guardian daemon

Can now do the following tests on the command line at a LHO workstation:
 * To test reading schedule and finding the next injection: guardian INJ WAIT_FOR_NEXT_INJECT
 * To test gracedb event creation: guardian INJ CREATE_GRACEDB_EVENT
 * To test awg and inject a signal from schedule into the detector: guardian INJ CREATE_AWG_STREAM INJECT_CBC_ACTIVE
 * To test schedule validation script: PYTHONPATH=/opt/rtcds/userapps/release/cal/common/guardian:${PYTHONPATH}; python guardian_inj_schedule_validation.py --ifo H1 --schedule /opt/rtcds/userapps/release/cal/common/guardian/schedule/schedule_1148558052.txt --min-cadence 300

NOTE: You will need glue and gracedb python packages to run some of these tests, and these packages are not system-installed on workstations in the control room. And for gracedb upload testing you need the grid credential tools which are not on LHO workstations. And for gracedb upload test you need to make sure dev_mode is False.

Test injections

Injections from last night are in aLog 26749.

Today I continued with some more development tests. Injections that are constant amplitude of 1e-26 for 1 second duration into H1:CAL-PINJX_TRANSIENT_EXC; start time of the injections are:
 * 1145554100
 * 1145555100
 * 1145555700
 * 1145560262

(i) Call to awg works and injection goes into INJ-PINJX_TRANSIENT_EXC.

(ii) Injections logged correctly and meta-data is propagating through infrastructure to inform the searches. Can see the hardware injection tests done with the guardian node on the detchar summary pages. The first three not logged with a injection type, eg. BURST, because in initial tests just wanted to correctly use the awg module. Can see thereafter the injections were flagged with a type in ODC and this propagates to the low-latency frames for the online searches and the segment database for the offline searches. Can see attached plots for ODC segments and segment database segments.

(iii) Destroying a node with an open stream that has trasmitted data to the front end does not perform the injection.

(iv) The gracedb upload functions have already been tested. Today I re-checked the functions and here is an example gracedb event that was uploaded T235981. Adding messages to the event log on gracedb was also tested again, notice the "This is a test." message on the T235981 gracedb page.

(v) Schedule validation script updated and tested.

Codebase developments

Some more changes:
 * There is now a dev_mode in the code to run the tests mentioned in the section above. At the moment this does two things (i) ignores to check if the detector is locked, (ii) ignores gracedb for now until we get the robot certificate sorted out, and (ii) waits in the INJECT_CBC_ACTIVE state instead of the AWG_STREAM_OPEN_PREINJECT state because we need to avoid jump transitions for the command line test above.
 * Schedule validation script works again (https://redoubt.ligo-wa.caltech.edu/svn/cds_user_apps/trunk/cal/common/scripts/guardian_inj_schedule_validation.py).

One thing of note is that guardian does not allow subprocesses to be created by states so the subprocess managment that I had written will not work with guardian. So right now once the injection starts the code will wait for the injection to finish, this is just the implementation in the awg package (see awg.ArbitraryStream.close); it can only be killed by stopping the node.

I've also renamed the base module (INJ.py) to something less generic, it is now CAL_PINJX.py.

See: https://redoubt.ligo-wa.caltech.edu/svn/cds_user_apps/trunk/cal/common/guardian/CAL_PINJX.py

So modify the examples in this aLog entry as appropriate, ie. guardian INJ becomes guardian CAL_PINJX.

Chris B., Jamie R.

Started up the node with guardctrl start CAL_INJ.

Used guardmedm CAL_INJ to control the guardian node.

Did a variety of tests with the hardware injection guardian node, these all passed:
 * Tested killing injection before injection awg call is active by requesting KILL_INJECT.
 * Tested killing the injection during awg.ArbitraryStream.close call, ie. inject is in active state, by requesting KILL_INJECT.
 * Tested scheduling injections minimum number of seconds apart to make sure guardian picked the correct injection.
 * External alert happened while injection was scheduled, aborted injection successfully from AWG_STREAM_OPEN_PREINJECT to ABORT_INJECT_FOR_EXTTRIG. Commented out this check to continue working.
 * Tested out of order schedule file.
 * Tested FAILURE_READ_WAVEFORM, eg. waveform file does not exist.
 * Tested all injection states (INJECT_CBC_ACTIVE, INJECT_BURST_ACTIVE, INJECT_STOCHASTIC_ACTIVE, INJECT_DETCHAR_ACTIVE).
 * Tested that injection does not go into the detector if we turn off dev_mode so that it checks that detector is locked.
 * Injection start, injection end times, injection outcome values are all being set on MEDM screen.

Made another failure mode. If the call to awg.ArbitraryStream.close is too close in time to the start of the injection, then there is a error. Added FAILURE_DURING_ACTIVE_INJECT. awg returns a generic AwgStreamError so without doing some hacked parsing of the error message, there's not much to differentiate why it failed during the function call.

None of the gracedb functionality was tested during this, since we need to create a robot certificate still.

After doing a few more tests, I've started scheduling a long series of injections. The schedule file is here:
https://redoubt.ligo-wa.caltech.edu/svn/cds_user_apps/trunk/cal/common/guardian/schedule/schedule_1148558052.txt

Injections were still going this morning from last night as expected, every 400 seconds.

Attached is an hour of last nights injections. Also I've attached a zoomed in plot on the fast channel for one injection to check the timing of the start of the injection. Looks good.

guardian hardware injection node development tests

Chris B., Jamie R.

This aLog is documenting the work that has been done so far to get the guardian hardware injection node running at LHO. Documenting the work over the next few days could ease the installation at LLO, after we have sorted out everything at LHO.

It also includes some tidbits about the development that's been done, since several members of the hardware injection subgroup wanted to be kept in the loop.

Installations

There are only a few things we need on the guardian script machines that were not there are already. Things we have done:
 (1) Updated userapps SVN at /opt/rtcds/userapps/release/cal/common/guardian
 (2) Checked that we can instantiated guardian node with: guardctrl create INJ
 (3) Installed awg on the guardian script machine

Things we have yet to install on the guardian machine:
 * glue
 * ligo-gracedb
 * grid credentials

Codebase Development

This afternoon was mostly spent implementing several new things in the codebase. I have attached a new graph of the node to this aLog since there a number of new states, eg. new failure states, renamed the active injection states (formerly called CBC, BURST, etc.), renamed the IDLE state, and the renamed GraceDB state. And as always, the code lives in the SVN here: https://redoubt.ligo-wa.caltech.edu/svn/cds_user_apps/trunk/cal/common/guardian/

Some changes of note:
 * Changed jump transitions in successful injection pathway to edges. This changes how the node should be run. The model now is that the requested state should be INJECT_SUCCESS while running the node.
 * The modules (eg. inj_det, inj_types, etc.) have been moved to a new injtools subpackage.
 * Added success/failure messages for GraceDB events after injection is complete.
 * Added guardian decorators for a few tasks that are often repeated, eg. checking for external alerts.
 * Process managing for the awg call.

These changes have made the schedule validation script (https://redoubt.ligo-wa.caltech.edu/svn/cds_user_apps/trunk/cal/common/scripts/guardian_inj_schedule_validation.py) out-of-date, and that will need to be updated.

Also the GraceDB portions of the code have been commented out for now, since we're holding off on that for now until we have the grid credentials/how we will run the guardian process sorted out.

Tests

As I was developing I did a couple tests with the guardian node today. I did three hardware injections of constant amplitude (1e-26) into H1:CAL-PINJX_TRANSIENT_EXC. Each injection has a duration of 1 second. GPS start times are:
 * 1145497100
 * 1145497850
 * 1145500600

These tests were mostly to check that the call to awg was working properly.

The PINJX_HARDWARE filterbank has already been turned off (aLog 26748) so the signal will only appear in the PINJX_TRANSIENT and PINJX_HARDWARE channels.

In the attached plot below I shows the injection at 1145500600.

controls@h1guardian0:~$ cat /etc/apt/sources.list.d/lscsoft.list
deb http://software.ligo.org/lscsoft/debian wheezy contrib
deb-src http://software.ligo.org/lscsoft/debian wheezy contrib
controls@h1guardian0:~$ 

controls@h1guardian0:~$ grep gds /etc/guardian/local-env
. /ligo/apps/linux-x86_64/gds/etc/gds-user-env.sh
controls@h1guardian0:~$ 

PSL high frequency glitches, a little locking

Matt, Sheila

We locked and saw that we have 3 orders of mognitude or so to much noise at DC readout.  It seems this is related to PSL problems, the atached screenshot shows coherence between the ISS second loop PDs and DARM up to 100 Hz, as well as coherence between frequency noise and DARM.  The pink trace shows the coherence between intensity and frequency noise.  With this noise, we were able to power up to 10 Watts, but saturated the DC PDs.  

We looked a little but at impementing a dither loop for SRM control.  The alingment dither system currently would allow us to demodulate POP18, but POP90 has much better signal, as you would expect.  The IPC for adding this is already in the models, so I've just added it to the matrix but didn't do the model restart yet (WP5840).  

We then gave up on locking and went to the PSL racks to look at some signals at high frequencies.  We saw a glitch that happens at a repetition rate of 37 kHz, and has frequency content of nearly a MHz, which shows up in the laser intensity noise.  Matt has a picture of this in the ISS PD and the ref cav transmission.  If we turn off the ISS this is still there (as expected since its above the bandwidth), but it is harder to see on top of low frequency intensity noise.  

When the ISS is off, the ISS PDs wander from rail to rail, and oscillate only when they are near the upper rail.  

Turned off INJ-PCALX_HARDWARE filterbank

Chris B., Matt E.

Turned off the INJ-PCALX_HARDWARE filterbank output at 16:07 UTC.

From Today (April 23) to April 27, I plan to test the guardian hardware injection node. Turning off the output of INJ-PCALX_HARDWARE means that no signal will go into the detector.

OPS Eve Shift Summary

TITLE: 04/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: After battling ISS oscillations and rotation stage issues all night, I was unable to get past INPUT_ALIGN in initial alignment.  I think the IFO is tired of getting beat on this week.
LOG: See previous 2 aLogs from Sheila for details of most of the issues.
 

Locking today

Today, many of us worked to relock H1.  We've been able to get to DC readout, and had trouble powering up because of rotation stage problems we think we've solved, and as people have noted, there is something wrong with the ISS.  

Here is a list of things that were small problems:

• Wrong settings on ETMX SUS model left over from a model restart. 
• Alingment onto AS air diodes needed to be realinged
• OMs were not damped, although the guardian indicated that they were alinged.  
• We had one lockloss trying to engage ASC (part2), but that has not happened since
• We had high winds all afternoon and are now using BRS for tilt corrected sensor correction at both end stations.
• We also had a lockloss when the triple coil drivers switch, so I've moved this to the coil drivers state (later in the sequence, which doesn't solve the problem but means we won't have to worry about it for now)
• OMC PZT HV was off, also the PZT driver chassis was powered off with the rocker switch which took us a while to figure out. 
• Rotation stage issues: 26741 and comments

We spent some time trying to diagnose what was going on with the intensity noise of the laser. 

a remaining rotation stage bug

Craig, Sheila, Jenne, Travis

There is still at least one problem with the rotation stage, that sometimes it goes the wrong way when it first starts moving. 

The attached screenshot shows our first attempt at increasing the power since the vent, the rotation stage velocity and request are correctly set to 22 Watts and a velocity of 10, before the rotation stage starts moving.  When the rotation stage starts moving, you can see a drop in the input power (which happens faster than the normal rotation stage motion), the normalization (PSL-POWER_SCALE_OFFSET) is lowered by the guardian to adjust for this, but we loose lock anyway.  After the lock is lost the rotation stage proceeds at a reasonable pace to 20Watts, and the normalization follows correctly. 

This is not a new bug, we've had random locklosses like this for some time, but it is a bug that remains after last week's rotation stage fix. 

Manually over-filled CP3 at 23:01 utc

1/2 open LLCV bypass valve, and the exhaust bypass valve fully open. Flow was noted after 53 seconds, closed LLCV valve, and 3 minutes later the exhaust bypass valve was closed

changes in IMC alignment from April 4th to April 22:

Changes as measured by OSEM values: last IMC lock before laser work to stable IMC lock today

	April 4th	April 22	diff (urad)
MC1 p	-27	-13.5	+13.5
MC1 y	-1044	-1036	+8
MC2 p	500	504	+4
MC2 y	-688	-686	+2
MC3 p	-895	-895	0
MC3 y	-1027	-1018	+9
IM4 Trans p	-0.425	-0.490	-0.065
IM4 Trans y	0.103	0.081	-0.022

I worked on the IMs (IM1-4) this morning, and have verified that they are aligned to the April 4th values, so cannot account for the change in IM4 Trans.

The alignments of MC1-3 follow the alignment of the IMC input beam, so while the alignment changes for these optics are bigger than what we typically see, after the laser work they are not unexpected.

Power stabilisation work

This morning I measured the AOM drive voltage as a function of "offset slider %", and also the amount
of diffracted light both with and without the digital support loop.  All measurements were taken with
the HEPA fans running.

    With no diffracted light, the maximum output voltage on PDA was (-9.6,-9.3) V and on PDB was
(-9.65,-9.4) V.  No saturations were observed with the incident power at this level.

    Attached are the spectra and transfer function measured.  The relative power noise is about a
factor of 10 higher than it should be.  In the spectrum is a peak at ~1 kHz, the source of which
is unknown, but is probably related to the problems mentioned by Keita.  Pushing the gain higher
seems to increase the noise beyond 10 kHz.  The noise at frequencies below 100 Hz is significantly
higher than with the low power mode.

    At low frequencies the transfer function is not as clean as the measured result in the low power
mode.  I am not sure but this might be related to the linearisation of the AOM drive.

Attached are the plots for the AOM drive voltage and diffracted power as a function of
the offset slider.

    With the current settings with the offset slider at 6%, we are diffracting ~4.5 W.
The plotted and reported diffraction percentage on the MEDM screen needs to be recalibrated.

ISS issue (Sheila, Kiwamu, Jenne, Jim, Keita)

During relocking effort, for no apparent reason ISS went crazy several times (attached left), oscillating at about 1.6kHz (attached right). This is not unlike 2.5kHz oscillation reported in   alog 26666, but the frequency is different.

This is not FSS though FSS is affected by this, as the refcav was once knocked out of lock during the oscillation but the oscillation in ISS continued. It seems that FSS is affected because there's simply too much intensity noise (attached right top cyan and right bottom shows the refcav transmission DC).

HPO output itself seems to be fine (right top, brown trace), so this really sounds like something that happens in ISS.

Anyway, when this happens, do the following.

1. Open PSL_ISS MEDM.
2. Press AUTOLOCK OFF.
3. Press Manual Mode. Manual mode MEDM screen will open.
4. Press red OFF button in Manual mode MEDM.
5. If Refcav is unlocked, open the PSL_FSS MEDM, press AUTOLOCK OFF, then AUTOLOCK ON, and wait for the FSS to acquire lock.
6. Go back to Manual mode MEDM screen and press green ON button.
7. At first the red average trace in "DIFFRACTED POWER in %" on the ISS screen will be off the screen. Wait until the red trace becomes visible.
8. Press green ON+INT button.

Optical Lever 7 Day Trends

With the main laser down, it looks like some work effecting the oplevs must have happend over the last week, because some signals sit at 0 and then recover to typical levels.

I zoomed in on the Y axis to the sections of data that have reasonable signals.

Attached:

• oplev SUM
• oplev PIT
• oplev YAW