K. Izumi, J. Kissel, P. Thomas, C. Vorvick

The H1 IMC is back to stable, many-minute lock stretches with good alignment, with all LSC and ASCIMC loops running as designed (using all three stages of MC2 for length control), and the HAM2 and HAM3 isolation systems performing at their best thus far (HPIs floating with alignment offsets, ISIs at Level 3, and SUS with old Level 1.5 damping loops). It was a long battle because many things were wrong with ISC/IOO side of things, but Kiwamu and I think we've managed to put the whole picture together, which is outlined below. Note, that this covers some of the material found in Cheryl's earlier entry, so see LHO aLOG 7229 when referenced. The PSL is still in commissioning mode -- we know that's what's causing the high-frequency intensity noise (as seen on the MC REFL camera), and may also be the cause of the occasional lock losses. The simple (non-guardian) MClockwatch auto-locker is currently running on opsws5.

-----
The Problems
-----
(Here's what we've fixed:)
- The power outtage from this past Thursday (LHO aLOG 7646) look down all front ends, but -- most important to this story -- the imcasc frontend. This frontend controls the Periscope PZTs that steer the IFO beam from the PSL into the IMC. These PZTs are known to have terrible hysteresis, such that even if you burt restore their alignment settings to a known good time, you're not guaranteed that the same alignment returns. This is what happened here.
- Slowly, over the course of days, the alignment of the mode cleaner appeared to decay. This was because the MCWFS loops (in particualr DOF 3 which uses those same PZTs as actuators to control the input beam pointing) were slowly steering the input beam to a bad place. Eventually, the PZTs had been steered so much -- and the mode cleaner's integrators followed enough -- that light fell off MCWFSA and B. This meant that any attempts WFS control would fail. Also, or perhaps more probably, the MC WFS alignment offsets had been offloaded a few times, but offloaded with worse and worse alignments.
- Until this afternoon (as Cheryl mentions in 7229), we had forgotten to restore the H1 LSC front end epics. The LSC front end contains not only the H1:IMC-L filter bank and the H1:LSC-MC filter bank, but also the LSC-MC_FM_TRIG_THRESH channels which define the point at which to turn off a 3[Hz] low pass in FM10 of the H1:LSC-MC filter bank. These thresholds *had* been set to zero, so this extra low-pass filter (designed to let the IMC swing through a few fringes/FSRs before locking, so it doesn't lock on a non-00 mode) remained in the length path, distorting the frequency-dependence of the loop gain, and therefore causing the loop to go unstable.
- (Our best guess), In one of the MC WFS scripts that was mistakenly run (see 7229), the ASC IMC loops were turned off in a hidden place -- the input to actuator output filters (e.g. H1:IMC-MC1_PIT_INPUT, H1:IMC-MC3_YAW_INPUT, etc) which are not shown on the overview screen. We created a successful work-around for a while, but discovered this at the very last minute (see solutions story below).
- The screens for the MC WFS output matrices were full of copy-and-paste errors, such that both PITCH and YAW screens were labeled as "PITCH," and even some calls to matrix elements from PITCH were found in the YAW matrix.

(The following is still wrong/broken/incorrect, but doesn't seem to matter:)
- The PSL is in commissioning mode, with HEPA fans still blowing around and room lights on inside the enclosure. Not Tragic.
- The PZTs that control the beam-steering onto MCWFSA and MCWFS B are non-functional. We did all of the  WFS centering by going into IOT2 and manually tweaking the knobs of the mirrors. Kiwamu says that Shiela was in the middle of debugging this, which we (Kiwamu and I) think is a problem with the Beckhoff computer. Thankfully we can still operate the knobs manually.
- The MC2 TRANS QPD's analog whitening filters are somehow busted -- when engaged the TRANS signal goes negative. Yuck. Strangely enough, the only stable configuration we've found with all analog filters OFF, but ONE of the digital compensation filters ON, as though the IMC WFS DOF3 loop was designed with this configuration. Double Yuck.

(The following definitely doesn't matter, but are things I noticed along the way that NEED to be fixed to improve the interoperability SEI/SUS systems)
- The SEI/SUS IOP Watchdog blocks the output to HEPI, but DOES NOT trip it's internal watchdog system. Hence, when you reset the IOP watchdog all output remains (which is currently only actuator basis alignment offsets; isolation loops are not-yet-commissioned), and you send a jolt to the chamber. Thankfully, the alignment offsets were small this time. 
- The HAM-ISI checker script is still not yet functional. So in order to reset the HAM-ISI watchdogs, you MUST use the "turn off isolation loops for HAM(2/3)" script to ramp down the other-wise-huge output of the isolation filters. If not, the watchdog will just happily keep cycling through the states, laughing at you, and calling you silly for trying to reset the watchdog when the output request is 532 billion.
- I'm not sure how the ISI's blinky performance matrix was copied the to IMC OVERVIEW screen, but its matrix elements' color schemes DO NOT match up what's in the top middle of the HAM-ISI OVERVIEW SCREEN. What's worse (who cares about the color scheme), is that the matrix elements BLINK DIFFERENTLY. What appears to be a completely happy, stable, meeting-or-beating-requirements matrix on the ISI screen, looks like a constantly fluctuating ISI (flashing BLUES and PURPLES) on the IMC OVERVIEW screen. I've checked, and the matrices are looking at the exact same channels, but it seems as through whomever copied them over to the IMC screen CHANGED THE THRESHOLDS on the MEDM screen or something. I haven't looked at in any more detail.

----------
The Solutions
----------
(Note, during entire problem solving stage, MClockwatch was running in the background, grabbing the length lock whenever it could.)
- Now stuck with a new alignment of the input beam from the PSL because of the hysteric (intentional typo) periscope PZTs, Cheryl, re-aligned the MC1, MC2, and MC3 suspensions to maximize MC_REFL (as mentioned in 7229). The new alignments are

      P      Y  [urad]
MC1  +314   -984 
MC2  +428   +251
MC3  +237  -1079

- Kiwamu found the bug with the with MC TRIGGER thresholds, and restored them to a reasonable value:
H1:LSC-MC_FM_TRIG_THRESH_ON  = 1000
H1:LSC-MC_FM_TRIG_THRESH_OFF =  900
- Kiwamu also fixed the MC WFS output matrix MEDM screen bugs.
- Kiwamu and I manually re-centered the MC WFS to at least get the beam back onto their PDs. 
- Kiwamu started up the MC WFS loops' DOF 1 and DOF 2 (which use the MC Suspensions to align the IMC waist). Now with light on the MC WFS, the loops were completely stable as normal.
- Kiwamu tweaked the input beam alignment by (digitally) adjusting the PSL Periscope PZTs, using the P & Y signals on the MC2 TRANS QPD as the figure of merit. He made sure to do this slowly enough that the MCWFS DOFs 1&2 could follow and pull along. 
- Kiwamu tried to turn on DOF 3, but found it pulling the IMC out of alignment, eventually to break its lock. A discussion of the output matrix led him to zero out the DOF 3 contributions to MC Suspensions, and leave it only actuated by the PSL Periscope PZTs. In this configuration, with all three DOFs closed, we were able to get nice long lock stretches.
- Kiwamu offloaded, or "relieved" the DC offsets in the MCWFS  loops' error points into offset field of the MCWFS DOF actuator output filters (e.g. H1:IMC-MC1_PIT_OFFSET, H1:IMC-PZT_YAW_OFFSET, etc.), using 
${userapps}/ioo/h1/scripts/imc/mcwfsrelieve
(At this point, the mode cleaner is at the ideal locking alignment)
- Kiwamu went out and re-centered the MC WFS to this newly-ideal alignment (with all IMC L and ASC loops running), but while out there, he noticed the input to both MCWFS PZT actuator output filters (H1:IMC-PZT_PIT_INPUT, H1:IMC-PZT_YAW_INPUT) were OFF. Our best guess is that one of the defunct scripts that was run earlier turned them off. This explained why DOF 3 was going unstable with the nominal output matrices -- the DOF 3 was trying to actuate on the PSL Periscope PZTs, but was only getting action from the MC SUS's.
- Finally, Kiwamu turned ON these inputs, and restored the output matrices, and declared victory. 


*phew* Can anyone say "Configuration Control Nightmare?" I sure can.

Cheryl V., Jeff K., Kiwamu I., Patrick T. working on MC locking (alog 7729)
ARK staging roofing materials (WP 4126)
Apollo moving clean room in the LVEA (WP 4125)
Continuous communication problems with dust monitor 15 in the LVEA
One of Dale's students filing sharp edges off piece of old beam tube near the warehouse

09:00 Corey G. going to take pictures in the squeezer bay
09:17 Arnaud P. running transfer functions on TMSX
09:24 Andres & Jeff B. going to work on ITMX
09:47 Dave B. rebooted the DAQ to add GPS coordinates to the frame files
10:34 Andres & Jeff B. going to work on ITMX
12:37 Andres & Jeff B. going to work on ETMX
13:27 Thomas V. going to work on assembly baffle
14:50 Andres & Jeff B. back from end X
15:24 Arnaud P. going to LVEA to take hammer test measurements on cage of ITMX

  Andres & Jeff

  We installed the vibration absorbers, upper structure cross braces, and wafer holder on ETM-X. This completes the payloading for this suspension.   

   Andres & Jeff

   We installed the vibration absorbers on ITM-X. This completes the payloading for this suspension. 

This morning Arnau ran another set of Transfer Functions for the H1 TMS EX, and after the measurements he gave the "GO AHEAD" that the system looked good to go.  So, after SUS crew did work installing Vibation Absorbers, I went about staging ALL (6) TMS cables in coils on the ISI's Stage0.  They now await Jim to dress them on the ISI.

Notes

• Yesterday during cable testing, we discovered that the Green QPD's cable connector which connects to the BSC9 feedthru is missing set screws (so it can't be screwed into the feedthru).  We'll probably try to fix this in-situ.  This cable is D1000924 s/n S1104104 (as noted in table, here).
• Size #18 caddis fly (i.e. a flying bug!) was observed on the ETMx quad (on cyllinder of penultimate mass).  Jeff tried shooshing it away, but it ended up flying upstairs and we lost track of it.

Attached are plots of dust counts requested from 4 PM September 2 to 4 PM September 9. Each plot is 1 day.

- Patrick, Cheryl, Jeff, Kiwamu 

Morning spent with Patrick and I investigating IMC poor locking, i.e. only brief locks.  Keita started us on the issues of too much gain on H1:SUS-MC2_M2_LOCK_L_GAIN, which he changed form 0.2 to 0.008, in order to get any locking last night. 

Looked at Stefan's autolock script and the channels it changes, and all was OK. Compared MC Common Mode channels/settings, all was OK. 

Tracing the GAIN path backward, I found we could lock with autolocker settings and MC2_M2_LOCK_L_GAIN at 0.2, if I lowered the overall MC2 LSC gain from 1.0 to 0.15. 

Jeff joined us, we burt restored h1lscepics and MAGICALLY the IMC locked with the MC2 LSC gain back at 1.0!  Somehow that fixed the gain issue, though we have not located the exact setting that changed.

At this point Jeff reenabled HAM2 ISI and set it to Level 3 isolation. This helped with 1Hz oscillation seen in REFL. Then we set HAM3 isolation to Level 3 from Level 2, and further decreased the 1Hz noise. This allowed for a long lock and I was able to align manually to get REFL down to 240 and MC TRANS around 2200.

We tried to engage WFS, but found it driving the lock to a bad place, REFL = 800. Kiwamu arrived, and is now working on WFS.

WFS are not well aligned, and MC2 TRANS QPD is high in pitch.  This means the IMC alignment is good for REFL, but not for the IMC as a whole, which could be my alignment changes to MC1, or changes in the PSL input beam pointing from PZT's being turned off and on yesterday (see NOTE), or changes to the PSL input beam pointing from temperature change made by Rick in the PSL enclosure, or all 3 combined.

NOTE... I ran scripts attached to the WFS medm screen yesterday, which apparently have NEVER been run here before, and were written at LLO.   Why do we do this to ourselves?  I was telling Jeff the script to turn off ASC turns off the PZT offsets WHICH SHOULD NEVER BE TURNED OFF!!!  Now I learn from Kiwamu that no one has ever run any of those scripts at LHO.  Are these scripts even used at LLO?  This sort of thing is a death trap for an IFO.

Currently Kiwamu is working of WFS.

The staging building X1 triple test-stand has been configured ready to accept OMC (OMCS) suspensions for Phase 1b testing. This procedure is similar to that which I had previously carried out on the LLO triple test-stand (see LLO aLog entry 6842).

I've remotely logged-on and set-up the test-stand as follows:-

1) svn'd up the /ligo3/svncommon/SusSVN/sus/trunk/OMCS$ directory

2) svn'd up the /ligo3/svncommon/SusSVN/sus/trunk/Common$ directory

3) Started the x1sushxts05 model, thanks to the local efforts of Jim Batch (see LHO aLOG entry 7727).

4) Burt restored the most recent working triple suspension configuration from /opt/rtcds3/tst/x1/userapps/release/sus/x1/burtfiles/20120911_x1sushxts05_PRM.snap

5) Reset all stage OSEM INPUT FILTER gains and offsets to their default values, +1 , and -15000 counts, respectively.

6) Used updated ligo3/svncommon/SusSVN/sus/trunk/OMCS/Common/MatlabTools/make_susomcs_projections.m Matlab script to populate OSEM2EUL and EUL2OSEM matrices with elements for OMCS geometry, using the "fill_matrix_values" script, as follows:-
- fill_matrix_values('X1:SUS-HXTS_M1_OSEM2EUL',OSEM2EUL.M1)
- fill_matrix_values('X1:SUS-HXTS_M1_EUL2OSEM',EUL2OSEM.M1)

7) Copied M1 DAMP filter block and gains from LLO triple test-stand.

8) Taken a new safe BURT snapshot, which has been committed to the svn, /opt/rtcds3/tst/x1/userapps/release/sus/x1/burtfiles/20130911_x1sushxts05_OMC.snap

Finally, to expedite testing, I've copied over some of the most relevant DTT OMCS templates from X2, in which the excitation amplitudes have already been tuned to optimize coherence in all 6 DOFs. These DTT templates (and accompanying exported ASCII data) are available on the svn as follows:-

X1/OMC/SAGM1/Data/2013-09-11_1200_X1SUSOMC_WhiteNoise_L_0p1to50Hz.xml
X1/OMC/SAGM1/Data/2013-09-11_1200_X1SUSOMC_WhiteNoise_T_0p1to50Hz.xml
X1/OMC/SAGM1/Data/2013-09-11_1200_X1SUSOMC_WhiteNoise_V_0p1to50Hz.xml
X1/OMC/SAGM1/Data/2013-09-11_1200_X1SUSOMC_WhiteNoise_R_0p1to50Hz.xml
X1/OMC/SAGM1/Data/2013-09-11_1200_X1SUSOMC_WhiteNoise_P_0p1to50Hz.xml
X1/OMC/SAGM1/Data/2013-09-11_1200_X1SUSOMC_WhiteNoise_Y_0p1to50Hz.xml

This should now provide sufficient infrastructure to proceed with Phase 1b testing of the OMC suspensions.

I have the x1sushxts05 model running now. 

In X1SITEMAP_TRIPLES.adl, the macro substitution used for the 05 models used FEC=17 instead of FEC=18.  I changed the macro to use FEC=18 for both HLTS05 and HSTS05, which allowed me to open a working X1SUSHXTS05_GDS_TP medm screen.  This in turn allowed me to press the BURT button after the X1 IOC started, which allowed the model to continue to run.

It's been broken for quite some time, so I don't know why the wrong FEC number was in the X1SITEMAP_TRIPLES.adl file.

I note that there appears to be an excitation test point 119 that shows up on the GDS_TP screen.  When I run diag, it indicates no test points are set, and no excitations are running.  Not sure where it's coming from, but if it bothers you I can try harder to get rid of it.  It'll probably involve rebooting stuff.

I restarted the daq. It was configured to read the Beckhoff GPS EPICS channels from the timing master with a new ini file (H1EDCU_GPS.ini)

Cheryl reported that IMC wouldn't lock and WFS was breaking the lock.

When I came into the control room the DC alignment of MC seemed OK. Disabled ASC output and manually enabled the PZT offset.

MC2 M3 watchdogs were found tripped, and assuming that this stage is necessary for locking I reset the watchdog.

After this IMC would lock but it lasted for 3 seconds that's incidentally the same as the ramp time for M2 and M3 lock filter. The lock was repeatedly broken by an oscillation at a few Hz that develops right after M2 and M3 lock filters were turned on.

I disabled the L-L element of the drive align matrix of M2 and M1, and the lock would last until the M3 coils rail.

Apparently something changed between this morning and afternoon, and now I had to make the M2 and M1 gain ridiculously small, otherwise a few Hz oscillation just kills the lock.

For the moment I changed the setting in Stefan's lock script such that H1:SUS-MC2_M2_LOCK_L_GAIN is 0.008 (VS the nominal 0.2) and M1_LOCK_L_GAIN is 0.3 (VS the nominal 1). I didn't fix anything, whatever the problem is it's still there, and the lock lasts for only a few minutes at most until M3 coils rail.

I haven't done anything to WFS, I don't know if it was doing something bad, but the DC alignment right now is OK.

One of the BIO bit representing the status of the analog dewhitening filter of the TMSX is either not displaying the right value, or the dewhitening filter is not switched to the requested state, cf bit 8 (RT LP) of picture attached. I will check tomorrow with Richard to see where the problem could come from.

J. Kissel

Here're some spectra that are the same as what Vincent had posted, but focused on the LF and RT OSEMs, and calibrated into [um/rtHz]. I show four configurations (each config has the ISI/HPI in its best performance state):
(1) As we discovered the problem, with all SUS damping loops closed (shows giant feature)
(2) With only V loop turned off (still shows giant feature)
(3) With V and R loops turned off (does NOT show giant feature)
(4) With V and R loops turned off, and with a broad-band 1000 [ct] excitation sent out of the RT COILOUTF bank (does NOT show giant feature).

Note, I also measured 
(5) With only R loop turned off (still showed giant feature), 
(6) With V and R loops turned off, and with a broad-band 1000 [ct] excitation sent out of LF COILOUTF bank (still did NOT show giant feature), 
(6) With V and R loops turned off, and with a broad-band 100 [ct] excitation sent out of LF COILOUTF bank (still did NOT show giant feature), 
(6) With V and R loops turned off, and with a broad-band 10 [ct] excitation sent out of LF COILOUTF bank (still did NOT show giant feature), 
but they were redundant with the above information, and would have over-crowded an already crowded plot.

First page of the attachment shows configurations (1)-(4). Second page shows the ASD of the 1000 [ct] excitation, meant to look roughly like a damping loop signal. 

Given that we only see this feature with the damping loops that involve LF and RT ON, and we don't see it by turning the damping loops OFF and driving an artificial, non-OSEM-sensor, signal AND we know it's not a loop instability, I think we can narrow down the source to something in the analog sensing chain that is oscillating.

We should replace the M0LFRT, R0LFRT satamp tomorrow, and see if this feature goes away. I blame the satamp, because the satamps have plagued use wih a terrible history of oscilliations.

I bet $1.00 USD.

If anyone is betting against, please please please, shoot me more ideas on how to narrow down the source further.

J. Kissel, V. Lhuillier

With all SUS damping loops closed and with the ISI in full isolation, I've taken open loop gain transfer functions of the H1 SUS ITMY's Vertical and Roll Level 2.1 damping loops. As predicted by design, and by what was seen on H1 SUS ETMY there is plenty of phase margin, and no features around 6.18 [Hz].

NEXT: Determine whether it's the coils or the sensors, or some nasty interaction between them.

Sebastien, Jim

We are currently testing the BSC-ISI unit #6 in the staging building. The 2 stages are unlock and balanced, everything seems to be well connected.

The power spectra show something wrong on GS13-V1 and GS13-V3 (see figure attached).

After further investigation, the issue seems to be 2 bad cables. They will be change tomorrow.

In the optimal laser configuration (current = 1.503A, see alog7695), the power before ALS-FI2 was measured to be 39.7mW, while the power after the FI was measured to be 37.6mW. Thus, the throughput of the ALS-F12 was about 94.7%. The first jpg (532nmALSfI2FullCurrent.jpg) shows the beam profile after the ALS-FI2.

The first pdf (BeamProfile532nmRefALSFI2.pdf) is a graph displaying the mode measurement of the 532nm beam with ALS-F12 as a reference. Meanwhile, the second pdf (BeamProfile1064nmRefALSM1.pdf) is a graph displaying the mode measurement of the 1064nm beam with ALS-M1 as a reference. Using the 13.5% width (or 1/e^2 diameter) of both the horizontal and vertical profiles, the radii for the two profiles were determined at various distances from the respective reference points. In addition, the graph includes error bars of one standard deviation. A line of best fit was produced, and the difference between the line of best fit and the data points was plotted in the lower portion of both figures.

Phase 3b spectra have been measured on the beamsplitter after Jeff Kissel implemented the new filters.

The attached pdf are showing a comparison between spectra taken in March 26th phase 3a (in blue), and May 17th phase 3b (in green) with the suspension damped (first pdf) and undamped (second one). The ISI was damped.

Data, measurement lists and figures have been commited on the svn