Reports until 16:40, Sunday 11 January 2015
H1 ISC (ISC, SUS)
jeffrey.kissel@LIGO.ORG - posted 16:40, Sunday 11 January 2015 - last comment - 10:53, Monday 12 January 2015(16005)
H1 IMC WFS Failing to regain spots after mass safe.snap capture
J. Kissel

Something went wrong with the IMC after my safe.snap captures. It seems like the MC WFS DC signal indicates that they've lost there spots, so the ASCIMC servos are continually steering the IMC slowly off resonance. I attach two trends. The first is the past 6 hours, showing that when I started taking down suspensions, the MC WFSs lost there spots. The second is the last hour, showing how the IMC is in a bad locking loop, where it acquires, and then slowly tanks.

I've burtrestore'd all relevant epics IOCs to 02:10p PDT (before I came in) I could think of, i.e.

burtrestore h1susmc1epics 14:10
burtrestore h1susmc2epics 14:10
burtrestore h1susmc3epics 14:10
burtrestore h1ascepics 14:10
burtrestore h1ascimcepics 14:10
burtrestore h1sushttsepics 14:10

But things are still stuck in the loop. 

I'm going to continue to try to debug, but I solicit any remote help, if anyone's out there reading...
Will post if I find a/the solution.
Images attached to this report
Comments related to this report
jeffrey.kissel@LIGO.ORG - 19:41, Sunday 11 January 2015 (16009)
J. Kissel, w/ a remote A. Staley & E. Hall

Though we're still not sure why the spots have gotten mis-centered on the IMC WFSs, I was able to offload the previously functional IMC WFS requested values to the OPTICALIGN alignment offsets in MC1, MC2, and MC3, and this extended the cycle by a few more minutes. This still did not get light on MC WFS B so the ASCIMC loops still eventually pull the alignment of into la-la land. Thinking of what happened this past friday, Alexa suggests just to leave the IMC WFS OFF, and I have done so. The IMC has been locked rock solid since. 

A couple of debugging notes, but still no answer:

Also -- Alexa informed me that the power 50% power drops were because the DRMI guardian had not been set to the DOWN state, so once the IMC locked, it would downgrade the input power from 10 to 5 [W]. This was a red herring, which I thought was more MC WFS mysteriousness that I couldn't figure out at first.

Here's a comparison between previously functional alignment offsets (with MC WFSs engaged, so modified from the static H1:IMC-${OPTIC}_${DOF}_OFFSET), and those that have had the previously functional MC WFS DC values (as read by the output M1 LOCK banks) offloaded to them:

                       H1:IMC-${OPTIC}_${DOF}_OFFSET            H1:SUS-${OPTIC}_M1_LOCK_${DOF}_OUTMON        H1:SUS-${OPTIC}_M1_OPTICALIGN_${DOF}_OFFSET
                                P             Y                          P               Y                            P                Y
MC1 this morning               +0.7         +0.2                       -18.1           +16.8                        +1024.2         +2100.0
MC1 with WFS offload           +0.7         +0.2                        +0.7            +0.2                        +1006.1         -2083.2

MC2 this morning              -21.5        +27.6                       +18.4            +1.8                          515.8           -556.7
MC2 with WFS offload          -21.5        +27.6                       -21.5           +27.6                          534.2           -554.9

MC3 this morning               +5.8        +18.0                       -13.0            +1.5                         -527.3          -2100.0
MC3 with WFS offload           +5.8        +18.0                        +5.8           +18.0                         -540.3          -2198.5

I did NOT save these new alignment values to the SUS' ALIGNED file, it was just a stab in the dark.

I've taken a whole bunch more trends, which indicate that the alignment offsets were identical to before I got started, until I changed them as indicated above. However -- zooming in on when, and in what order I turned ON and OFF the suspensions vs. relocking the IMC, I find there may be a pattern / problem in the order and timing in which I took down the three MCs. After staring at all five plots at once (I know, it's hard to do without a lot of screen real estate, but maybe open in 5 different tabs and flip between), I recall that I turned OFF MC1 first, *without* pausing the IMC guardian or requesting it to be DOWN. Before I moved on to the other suspensions, I restored MC1 to confirm that the IMC came back. As such, in the interim, I think the IMC WFS began to steer the IMC in a bad direction with the misaligned (i.e. OFF) MC1, pulling the integrators to mis-center the spots -- especially on MC WFS B. Further, I didn't have nearly as many IMC screens open as I did after trying to diagnose the problem, so I didn't see that the MC WFS were being pulled off course.

I thought, that if this is true, it should be as simple as restoring the Sunday "morning" offsets, and clearing the history on the MC WFS integrators, and the spots should immediately become re-well centered on the MC WFS again. BUT, that didn't work either. With the "morning" offsets on the MCs, and history cleared on the ASC IMC loops, the spots reappear on the MEDM cross-hairs, but just barely. Which means the TRANS power began to tank again with full gain MC WFS on.

Now I think, we should use the picomotors to recenter the MC WFS. But, I have zero experience with the pico-motor game, and I have learned to harbor great fear of them, especially after Suresh informs me that they're somehow wired such that a YAW request moves the beam in PITCH. Do we not have a DC centering servo on the MC WFS?

For now, I leave the MC WFS OFF (via the gain slider in the bottom left corner), and I've cleared the history of the ASC IMC DOF loop filter banks.
Images attached to this comment
jeffrey.kissel@LIGO.ORG - 21:18, Sunday 11 January 2015 (16011)
J. Kissel, with Suresh (remotely) this time

For starters -- thanks for all your help remote commissioners!

So I think Suresh might have nailed down the problem:
- Last week, Thursday afternoon, Jan 8th, Suresh commissions the MC WFS DC Centering servos, a.k.a. DOF 4 servos, a.k.a. the MC1 / MC3 differential pitch and common yaw servos (see LHO aLOG 15865). First with a gain of -0.1, and slightly later with a gain of -1.0. In doing so, he changes the alignment sliders of the IMC suspensions, but does not *save* the new alignments to the aligned.snap text files.
- The next morning, when the IMC guardian was changed from LOCKED to DOWN, the old, now invalid, alignment values returned (see LHO aLOG 15968). When the new centering servos started pulling the spots off the MC WFS from the bad alignment, the DOF 4 gains were changed to zero. "Turn it off! Turn it off!"
- When the alignment values were restored, the DOF4 centering loops were left off, because they didn't instantly work as designed, and there were bigger fish to fry. We now know it's because the MC WFS are so miscentered that even with a good IMC alignment for starters, then pull things off the quadrants.
- These new alignment values remain stored, and have new even been captured in a safe.snap.
- That the IMC can lock by itself, and stay locked robustly, without guardian or WFS implies that this indeed a good alignment on the rest of the REFL path (i.e. on the trigger PD and the IMC REFL Length diode).
- So, we should re-center the spots on MC WFS using the pico-motors by hand, as I mention above in a previous comment. 

I didn't yet enact on the solution, because I wanted to do a few other things tonight (and because of my previously mentioned superstitious fear of picomotors), but I write it down for those to attack whenever they can.

Images attached to this comment
sheila.dwyer@LIGO.ORG - 10:53, Monday 12 January 2015 (16022)

Evan and I recentered the IMC wfs, they are working now.