Today I looked more closely at some of the times last week when the SR3 pointing changed unexpectedly.
Zooming way out it looks like these dips only happen during business hours, and not on weekends so I was suspcious of loud activities in the LVEA. The hops aren't associated with any particular part of lock acquisition - they are visible in the OSEMs before we started locking. I went stomping around HAM5, tapped the chamber and poked the HEPI beams but couldn't reproduce the jump in alignment we've been seeing.
I looked at SRM and the OFI OSEMs and found that they also see these alignment shifts (top and third row of attached plot), making the ISI or HEPI look very suspicious. I added the HAM5 ISI GS13s to my plot (see attachment) and it looks like they get a big kick just before the optic OSEMs record an alignment shift.
We should look into this more closely. We haven't tried locking since Friday but I don't have a reason to believe this problem has solved itself. Today the suspensions were in SAFE and the HEPI offline for HAM7 activities so I don't see any hopping but we'll see if the problem returns.
It definitely looks like activity in the HAM5 area could be causing CPS "hops". Attached plot shows the actuator outputs for a window on March 4th, the "dc" output for the actuators shows several jumps which are not normal. These probably wouldn't show up as anything other than glitches on the other sensors, but indicate that the measured location has shifted. We probably aren't moving mass on the table in vacuum. This very much reminds me off either master clock issues or grounding issues I've seen in the past.
It looks like the corner 3 CPS satellite chassis may have been the culprit. When I looked at CPS spectra from overnight, I saw that the H3 and V3 sensors had peculiar high frquency noise above 100hz. Looking over trends, the drive hops have been happening since roughly Feb 18th. On Feb 22nd, I had spent a fair amount of time diagnosing some other, similar, issues with CPS on this chamber (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=61835) and I also reported H3 and V3 sensors jumping values then, but I don't remember their spectra having odd high freq stuff then. Attached spectra are before and after Fil power cycled the interface chassis in the HAM5 CER rack. First plot shows the high freq bump in the H3 & V3 sensors. V2 is also a bit higher, but not a concern yet. Second plot shows the corner 3 noise has gone away. I've been watching the asds on an expo plot and the noise hasn't returned.
SR3 hopped again last night at 20:06PT, and at 07.52, 8:28amPT this morning. Last night's corresponds with movement in the LVEA (seen in PEM accelerometer data at HAM4) and a glitch in HAM5. See attached.
