Just got done talking to Terra and Keita. With Keita's approval (because it seems to be required to get the IFO running stably), Terra walked me through making a new bandpass filter for this mode:

cheby2("BandPass",4,40,237,240)butter("BandPass",2,237,240)notch(238.5,20,30)gain(31.6228)

This is installed as 18038p5Hz in FM5 on Mode 27 (Terra suspects this mode is also on ETMY). Turning it on created SDF diffs that I've accepted, and I hit the intent bit 20 seconds before 6:00 local.

Jim is awesome and mode seems to be holding stable.

This one was a bit confusing because we have two ETMY modes close to one another, one at 18038ish Hz and one at 18042ish, Hz as seen in Jim's spectum. This is probably the source of TJs problems last night (sorry TJ), though when I checked earlier today the peak I trended from his shift was within the BP so I didn't suspect that to be the problem. Jim scooted over the BP and PLL set frequency to be over the 18038 Hz mode. Next day I'm in, I will create a new mode so that the two can be differentiated. Note though that the 18042 Hz mode did not go unstable just now even though we weren't damping it. I'll look into the last several locks and see which seems to be the regularly problematic one. We also weren't driving quite hard enough; Jim increased the damping gain which worked much better.

For now: If Mode27 seems unresponsive, open PI DTT and check if the growing peak is at 18038 or 18042 Hz. Then check the following:

• Open Mode27 in the PI medm screen.
• Open the BandPass filter bank
• Change the mode27 Bandpass filter if necessary (filters for both options now exist) to cover the correct peak frequency 
• Open the PLL screen for that mode
• Set the PLL 'set frequency' to 238 (if the peak is at 18038) or 242 (if the peak is at 18042) 

Screenshot of PCal X line prior to being turned off.

I also had to accept the SDF diffs for turning this line off in order to go to Observing.  See screenshot.

Yeah, it's been one of *those* days...

HAM 2 needs a second look

Reran the HAM2 HAM3 CPS Sensor Spectra. New plot is posted below. Between 20 ans 60Hz, HAM2 looks a bit better. 

Attached is a zipped tar file of spectral comparisons for the two pump-on periods cited
and for an intervening pump-off period. Since I don't know what the pump frequency
is supposed to be, I'm not sure where to focus attention, but I'm not seeing gross differences
other than higher 60-Hz harmonics in both pump-on periods (is that expected from
the pumps?), along with the overall greater excitation / upconversion of violin modes and
the presence of  "silent" calibration lines at 22.3 and 30.1 Hz in the first pump-on period. 

Here are the FScanned data intervals used:

pumpon-1 -- 23.5 hours of SFTs collected between Nov24th, 0230 UTC and Nov 29th, 1800 UTC
pumpon-2 -- 11.5 hours between Dec 5th, 1900 UTC and Dec 7th, 0035 UTC
pumpoff -- 78 hours between Nov 30, 14:00 UTC and Dec 5th, 14:00 UTC

If there are particular frequencies where I should be expecting disturbances,
let me know and I'll do some zooming. 

Kyle informed me offline that the pumps run at 1720 RPM (28.667 Hz) 
with small turbos running at 90,000 RPM (1500 Hz). Attached are zooms
of bands around the first four harmonics 28.667 Hz (band sizes increase
linearly with harmonic number) and around 1500 Hz. I don't see anything
jumping out at me in any of the low-frequency bands that indicates an 
artifact restricted to pump-on times. 

The region around 1500 Hz is harder to interpret because of different average 
quad violin mode excitations and associated up-conversion.

Fig 1 - 28.5-28.8 Hz
Fig 2 - 57.0-57.6 Hz
Fig 3 - 85.5-86.4 Hz
Fig 4 - 114.0-115.2 Hz
Fig 5 - 1480-1520 Hz

Well I think things are stable now. I ended up flipping the sign for the both the phase and gain on mode 27. Mode 26 also came up and changing the phase got it to, very slowly, go down. I took some screen shots of the dtt template at 20min, 26min, and 32min into the lock. The modes that the Striptools said were ringing up, didnt necessarily agree with the dtt. I dont remember how much the modes can shift in frequency, but it seems like some of them were off by a couple of hz. 

A couple things:

• I'm not sure what you're referencing that seems 'a couple Hz off', could you elaborate? I checked the band pass filters against the frequency of the peaks at the times you listed and the most I found was Mode28 was off by 1 Hz. I have since changed the BP filter to account for this shift. 
• Remember that if a peak's frequency in DTT is different than the bandpass filter frequency, you should switch to a different band pass. There are explicit directions in the PI Wiki (which you can open from the PI medm screen and which I just updated to make more clear) and I've walked through this process individually with each operator. If something is still unclear, you can always still call me at anytime (number on the whiteboard). This shouldn't be much of an issue these days but is worth checking if you're having damping problems.
• Modes 27 and 29 have to be carefully watched during the first hour of lock. These have been successfully damped for weeks now but must be reacted to immediately. 
• Again, if you lose lock from a PI, you must wait in DC readout to give that mode time to ring down. You can watch the ringdown in DTT. If you try locking again, the mode will still be rung up.
• There are some modes that always have higher amp, such as the 13kHz modes TJ noted, that are not unstable - TJ that DTT from low noise you first posted is totally normal. You can always look at a reference from an eariler stable lock time to compare the amplitudes. I will add a good reference to the PI DTT in the background so it's easy to see what's 'good'
• Regarding DTT and StripTool not seeming to match - we use different error signals for different modes, while the DTT is just looking at one of those signals. The result is that Modes 27,28 look very rung up on the StripTool but their peaks in the DTT dont look so high. This is done on purpose, because those modes need to be damped faster. 

Please any operator contact me at any time if help/explanations/etc is needed.

For the couple hz off thing, I was mainly refering to the frequencies listed on the medm above the modes. I'm not sure how accurate these are suppose to be, but like mode 27 says 18041, and the dtt was saying that 18038 seemed to be more rung up (see second attached). I didn't remember how much they shifted, and I did not look at the actual filters themselves at the time.

I was able to make it work (on the second try) with just changing phase and gain so I didn't look into it too much after that. But, yes, following your instructions and changing the BP would have been next on the list.

In alog32385 you mention that you will put a reference trace to show what is normal and what is not. I would love this because I looked at the dtt picture that is on the wiki and saw that it was much different. Either way, it was a good learnign process for me since I have only ever had to change phases for PI modes.

Also, the take snapshot button for camera 17 just saves blank images.

Here are 2 other old alogs that are relevant if you are worried about scatter from CPs:

Robert saw scattering from CPY in the PR2 camera:  31243

I saw that 6 times higher drive was needed on CPX than CPY to make noise show up in DARM, and the noise that did show up was clear fringe wrapping shelves for CPX and a broad shelf for CPY.  30979

CPY scattering seemed like it is not an immediate problem since the drive reuiqred to see noise in DARM was 100 um at the error point of the damping loop at 0.1 Hz.  I don't know the gain of the damping loop, but assume that it is not less than -20 dB at 0.1 Hz, so that we are pushing the mass at least 10 um, probably more. This should be larger than the normal path length modulation.  It would be good to look at what the gain of the damping loop actually is to see if this is really a much larger path length modulation than what we would normally expect.  

SRC model shows that CPY is installed parallel to HR surface of ITMY (misalignment angle of 0.07 degrees). This number gives a vertical offset of the ghost beam on SR2 of 2 cm and on SRM of 10 cm. This is how I read the camera image. One might also suggest that the plate is misaligned even further from the nominal position — by 0.14 degrees. In this case ghost beams swap and we can’t tell a difference.

C. Cahillane

I have changed the kernel the Gaussian Process runs on in order to get more realistic errorbars.

The plots for the UIM stage have a dot product kernel: k(x,y) = 1.0 + x.y + noise
The plots for the PUM and TST stage have a squared dot product kernel: k(x,y) = 1.0 + x.y + (x.y)**2 + noise

The results show a much more simplistic and physically realistic correlation between measurement points. (No more overfitting wiggles, slightly expanded uncertainty bars, no instabilities in the GP results)

Our minimum uncertainty in the TST stage in the bucket is on the order of ~0.6% and 0.3 degrees.  So still very good uncertainty.

Still to do:
(1) Remove time-dependence from the measurements by modifying the .conf files with the relevant kappas.
(2) MCMC to find physical parameters on only the reference measurement.
(3) LLO Actuation, LHO and LLO Sensing.