In alog 52927 I posted up a video of the phase camera data taken for each sideband and carrier from DRMI to just after nominal low noise. Attached are some initial fits of a model to this data to see how each sideband is behaving during the process of locking.

The model

The model I have used for this analysis consists of two gaussian beams, one being the reference field and the other being the IFO field. The phase camera images are these two multiplied together. To get a good idea of the reference beam I took stretch of data looking at just the reference field without any IFO beam and measured the intensity at the camera. Fitting the intensity of a gaussian beam to this data we can get a good idea of the size of the reference field. The intensity of the reference field stays constant enough for each sideband over a lock stretch. For the curvature I assume the reference field is flat, which should be a good assumption considering the beam is coming out of a single mode fiber and collimated into a large beam about 70cm before the camera. "Knowing" the reference beam we can now fit the combined beam model to the actual amplitude and phase images coming from the phase camera. This fits beam size, radii of curvature, offset from the reference, angle from the reference, and amplitude of the IFO beam.

Results

This isn't an exhaustive or quantative analysis, more to come no doubt, but just what was noticed so far. On most of the plots I have plotted the average of both the upper and lower, as they are very similar.

• The sidebands look well balanced in amplitude and shape.
• There's a change in all beam size and RoC just before we power up. I need to double check when I get onto the site, but I believe this is during the MOVE_SPOTS state.
• To optimise the dynamic range of the phase camera to image all the fields you need to adjust both the camera exposure and the amount of reference light being used. Given limited time and a lack of data as we aren't connected to the CDS network, we don't do this. We just optimise to get the best SNR at full power, which also provides just enough SNR at DRMI to do something with the images
  • From carrier resonance to before powering up the SNR of the sideband RoC isn't great or the beam 
• During powerup we can see that the sidebands have different curvature and also trend differently to the carrier field
• Beam size of the sidebands and carrier also trend differently, X direction being moreso than the Y. Y direction is showing the sidebands having different shape.
• Amplitudes:
  • the 45 sidebands experience a dip in power but then recover around 1800s, not sure why
  • The 9 have a sqrt(2) drop in amplitude at ~2200s, from reducing the modulation depth

TITLE: 11/03 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 3mph Gusts, 2mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.11 μm/s 
QUICK SUMMARY: No issues.

• H1 is still locked!
• Instrument Air pumps tripped off this morning during the handoff of H1, Kyle came in, all is well, though may need more work this coming week. 
• SQZ was unlocked and recovery has not included the full restoration of the  H1 range.

Everything old is new again! 

In casual conversation this morning with Cheryl V. and Jeff B, the possible link to cryopump frost flakes falling off of the inner reserviors of the site's cryogenic pumps being a possible source of the unexplained IFO glitches seemed to have been "news" to them.  In the distant past, it had been common knowledge that glitches might be resulting from frost flakes falling off of the surfaces of the inner vessel of the 80K pumps.  Glitches could result from the flakes crossing the beam as particulates or effecting the beam when they were evaporated upon contacting the relatively warm vacuum vessel surfaces. 

We know that "liberating frost flakes" result when we hard close the GVs or otherwise bang on vacuum equipment near the 80K pumps.  We have seen pressure spikes from the evaporated frost followed by the tell-tale pump down curve for some large cases of this.  It is unknown to what degree that this is happening for events too small to see by the pressure gauges.  Meaning, not seeing this in the pressure data may not exclude the possibility that it is still happening at IFO-impacting levels too small to see with the gauges. 

Cheryl V. asked that aLOG theory as it may be of interest to others. 

Jeff B. phoned me informing me that PT199 had fallen befow 50 psi and wasn't recovering -> Upon arival, I investigated and found that neither the LEAD nor the LAG pressure switches were mechanically closing (my previous "tinkerings" had only been directed at adjustments to the LEAD pressure switch).  Repeated cycles of manually bleeding-off the local receiver tank whilst visually monitoring the mechanical contacts so as to observe if/when they were closing allowed me to adjust the "cut-in" and "span" values of both units. 

I did not find an obvious explaination as what caused the initial problem (only one instance of not working).  My current theory is that there is an imbalance of spring forces occuring on the sheet metal plate which rocks/pivots to open or close the contacts.  There are two springs ("cut-in" spring and range or "span" spring) of differing spring rates which act on this plate.  The mismatched spring forces may be causing the plate to mechanically "bind" too much on its pivot.  As such I altered the spring preload combination a little.  We'll see. 

Anyway, I am leaving the two pressure switches with nearly the same settings in hopes that, at least, one will operated in the desired pressure range.  Though this ought to reduce the compressor run time(s), this is a side effect and isn't part of the motivation for doing this.  

0946 hrs. local -> Leaving site now.

   At 15:31 (07:31) received low pressure alarm on PT-199 Instrument Air. Cheryl trended the channel and found the pressure dropping rapidly; passing through 50PSIG. 

   I called Kyle. He gave me instructions on running the compressor by hand. Went to Chiller Yard and turned on both compressors by hand. Both worked in the Hand position but did not run in the Auto position. Switched compressor #1 to Hand and built pressure up to  95PSIG. Put switch back into the Auto position. 

  As of 15:52 (07:52) Kyle was on site looking into the problem.  

TITLE: 11/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 0mph Gusts, 0mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY: H1 is locked in Observe, PT199 instrument air had both pumps trip off, JeffB running manually to keep the GVs open, Kyle is on his way

• 15:31UTC - alarm for PT199, I cleared the alarm, and checked the value and saw that it was at 54PSIG
  • PT199 had been running between 57 and 81 PSIG
• JeffB talked to Kyle, and turned pumps on
• CURRENTLY PT199 is at 82PSIB

Plot attached, showing last cycle.

   Good first half of the shift. The IFO remains locked and observing. Environmental conditions are favorable. The range is floating around 116.3Mpc. No current issues to be concerned about. 

   At 10:00 (02:00) the clocks rolled back one hour as Daylight Savings Time yielded to Standard Time. Today, at 25 hours, is truly the longest day of the year. 

TITLE: 11/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Remained locked entire shift. Squeezer took us out of observing for about a minute. Craig fixed the high frequency noise by increasing the FSS gain. Robert, Dan, Craig and Cao fixed the scattering shelves by unplugging the CMOS camera on ISCT1.
LOG:

22:53 UTC Out of observing for phase camera fan test
22:55 UTC Dan and Robert to ISCT1
23:11 UTC Robert and Dan back. Craig adjusting FSS gain.
23:17 UTC Back to observing
00:04 UTC Dropped out of observing by squeezer
00:05 UTC Back to observing
05:15 UTC Dust alarm at end Y

This week we have been working to tune the phase camera's phase locking loop and the camera exposure settings to enable us to image all the sidebands and carrier going from DRMI all the way up to full lock. The aim is too see how they may change in balance and shape. Unfortunately the only coincident times when the phase camera worked (nearly) as intended and we got a clean lock up to maximum power was Friday 1am PST before the run started and it was switched off, I was hoping to get a few powerups to compare before pulling it out on Tuesday.

https://lhocds.ligo-wa.caltech.edu/exports/powerup_odd.mp4

The above is a movie of the sidebands and carrier from left to right, carrier, +9, -9, -45, +45. Top row is phase and the bottom is the amplitude. I have applied a rolling average of 10 frames to the amplitude to stop it jittering around so much frame to frame. Each frame of the movie is ~20s. We start whilst acquiring DRMI and see the sidebands first appear and the carrier not very visible. We then reach resonance of the carrier. Next we see the powerup and the fields shifting from moving the spot positions. Shown in the bottom plot is the PSL and CO2X powers normalised. We also wanted to try pulsing CO2X on and off to see if we could see similar effects as we did before with the beam becoming more astigmatic and displaced, but with the new IFO alignment.

At around 25s seconds into the movie we see some odd shapes starting to emerge. The fields all become distorted and the signals oscillate around. This is not what is actually happening to the POP beam, at least I hope not. However it is also not a behaviour we have witnessed whilst using this phase camera previously, we had long stretches of DRMI time where this was not present. Our best guess is that phase lock became unstable, glitching maybe, with the increased power on the photodiode. Each frame in the movie is actually 10 frames averaged over 1s for each field, so it could be the glitch was large enough to distort each set of frames consistently. Which seems unlikely given the images of the fields change gradually over many frames. We weren't storing each individual frame to minmise the load on writing to the disk and also how much space was required, assuming we would leave it running over the weekend, so we can't go back and look. Although at the end of the movie it settles down to a constant shape which is odd. It doesn't really seem to match up with us changing the CO2 power, and I can't think of any reason why it would even effect the phase camera output like this.

We were seeing "frequency noise" become very apparent in DARM again.
("frequency noise" is in quotes because it is the underlying cause, but probably not the direct agent actually putting noise into DARM.  That honor belongs to intensity noise.)

We've seen a continued plummet in the reference cavity transmission over the last week.  We are now at the point where the ref cav transmission has dropped 47%.  (Attachment 1)
Jason posted in alog 52882 about trying to fix up the alignment, with some success in recovering some transmitted power (seen in attachment 1 between -200000 and -100000 seconds ago), but not resolving the underlying issue.
Probably the best witness of FSS health is the IMC_REFL_DC signal (red trace, first and second attachment).  When signal goes above ~10 mW, we are likely to be having frequency stabilization problems originating in the FSS, high high-frequency noise in DARM, and worse range.

I increased the FSS common gain from 23 to 25 dB
In the first PDF attachment, we see DARM and laser noise coherence at FSS gains of 23, 24, and 25.  
After increasing from 23 to 24 dB, pretty much all the "frequency noise" we were seeing in DARM went away.  There is no apparent change in laser noise coupling when increasing from 24 to 25 dB.

ASDs:
In the second PDF attachment, I plot DARM, CARM, and ISS secondloop ASDs before and after the FSS gain increase.
It's clear that the noise levels of all go down significantly. 
 
I expected there to be significant coherence between the ISS and CARM, but this is not the case.

Coupling:
In the third PDF attachment, we see there is high coherence between the ISS secondloop and DARM with lower FSS gain, and much smaller coherence after the gain is increased by one dB.
In the fourth PDF attachment we see low coherence between DARM and CARM at all times, likely because the true frequency noise is drowned out by the high intensity noise apparent in DARM.
However, at no time is there coherence between CARM, IMC, or FSS error signals and the ISS loop. (fifth, sixth, seventh PDF attachments)

Jury is still out on why falling FSS gain causes high intensity noise.  I would have thought this sort of intensity noise would be generated by the frequency noise incident on the IMC.  
It is possible that this noise coupling is nonlinear, or I have looked at the incorrect witnesses.

Just processed an intensity noise injection from earlier tonight.  The coupling is about the same as before and does not limit DARM.

Injection script in /ligo/home/craig.cahillane/Git/IFO/general/scripts/Intensity_noise_injection_caller.py
Plotting script in  /ligo/home/craig.cahillane/Git/IFO/general/scripts/intensity_noise_injection_plotter.py

Command Line: python intensity_noise_injection_plotter.py /ligo/home/controls/craig.cahillane/Git/IFO/IntensityNoise/data/Injections/20191031/1256622575_GPSstart_Intensity_inj_7_7300_Hz.pkl 1256625100 1256625300 --logbin