• H1's been locked almost 23hrs (observing 3+hrs).  
• Weekly Calibration time will begin at 1pm & Squeezer work will also occur in parallel.
• SEI_CONF still in USEISM state.

25ml added to Crystal Chiller

Corey, Sheila, Daniel, Keita

Summary: We are back to squeezing and in observing after having introduced some yaw offsets in the 42MHz WFS, for now this has addressed the problem that Jim describes in 53956.  It seems that the original problem is that the nonlinear gain has slowly degraded since the crystal move yesterday.  This issue cuased us to be out of observing for about 4 hours this morning, and in preventative maintence for a little while. 

• There was a race condition in the squeezer guardian, where when the LO loop looses lock, the beam diverter opens and the guardain goes to SQZ_READY_IFO, and checks if all the other squeezer loops are OK.  If so, it tries to go back to squeezing and open the beam diverter.  In our most common squeezer failure mode (SHG or OPO PZT railing) it takes a few seconds for all the other squeezer loops to relock, so that the guardian will not try to re-open the beam diverter until a few seconds after it finishes closing.  In this case, only the LO and ASC loops came unlocked, so the guardian would issue the command to re-open the beam diverter before the beam diverter was "ready".  Based on notes in the guardian, it seems like this has been encountered before and a busy flag was added to the beam diverter library for this reason, but the guardian code hadn't been updated to check it. I added a check that the beam diverter be ready.  This should mean that if a similar situation occurs again the squeezer would be able to re-lock itself.
• We relocked the squeezer, and saw that the 3MHz power through the OMC was degrading as the ASC control signals moved away, repeating the pattern that happened during Jim's shift.  We disabled the ASC (using the gain slider on the SQZ ASC overview) and saw that the transmitted 3MHz power stabilized.  I then manually walked the alignment of the ZMs, and found that ZM2 yaw was able to restore the 3MHz power to nearly -30dBm, about 1dBm lower than the level before yesterday's crystal move (all of this is shown in the first attachment).  I adjusted the offsets in the AS42 yaw signals, and we reclosed the ASC loops.  The second screenshot shows the offsets that we accepted in SDF to go back to observe.  
• 3rd screenshot:  It seems that the nonlinear gain was started to slowly degrade after we adjusted the crystal position yesterday (the time of the crystal adjustment is just after -80000 seconds in the 3rd attachment) (53943).  The OPO reflected power was increasing, while the fiber launch power also slowly increased, (OPO green trans is servo'd).  The 6MHz power dropped, which indicated that the nonlinear gain was dropping, as well as the 3MHz power on the OMC.  
  • It seems likely that the crystal green absorption has increased quickly in this new position.  I seem to remember hearing about a fast degradation at LLO after they moved the crystal (48897) although I'm not finding the right alogs at the moment.  If this is the case, the crystal temperature would have changed due to a change in the local heating which isn't sensed by the thermistor.  This means we can probably restore our nonlinear gain by adjusting the thermistor temperature setting.  
  • We didn't attempt to optimize the crystal position yesterday to overlap the co-resonance with the phase matching temperature.  Doing this could make us a bit less sensitve to changes in temperature, which would be good if the absorption is gonig to be changing.  

As a summary of my previous alog, which detailed a number of issue I had whilst propagating UIM dynamics recorded in another alog, with the early indication of of their relevance being alluded to in yet another alog.

One of the main things I found is that when you have a sufficiently complicated transfer function, Matlab appears to start breaking down in ways that vary in each Matlab version (see last comment in previous alog). I have written a script:

/ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/matlab_scripts/

sus_ss2zpk.py

Which takes a Matlab SS model and converts it to ZPK. The comparison can be seen in pythzpk_vs_matlabss.png. Matlab appears to struggle to numerically resolve the TF after the 150Hz feature, giving rise to apparently large errors in the TF comparisons. In the version of the TF that didn't have the UIM dynamics in the range 45-150 Hz, this numerical error was happening at much higher frequencies. The main point is that the ss2zpk function in matlab is creating hugely spurious features at 0.5 Hz, and here we keep the error at that frequency at 1%. This may not be perfect, but it is a hell of a lot better than what Matlab is doing (Compare the magnitude error between this (matlab SS vs matlab zpk) and this (matlab SS vs python zpk)).

Having propagated this transfer function fully into pyDARM, with a new parameter model in:

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/params/

modelparams_H1_20190909_UIMdyn.py

, using the fit (2019-11-20_H1SUSETMX_L1_LFMeas_vs_Fit.pdf), we can see the the effect on the DARM response: 2019-12-16_O3_H1_DARMLoopCritique_ContributionsToR_mag.pdf. In comparison to my first estimate on the impact of this update to the response (old alog), you can see the little blip created at 150Hz quite clearly in the 1/C curve.

The hope is that this will resolve the observed miscalibration at 150 Hz. I am working on regenerating that same plot to take into account this new dynamics factor to (hopefully) show that we can be correctly calibrated there as a result of this work.

Ethan, Sheila, Evan, Jeff, Corey

At 2019-12-18 16:07 UTC (1260720463), the PCAL X optical follower servo loop turned off with no discernable cause (orange button centre screen on PCALX screen). There was no changes made to the SDF for this to occur. The SDF information is located under EX_ECAT_PLC3 with channel names including PCALX_OPTICALFOLLOWERSERVO. This is distinctly different from the OFS saturating, which would not turn off the loop. H1:CAL-PCALX_OPTICALFOLLOWERSERVOENABLE encodes the state of the OFS loop, and has been plotted in OFSENABLE_turnoff.png. At 400s I turned the loop on briefly before turning it off for fear of losing lock (large excursion of broadband PCALX noise into DARM due to lack of ramp time). The OFS loop was off for ~50 minutes, fortunately we were not observing during this time.

Attached are two plots (OFS_off_worst_case.pdf and OFS_off_typical.pdf) which, as their filenames suggest, show the worst case of the elevated noise from PCAL X when the OFS control loop was off, and the typical elevated PCAL X noise observed during this period of time.

We should look into this further.

Summary:  H1 was out of OBSERVING from 14:04 - 17:05 utc.  (Remains locked throught all of this.)  Multiple activities noted below (& I'm sure alogged in-detail by others).

Squeezer ASC (Alignment) Drift Begins After Maintenance (Sheila, Jim, Corey)

Jim experienced issues with the Squeezer during his shift and we were not able to bring it back.  Sheila arrived and investigated (I'm sure she'll summarize).  First change she made was related to logic for the Beam Diverter.  But it looks like the main change was a noticeable drift in alignment for the Squeezer which started pretty much from the beginning of this current lock (which was first one after Maintenance).

Sheila turned OFF SQZ ASC, and then made adjustments (offset values added) to fix the Sqeezer ASC system.  BUT we want to monitor the Squeezer for (1) alignment drifts and (2) if we want to keep these new offsets.

OBSERVATORY MODE Notes:  For most of the out-of-OBSERVING time, Observatory Mode was in UNKNOWN state.  Once Sheila was happy with Squeezer, I had transitioned to the CORRECTIVE MAINTENANCE state for a few minutes, and then we went to OBSERVING.

PCalx (Ethan, Sheila, Evan)

While Sheila was addressing the Squeezer she noticed the PCalx (pink) trace on the running DARM spectra on the wall increased broadband (almost up to DARM level!).  Went to get Ethan for help.  They discovered a switch randomly turned OFF (coincidentally while Sheila was working on Squeezer!).  Ethan/Sheila toggled the switch and PCalx is back to normal.

CDS (Dave, Corey, Sheila)

Dave called in (coincidentally while we were still out of OBSERVING & addressing Squeezer) because of errors he noticed on the CDS Overview.  In order to proactively clear these, he phoned in.  Since we were out of OBSERVING:

1. SUSAUXB123 was DIAG RESET by Dave (around this time the PCalx issue was noticed...probably not related, just noting)
2. SUSPROC & SUSETM had Filter Change errors (CFC).  Dave wanted us to confirm these are known changes (because we wouldn't want a crash to happen during the holidays and these changes come up without being known/approved).  Attached is a snapshot of the errors/notes.
   • SUSPROC was related to ETMy MODE12 filter change & Rahul approved a COEFF LOAD
   • SUSETM was related to ETMy TRamp change from Sudarshan/Sheila work yesterday.  This was approved by Sheila & we clicked COEFF LOAD.

HAM6 RGA End Cap Unplugged (Chandra, Robert)

This RGA was brought up as a noise source.  Since we were out of OBSERVING, Chandra quietly walked out to it at the HAM6 End Cap door (with Sheila giving OK).  She unplugged/plugged for a few secs and then left unplugged.

Back To OBSERVING!

There were some SDF diffs, but these were related to ZM1/2 alignment changes and Sheila had me accept.  Sheila mentioned she had some time later today for some other Squeezer Work, but she might use this time to check up on the Squeezer post-changes from this morning.

A new residual gas analyzer (RGA) was installed on the end cap door of HAM6 during Oct. break and its cooling fan is noisy. This morning I entered the LVEA at 16:45 UTC while we were out of observing mode but still locked to unplug RGA. Since Robert is on site, we did a test where I unplugged the unit for ~ 40 seconds at 16:48 UTC, plugged back in for 30 seconds at 16:49 UTC and left unplugged before exiting LVEA at around 16:52.

Corey, Dave:

h1iopsusauxb123 had a momentary ADC glitch at 13:09 UTC (05:09 PST) this morning. While we were out of observation, I cleared it with a DIAG_RESET on this model.

TITLE: 12/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Unknown
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    SEI_CONF state: USEISM
    Wind: 4mph Gusts, 3mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.57 μm/s
QUICK SUMMARY:

H1's been out of OBSERVING (Obs Mode is in the UNKNOWN state & I'll go ahead and leave it there) due to issues with the Squeezer.  Jim mentioned trying an INIT on the SQZ_MANAGER, but it would drop out within minutes later (I tried that & observed the same.).

Sheila is here and taking a look.

Microseism has been lowering over the last 24hrs.  Currently is hovering at/just-under the 90th percentile.  Asked about whether we should transition to WINDY, but might stay in Microseism.

TITLE: 12/18 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Unknown
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
LOG:

11:21 some CW gain change knocks us out of observer

13:00 till end of shift SQZ starts unlocking, won't stay locked, out of observe the rest of the night.

Not sure what is going on, but twice now the squeezer has kicked us out of observe. Each time the guardian reaches the SQZ_ASC state, and won't move on its own. First screen is the log, second are the sdf diffs I find. Each time I have inited the guard, SQZ relocks and the SDF diffs clean up. But it's happened twice now in about 10 minutes.

We were just knocked out of observe twice by something adjusting the CAL-INJ_CW_GAIN, the first time it was set to 0 at 11:21 utc, then again when it was set to 1 again at 11:27. I don't know if this was intentional, done by a guardian. The only remotely connected person I see is Keita, no one else is here. If this is being adjusted by some guardian should we unmonitor this gain?

TITLE: 12/18 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 112Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: 10.5 hour lock. useism trending down.
LOG:

• 0245 SEI_CONF to EQ
• 0327 SEI_CONF back to USEISM

Switched to earthquake mode from 0245-0327 UTC. The transition back to USEISM had me thinking that we were going to lose lock, but we made it.

6.5 hour lock.

TITLE: 12/18 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 110Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: USEISM
    Wind: 3mph Gusts, 2mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.68 μm/s
QUICK SUMMARY: Quick recovery from Tuesday maintenance, we have been locked for 2hr 45min. useism is still very high, but maybe on its way down.