I had previously raised a preliminary concern about narrow line noise couplings to h(t) from the NCal electronics (see LHO aLOG 53666). Robert had suggested that I can use the temporary magnetometer left on the NCal encoder electronics, close to the NCal mechanism itself (much closer than the VEA magnetometer), and compare coherences between h(t) and the temporary magnetometer with h(t) and the VEA magnetometer.

Using the timeline that Timesh has left us (LHO aLOG 53629), I first selected the necessary segments using the following segment database queries:
$ ligolw_segment_query_dqsegdb --segment-url=https://segments.ligo.org --query-segments --include-segments=H1:DMT-ANALYSIS_READY -s 'lalapps_tconvert Nov 26 2019 14:00 PST' -e 'lalapps_tconvert Dec 4 2019 23:39:00 utc' | ligolw_print -t segment -c start_time -c end_time -d ' ' > segments_ncal_on.txt
$ ligolw_segment_query_dqsegdb --segment-url=https://segments.ligo.org --query-segments --include-segments=H1:DMT-ANALYSIS_READY -s 'lalapps_tconvert Dec 4 2019 23:39:00 utc' -e 'lalapps_tconvert now' | ligolw_print -t segment -c start_time -c end_time -d ' ' > segments_ncal_off.txt
Then I computed 1800 s long FFTs, Hann windowed and 50% overlapping for H1:GDS-CALIB_STRAIN, H1:PEM-EX_MAG_VEA_FLOOR_QUAD_SUM_DQ, and H1:PEM-EX_ADC_0_11_OUT_DQ for the two different periods, whether the NCal was ON or OFF. These were computed and averaged using standard LALSuite tools lalapps_MakeSFTDAG and lalapps_spec_avg_long.

Finally, I used the coherenceFromSFTs.py script written by Greg and Kara to compute coherences over these two time periods for comparison.

Attached is the result from 10 Hz to 1000 Hz, plotted in 100 Hz sub-bands. Each page has 3 figures: left shows the ratio of whitened h(t) ASD (ON/OFF), middle shows the coherence between the temporary magnetometer and h(t), right shows the coherence between the VEA magnetometer and h(t). The whitened ASD is computed with a running median across 101 frequency bins of the time-averaged spectra. This is the basis for Fscans and effectively removes slow variations in the spectrum over different time periods. I'm only concerned in this study with narrow artifacts, so this is ok to do.

The ratio of normalized spectra show some line changes (excursions above 1 indicate worse lines), especially below 100 Hz, but we also see excursions below 1. I speculate that the lines have actually changed in frequency by more than 1 bin. I see this effect other times as well, so I don't believe this is coupled with the NCal being on or off. Most of the rest of the 100-1000 Hz spectra look unchanged.

Since I don't see very much difference between coherence plots, I feel a bit more confident that the NCal electronics are ok, but I don't think we can quite put this to rest yet until we move the temporary magnetometer to sit very close to the NCal power electronics. Robert suggested that we could move this at the next lock-loss opportunity, pending permission of course.

In any case, since the NCal is not essential IFO operations electronics, I'd suggest to leave them OFF and only turn on as-needed for measurements.

   Jeff B., Scott, Tyler, Chandra, Robert. 


   Coffey Refrigeration was on site to check the backup Chrystal chiller, which was not cooling. They found the system was 1lb low on R134A (out of a 1.75lb system). They put enough R134A to clear the bubbles from the sight glass and to get the compressor running. They put dye in the refrigerant system. 

   The plan is to run the chiller until at least next Monday. Coffey will come out on Tuesday to look for dye tags of a leak. Further repairs or service will depend on the results of the dye test. 

   I will ask the day shift operator to check the chiller once a day until Monday and give me a call if the chiller is not running. The front panel should show a temperature of 20c. I will be in for Evening shift starting on Monday and will take over checking the unit then. 

   Per Robert's input, we placed the chiller on rubber dampers to prevent the chiller noise from perturbing the IFO over the weekend.       

TITLE: 12/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 7mph Gusts, 4mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.46 μm/s
QUICK SUMMARY: Locked for 16h30

TITLE: 12/19 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 7Mpc
INCOMING OPERATOR: Camilla
SHIFT SUMMARY: Other than earthquakes, it was quiet
LOG:

12:00 Got several notifications for incoming earthquakes

12:43 Saw signals in IMC-f and EQ_PEAK so, transitioned to SEI_CONF to earthquake

13:33 Watching ground and ENV_STAT, went back to WINDY

TJ and I have been working on a guardian that we hope will eventually automate at least some of the seismic configuration transitions. I think TJ has two tests in this guardian, one that just watches the EQ_PEAK channel to go above a certain threshold and one that triggers when SEISMON sees a new earthquake (based on the metric from my eq response plot). If an earthquake that we might care about shows up on SEISMON, the guardian goes to a holding state (SEISMON_ALERT), where it watches the EQ_PEAK channel to get above a certain threshold, then the ENV_STAT node goes to its earthquake state. I emphasize here that this guardian doesn't actually make any changes yet, this is just a test to see if automating is a crazy idea or not.

TJ had an earthquake yesterday where he got a SEISMON notification, and the earthquake was big enough to be an issue when it arrived on site, so he transitioned SEI_CONF to it's earthquake state. FIrst attached plot shows the state progress of SEI_CONF (green? I can't tell) and the ENV_STAT (blue) nodes on the top plot and the ground velocity EQ_PEAK monitor channel in yellow on the bottom. The green trace changing values is TJ deciding on his own when to change the SEI_CONF state (40 is the normal WINDY state, 19 is EARTHQUAKE), the blue trace is when the tests we've put into ENV_STAT suggest when we should be transitioning (20 for blue is the alert state, -10 is the earthquake state). I think the tests for ENV_STAT could use some tweaking, I don't think the difference between what TJ actually did and what ENV_STAT wanted to do are wildly different. Not saying did or didn't do the right thing (probably), but the new guardian isn't totally bonkers.

I think the biggest advantage of automating the seismic earthquake response will be forcing consistency so we can tell if we are actually doing the right thing. Looking over the run, I would believe that each of the operators has a different way of thinking  about whether or not to make the earthquake transition and when.

Currently watching a cluster of like 4 earthquakes from the Pacfic and Central America roll through, and again, the ENV_STAT doesn't seem totally crazy. Second plot is same as the first plot, but for my group of earthquakes this morning. I transitioned a bit before ENV_STAT wanted to, so maybe the "transition to earthquake " threshold should be lowered a bit.

TITLE: 12/19 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: useism is down and we have a 8 hour lock.
LOG:

• 0158 Observing

ETMY violin mode damping gains are set at nominal for all modes except modes 1 and 6.  Mode 1's peak is so small I've turned off the damping.  I've started damping mode 6.  Mode 6 started with a gain of -0.1, which was slowly ringing it up.  I switched to a gain of +0.1.  I'm leaving this gain on mode 6, and am requesting it be monitored overnight, by checking the peak height in the spectra in the diaggui file called diaggui_tracking_ETMY_mode1_6.xml, which is in my home directory (first attachment is a snapshot of this dtt).  The dtt file has a reference from tonight at04:32 UTC, and opens with the X axis zoomed in to show only ETMY modes 1 and 6. 

Monitor filters for these two peaks currently overlap, so the numbers and ndscopes do not reflect each peak, thus the need to use dtt.

I updated the damping filter for mode 6.  As found, the mode 6 damping was biased toward the mode 1 peak, and I moved the filter up in frequency so that it's biased away from the mode 1 peak.  I've attached snapshots of the damping filters for modes 6 and 1, with vertical cursors showing the frequency of the peak each filter is damping.  Mode 1 damping filter is unchanged.

Locked for about 5 hours. We went back into observing at 0158 UTC.

Daniel, Sheila

We took a reference time with no squeezing from 1260747990 to 1260749058

We turned the fiber launch power up to 18.8mW, we have 1.7mW in reflection off the OPO, and the trans diode reads 2.6 (normally its normalized to 1), but the trans diode seems to be saturated, so we have turned off the green intensity stabilization.  We measured a nonlinear gain of 8.67.  

The OPO servo gain is reduced by 11dB, to -23dB In1 gain.  The CLF gain is reduced by 6dB to 0dB fast gain, and the LO gain is reduced by 8dB to to 0dB IN1 .  We then started a scan of the squeezing angle, which should finish around 1:10 UTC. 

We ran into trouble with the LO loop with these settings for CLF angles around 180deg, so we ended up resetting the LO loop gain to 12dB, which is 4dB higher than the nominal setting.  We restarted the scan after this.  We got another reference time with no squeezing from 1260754973 to 1260755635 

J. Kissel

I've captured a standard sensing function suite, and was able to get in a full L3 and some on an L2 actuator sweep today. Attached are updates to the sensing function collection of measurements. These thermalized data sets are showing *pretty* consistent results. I'm working on getting more channels from the frames (PSL input power, temperature in the LVEA) to see if I can find other things that can predict the trends I see in f_s^2. One idea I had was that the steps between lock stretches are correlated to whether an initial alignment was run. I think I can conclude that there's no correlatino there. We're also working on backing out how bad things are *during* thermalization, with the data gathered yesterday (LHO aLOG 53951). 

Raw data from today:
Sensing Function:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
    2019-12-18_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml
    2019-12-18_H1_PCALY2DARMTF_LF_SS_5t1100Hz_10min.xml
    2019-12-18_H1_PCALY2DARMTF_BB_3min.xml

Actuation Function:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOActuationTFs/
    2019-12-18_H1SUSETMX_L3_iEXC2DARM_12min.xml
    2019-12-18_H1SUSETMX_L3_PCAL2DARM_6min.xml
    2019-12-18_H1SUSETMX_L2_iEXC2DARM_12min.xml
    2019-12-18_H1SUSETMX_L2_PCAL2DARM_6min.xml    << only half a measurement, because of the canadian EQ

As always -- more to come!

TITLE: 12/19 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 8mph Gusts, 6mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.44 μm/s
QUICK SUMMARY: Some commissioning time is scheduled for now, and we just recovered from a lock loss.

TITLE: 12/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Unknown
INCOMING OPERATOR: TJ
SHIFT SUMMARY:

• H1 lockloss during calibration work (probably due to a nearby/fast AK/Canada 5.3 EQ).
• Transitioned SEI_CONF to the WINDY state (kept CPS Diff OFF).
• CDS Overview is all GREEN.
• Handing off shift while H1 is Commissioning time (Shiela for Squeezer).

LOG:

• ~16:30 Tyler working in Hi-Bay.
• 16:35 Sheila noticed PCalX  trace on the DARM spectrum on the wall increased (notified Ethan)
• 16:39 SUSAUXB123 on CDS Overview had a RED adc error.  Since we were out of  Observing, David clicked DIAG RESET.
• Dave also mentioned filter changes (yellow CFC errors on CDS Overview) for:
  • SUSETMx (related to TRamp change) ----Sudarsha
  • SUSPROC (related to Tramp, but also looks like error in code).  Want to know who did this and if we are OK with these changes, because 
• 16:47 Chandra out to address RGA on HAM6 End Cap (unplugged/plugged and then left unplugged).
• 17:05 Back to OBSERVING!
• 18:17 Optics Lab work (Vlad will be in/out throughout day),
• 18:25 PSL Chiller check (Camilla)
• 18:42 MX & MY visit (Kyle)
• 19:06 MY trip (Karen)
• 21:00 CALIBRATION begins
  • Note:  Lock Clock restarted when ISC_LOCK went to NLN_CAL_MEAS (to our surprise after chatting about this with TJ last night).  However, we do have Dave's medm Lock Clock on the CDS Overview and this is mainting the lock time (it looks at NLN states or higher).
• 21:08-21:11 SEI_CONF to WINDY
• 21:55 Alaska/Canada 5.3 EQ Alert received (with 4min heads up & clearly in earthquake regime).  Waited for Jeff to complete a data point and then....
  • 21:57 SEI_CONF to EARTHQUAKE
• 22:01 Lockloss
  • Obs Mode:  EARTHQUAKE
• Locking Note:  In GREEN ARMS, not clear when INCREASE FLASHES makes a 2nd attempt...didn't see it try after almost 30min, so intervended & Jeff & I tweaked up Green Arms.
• 23:25 OBSERVING
• 23:45 COMMISSIONING (~90min) begins (Sheila)

At 22:31 UTC (14:31 PST) we had another ADC glitch on h1susauxb123 (second for today, for 05:08 PST glitch see alog-53962)

To gather more diagnostics, I'm running a script on zotws6 which, once a minute, which checks for an ADC error, and if one is found issues a DIAG_RESET

At JeffK's suggestion, since the microseism has been coming down, we've transitioned back to our nominal WINDY seismic configuration.  However, the CPS DIFF is still off, since the microseism BLRMS is still above ~0.04 um/sec.  That seems to be about the level at which we started seeing glitches in Omicron.

So, if the microseism BLRMS is above 0.04 um/s (the units of the channels when viewed in ndscope is nm/s, so the value will be 400 nm/sec in ndscope for channels such as H1:ISI-GND_STS_ITMY_Z_BLRMS_100M_300M), let's keep CPS DIFF off.  If we go below that level, we can turn back on the CPS DIFF by setting

H1:ISI-DIFF_CONTROL_BIT to 1, or by opening the SEI DIFF controls screen and clicking ON in the lower righthand corner.  To get to that screen:

• Sitemap
• SEI button
• Choose ISI Sensor Config
• Yellow button labeled "ISI DIFF" near the top of the screen
• Choose Output Limiters
• On/Off button has the word "master" written next to it in the lower right corner.
  • There is a box around the On/Off button.  If it is grey, then you will see that all of the numbers on the far right side of the screen are zero, and the system is off.  If it is green, then you will see that there are nonzero numbers going to the ISI platforms and the system is on.

As a summary of my previous alog, which detailed a number of issue I had whilst propagating UIM dynamics recorded in another alog, with the early indication of of their relevance being alluded to in yet another alog.

One of the main things I found is that when you have a sufficiently complicated transfer function, Matlab appears to start breaking down in ways that vary in each Matlab version (see last comment in previous alog). I have written a script:

/ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/matlab_scripts/

sus_ss2zpk.py

Which takes a Matlab SS model and converts it to ZPK. The comparison can be seen in pythzpk_vs_matlabss.png. Matlab appears to struggle to numerically resolve the TF after the 150Hz feature, giving rise to apparently large errors in the TF comparisons. In the version of the TF that didn't have the UIM dynamics in the range 45-150 Hz, this numerical error was happening at much higher frequencies. The main point is that the ss2zpk function in matlab is creating hugely spurious features at 0.5 Hz, and here we keep the error at that frequency at 1%. This may not be perfect, but it is a hell of a lot better than what Matlab is doing (Compare the magnitude error between this (matlab SS vs matlab zpk) and this (matlab SS vs python zpk)).

Having propagated this transfer function fully into pyDARM, with a new parameter model in:

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/params/

modelparams_H1_20190909_UIMdyn.py

, using the fit (2019-11-20_H1SUSETMX_L1_LFMeas_vs_Fit.pdf), we can see the the effect on the DARM response: 2019-12-16_O3_H1_DARMLoopCritique_ContributionsToR_mag.pdf. In comparison to my first estimate on the impact of this update to the response (old alog), you can see the little blip created at 150Hz quite clearly in the 1/C curve.

The hope is that this will resolve the observed miscalibration at 150 Hz. I am working on regenerating that same plot to take into account this new dynamics factor to (hopefully) show that we can be correctly calibrated there as a result of this work.

We were just knocked out of observe twice by something adjusting the CAL-INJ_CW_GAIN, the first time it was set to 0 at 11:21 utc, then again when it was set to 1 again at 11:27. I don't know if this was intentional, done by a guardian. The only remotely connected person I see is Keita, no one else is here. If this is being adjusted by some guardian should we unmonitor this gain?

J. Kissel

I've made a second, better attempt at tracking the long-time-scale thermalization of the IFO for the first few hours of a nominal low noise stretch -- originally attempted in LHO aLOG 53824. 

The data analysis will take some time, but here're are the measurement details. (All times are UTC on 2019-Dec-17)

    21:22 Test Start
    21:23 All lines on at conservative amplitude, SEI conf = USEISM, with SEI DIFF ON
    21:24 Hit Nominal Low Noise, all lines visible in DELTAL_EXT
    21:26:(15-30) Accidental brief turn off of (specifically) DARM lines
    21:40 Begin increase in line strength, and turn OFF of SEI DIFF
    22:03 Increase in some PCALY line heights to increase coherence
    23:20 All lines OFF, test done.

Hopefully, with all the changes in the lines during the measurement, the only effect will be that the coherence in the transfer function improves with time. During today's measurement, the microseism was well above the 90th percentile, in the ~1 um/sec range, so the 15 - 50 Hz data was rather glitchy, full of scattering arches.

Here're the list of frequencies driven (exactly, in Hz, bold frequencies are standard calibration lines):
    freqlist_darmx = [5.625,    7.45,    9.1,   18.9]
    freqlist_pcalx = [  5.45,    5.8,   7.25,  7.625,    8.9,  9.275, 18.725,  19.1, 49.775]
    freqlist_pcaly = [17.100, 410.30, 1083.7, 23.525,   25.3,  30.85, 49.625, 84.65,161.625]

Script to turn on PCAL EXC lines:
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/CALCS_FE/setup_sensingfunction_callinecombs.py

DTT template to turn on DARM EXC lines:
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/2019-12-17_H1DARMEXC_SpringMonitoring.xml

(To turn off all the extra lines, I just hit Abort on the DTT template, and used SDF to revert all the PCAL settings. The only thing that *didn't* work for was the PCALX high frequency roaming line frequency, which I had to restore by hand at it's current sweep point, 3001.3 Hz).

J. Kissel,

As we've done in O3A (see LHO aLOG 51245), I hold this space for documenting start times of high-frequency roaming calibration line sweeps to be used for estimating the sensing function uncertainty above 1 kHz.

(All times UTC)
2019-11-01 16:55:02 HIGH_FREQ_LINES guardian reinitialized for the Start of O3B, starting at 4001.3
2019-11-12 18:35:25 data not found on ndscope while only shortly at last data point, 1001.3,  [CAL EX / EY front-end models restarted, and filter coefficients updated; see LHO aLOG 53188 LHO aLOG 53210]
resumes at
2019-11-12 18:38:37 re-started at 4001.3 Hz
2019-11-21 00:49:47 re-started at 4001.3 Hz
2019-11-29 14:40:20 re-started at 4001.3 Hz (and not yet finished)