TITLE: 12/07 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 2mph Gusts, 1mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.33 μm/s
QUICK SUMMARY: ETMY violin mode 20 is at 4.33. No other issues.
TITLE: 12/07 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
LOG:
Nothing to report. Weather got worse, but our range is better.
TITLE: 12/06 Day Shift 16:00 – 00:00 (00:08-16:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Rode through EQ, quiet shift otherwise. Locked 25 hours, Observing 20 hours.
LOG:
16:19 (08:19) Switching to EQ mode for 5.7 near Wallis and Futuna
16:48 (08:48) Switching back to Windy
17:10 (09:10) Dripta, Vlad to Optics Lab
18:29 (10:29) Dripta, Vlad out of Optics Lab
20:55 (12:55) Travis to Optics Lab
21:04 (13:04) Travis out of Optics Lab
21:58 (13:58) Vlad to Optics Lab
22:22 (14:22) Jack to Optics Lab
22:32 (14:32) Jack out of Optics
PT110 has become quite noisy lately (see attached graph). The HV interlock is no longer tied to PT110 but we should investigate this at some point anyway as, now, the resulting average pressure is higher than before the noise.
TITLE: 12/06 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Niko
SHIFT SUMMARY:
Quiet night. TCSy chiller low flow alarm and a few EQs rolled in.
LOG:
I'm dropping in a comment on these 2 earthquakes on Dec 6 (UTC) because riding through them helped LHO make the 113 hour record. After the fact, the USGS rated these
as a M6.0 at 13:04:46 UTC and M5.7 at 15:42:44 UTC, each at depth of 10 km (since it is shallow it makes more surface waves) and each about 170 km NW of Tonga (near Fiji). The peak vertical surface wave velocities were 'only' about 1 and 0.7 microns/sec. Big enough to be a problem, but well within the 'ride-through' range for the EQ mode.
The detector stayed in Observing mode for both. The range was not effected much, although you can see some scattering.
LHO should brag about this as one of many things done to set the new record. Nice work.
Fig 1 shows the USGS page for the events.
Fig 2-4 are the DetChar summaries for peak ground velocity, detector range, and normalized h(t) spectrogram.
Fig 5 is an overlay of Figures 2 and 4, showing the extra noise at 20 Hz is correlated with the quake.
(I've updated the figures to turn off the 'alpha' channel exported by the DetChar summary pages. This transparency was making it hard to read)
Ops Shift Transition: 12/06/2019, Day Shift 16:00–00:00 (08:00-16:00) - UTC (PT)
State of H1: Locked
Intent Bit: Observing
Weather: 0-5 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.4 um/s
Outgoing Operator: Corey
Quick Summary: Locked for 17 hours, Observing for 12 hours
12:36 Received "TCSy Chiller flow is low" alert (Attached is trend of the TCSy flow rate. (It takes a drop at 12:23 w/ no alarm. Then takes a small jump up at 12:36).
Went out to the chillers to check & top them off:
After the water was added, TCSy chiller returned to its normal Flow Rate.
Quiet shift with H1 locked for just over 13hours. Still chilly and foggy out.
TITLE: 12/06 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 2mph Gusts, 1mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.25 μm/s
Chilly drive in with temps dipping to 24degF & fog. Winds are calm and under 5mph. useism is just above the 50th percentile.
QUICK SUMMARY:
Patrick passed on all the latest info for h1 operations. H1's been locked almost 9hrs & current Observing mode stretch is 4+hrs.
TITLE: 12/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 115Mpc INCOMING OPERATOR: Corey SHIFT SUMMARY: Dropped out of observing for a period due to squeezer. No other issues. LOG: 00:17 UTC Rick, Dripta, Vlad out of optics lab 03:54 UTC Dropped out of observing by SQZ_MANAGER 04:01 UTC Hit INIT on SQZ_MANAGER 04:02 UTC Observing
Dropped out of observing for a period due to squeezer. No other issues.
Vlad and Jenne,
From the summary pages, as of 0200 UTC 20191204, the 3rd bandpass region of the sqeezing has beern reporting increasingly degrading squeezing starting at near -2dB, to sitting at above +2dB currently.
Jenne nagivated the the sitemap to find that this band corresponds to the region between 500-900 Hz, and suggested I run a BruCo before this happened, as well as more recently to help find an issue, to hopefully spot a noise contributor to DARM in that band.
I am currently running a BruCo - and should be able to inspect the data tomorrow.
In the attachments you can see the relevant plot of this band creeping up over the last two days. (its the green one)
Completed Bruo:
I shall guide your eyes to the frequencies 838 and 839.
Here you will see many channels correlating somewhat strongly in the After that were not present in the Before.
Importantly things like:
PEM-CS_MAG_LVEA_VERTEX_X_DQ, PEM-CS_MAG_LVEA_VERTEX_Y_DQ,PEM-CS_MAG_LVEA_VERTEX_QUAD_SUM_DQ
suggest something new in the magnetic spectrum
PEM-CS_RADIO_ROOF4_BROADBAND_DQ, PEM-CS_RADIO_ROOF3_BROADBAND_DQ
also suggest something electronic - visible on the LVEA roof? need to be sure whether it is internal or external
LSC-MOD_RF9_AM_ERR_OUT_DQ, LSC-MOD_RF45_AM_ERR_OUT_DQ
suggests that this is affecting the sideband generation before it is applied to the EOM in the PSL
LSC-MOD_RF45_AM_CTRL_OUT_DQ, LSC-MOD_RF9_AM_AC_OUT_DQ
suggests that the cavity control sidebands are affected
Many channels like :PEM-CS_ADC_5_30_2K_OUT_DQ
Visible in many electronics monitors
ASC-AS_B_RF72_Q_SUM_OUT_DQ, ASC-AS_A_RF72_Q_SUM_OUT_DQ, ASC-AS_B_RF72_I_SUM_OUT_DQ, ASC-AS_A_RF45_I_SUM_OUT_DQ, ASC-REFL_A_RF45_Q_YAW_OUT_DQ, ASC-AS_B_RF36_I_PIT_OUT_DQ, ASC-AS_B_RF36_Q_YAW_OUT_DQ,
That the noise propagates through to the various wavefront sensors
I am currently trying to use Lasso to see how one of the more interesting channels has evolved in that band:H1:PEM-CS_RADIO_ROOF4_BROADBAND_DQ
As of 0300 UTC 20191213, This noise has stopped and Sqeezing seems to be reported fine.
J. Driggers, J. Kissel, Y. Lecoeuche T. Shaffer
We've been running in to a problem with the MICH_BRIGHT initial alignment step for the past few days (months?) in which the interaction between the automated initial alignment guardian manager (INIT_ALIGN) and its subordinate (ALIGN_IFO) does not make sufficient checks of the MICH system upon failure to lock before requesting to reacquire. As such, Jenne, TJ, and I set out to find a solution.
The conclusion: there are two problems --
(1) During a successful first attempt at locking MICH BRIGHT, while the WFS loops are trying to converge MICH alignment, the M2 stage saturates (with a quite fast, but normal noise excursion). This saturation impulse kicks the ASC loops slewing the alignment to slowly bad, dropping the AS port QPD SUM which is used as a length-locking error signal, H1:ASC-AS_A_DC_SUM_OUT16, down below the threshold for a check whether MICH_BRIGHT is locked.
See first attachment.
(2) Once it's lost lock the first time, there's some screwy logic corner between the INIT_ALIGN manager, and the ALIGN_IFO subordinate which means the manager never realizes that the beam splitter is oscillating wildly, and at the same time trying to reaquire, which is blasting the SUS, pitching it more, and causing an AS port, Beam Splitter Dance Party / Laser Light Show.
See second attachment.
TJ and I worked a bit on Problem 2, by changing some of the logic and clearing up the issues with the screwy logic corner this past Tuesday. These change were all inside the INIT_ALIGN guardian, in the states MICH_BRIGHT_ALIGNING: we
- used the (IMHO) much more clear guardian call to gather the current state of a guardian (e.g. nodes['ALIGN_IFO'].STATE == 'DOWN', instead of just nodes['ALIGN_IFO'] == 'DOWN', which looks much like the request to *change* the state, nodes['ALIGN_IFO'] = 'DOWN', and only differs by one equal sign)
- Instead INIT_ALIGN's MICH_BRIGHT_ALIGNING state making of two ambiguous checks that ALIGN_IFO has locked MICH BRIGHT before advancing to OFFLOADING_MICH_BRIGHT by only checking whether ALIGN_IFO has "arrived" in any state and that state is "done", we use more rigorous check that it has arrived and has completed the state "MICH_BRIGHT_ALIGN," i.e. the state that cooks the initial alignment.
- Hoping that we've removed the logic flaw, we also reduced the "chill out" timer -- which is used after several attempts at locking MICH -- from 75 seconds to 45 seconds.
Since we've made the change (on Tuesday), we've NOT run an initial alignment beyond INPUT_ALIGN_OFFLOAD, so this change has not been tested.
Also -- while I *thought* that we'd added an additional check in the ALIGN_IFO state for ACQUIRE_MICH_BRIGHT to check that the max-min of the optical lever pitch signal is less than 0.5 urad, that test appears to have gotten lost in the fray.
We have some more work to do to understand Problem (1), and to develop tests against it. [And today, as I was writing LHO aLOG 53711, I realize that we may have been having his problem since August 2019 because we have been in the wrong coil driver state!]
Just went through this state a few times, including intentionally breaking the MICH bright lock, and it seems that Jeff's find and fix of ensuring that the BS coil driver is in the correct state did indeed solve the problem.
In order to understand the angle to length coupling, we made some A2A and A2L to DARM transfer function measurements, few weeks ago, using broadband excitations and noticed some phase mismatch between the measurement and the model as seen in the first attachement [AngletoDARM_20191126.png].
In order to make sure the phase mismatch, we observed, was not due to some non-linear coupling due to the high coherence excitations, we made additional measurements using sinusoidal exciations with few different amplitudes. The second and third attachment [ 2019-12-04_A2LEXC_DARM_38W.png and 2019-12-04_A2AEXC_DARM_38W.png] below show similar phase offset that we saw in our broadband injections ( ~10 deg compared to model) for both A2A to DARM and A2L to DARM measurement.
Since I've been poking around with pyDARM code for a little bit, I notice that there are a lot of corrections for delays. Perhaps the phase error you are seeing comes about from the combition of delays: totalUserDelay, DAC delay, actuation delay and a specific "unknownPUMDelay"... there are few more delays maybe.
Have a read in the
/ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/src/actuation.py (computeDARM function)
You can use output from that function (actProd), to spit out the transfer functions of the relevant delays, to evaluate at your data points and subtract away.
J. Driggers, J. Kissel
We don't have the fortitude to investigate further in this scary snafu, but we want to note it for future reference in case it happens again:
Once we arrive at nominal low noise, the SDF system informed us that ETMX's global yaw control output for the L2 stage only (H1:SUS-ETMX_L2_LOCK_OUTSW_Y) was OFF. This meant that there was no global yaw control (DOFs DHARD, DSOFT, CHARD, CSOFT) getting out to the PUM.
We noticed it fast enough to
- set the ramp time to 4.0 seconds
- ramp down the gain of the L2 LOCK Y bank from 1.0 to 0.0,
- turn on the output switch, as is nominal, and
- ramp the L2 LOCK Y bank gain back to 1.0
in rapid succession such that we saw no noticeable effect on the IFO, and we were able to enter in to OBSERVATION READY unscathed.
I attach a trend of this channel, and it's similar channel for pitch over the course of O3B, since November. This switch has only been switched OFF 3 times (including today), and *other* than today, it happened while the IFO was down:
ISC_LOCK N State 11
2019-11-12 18:35:25 UTC OFF Ghost
18:38:37 UTC ON
ISC_LOCK N State 12
2019-11-26 20:35:57 UTC OFF Ghost
20:39:09 UTC ON
ISC_LOCK N State 600
2019-12-05 23:19:58 UTC OFF
23:51:27 UTC Jenne and Jeff turn it back ON
However, as one can see, these three times happen in three different ISC_LOCK guardian states, (11: READY, 12: LOCKING_ARMS_GREEN, and now in 600: NOMINAL_LOW_NOISE), and it's changing far LESS times that the ISC_LOCK guardian goes through the acquisition, implying that this switching has nothing to do with a rogue guardian.
We've also ruled out the ISC_LOCK guardian entirely, by perusing it for L2_LOCK_OUTSW toggles, and it's only in DOWN and DHARD_WFS that this switch is touched, and in both places, it's forced to 1.0 i.e. ON (maybe indicative of ghosts having spooked us in the past).
Jim, Camilla and I had a problem a few weeks ago where because the output buttons are the only clickable things on the OM overview, a stray mouse click can easily turn them off and it is not super obvious (late at night) that they are wrong on the screen. One proposal to prevent this is to remove the buttons from the OM overview screen, although I can't remember if anyone is planning to do this.
I seem to remember that for the quads these outsw buttons are one of the only things clickable from the overview screen as well. Perhaps these ghosts are just stray mouse clicks and we can eliminate them with medm science.
[Sudarshan, Jenne]
This afternoon we moved the ADS line at 22.347 Hz down to 22.147 Hz. All other ADS lines remain the same. This line was the most egregiously close to an interesting pulsar frequency.
Once the IFO relocked and the ADS alignment had converged, we opened the Yaw4 ETMX Yaw loop and changed its frequency. We made a new bandpass for the SIG filter bank, to bandpass at the new frequency. Sudarshan had calculated that we would only need to change the A2L gain by about 0.004, so we left it alone. We rephased by making I signal zero, which required increasing the phase by 5 degrees.
We then turned on the loop, and things seem stable and good. It's hard to really say, but maybe the range increased a teeny bit? We think that we have made these changes 'permanent' by editing the PREP_ASC_FOR_FULL_IFO guardian state, and accepted these values in both the safe and observe sdf files for the ASC.
During the last long lock, I had hand-selected the correct bandpass filter. However, the guardian selected the wrong bandpass for the new frequency. Since we've seen that having moved this line is okay, I just put the correct bandpass in the location that the old bandpass was, so that I didn't have to change guardian. (that place is FM6 in the demod SIG filter bank for ads yaw4). I have accepted this in the safe and observe snaps for the ASC.
Having this bandpass wrong seems to have misaligned the IFO (not surprising) enough that we had too much PRCL gain (the PRCL loop was oscillating on the high side of the phase bubble), but not enough that we had any other symptoms of poor alignment. Anyhow, next lock, having fixed the bandpass, there were no problems with the PRCL gain.
TITLE: 12/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.30 μm/s
QUICK SUMMARY: No issues.
J. Kissel I've processed the ETMX longitudinal actuation strength data from 2019-11-11 (see day-of LHO aLOG 53162) for each of the three stages. In doing so, I've trended the front-end produced time-dependent correction factors to see if the systematic error between "the actuator strength used in the currently-still-in-use 2019-09-09 model" (which is actually the same numbers we've been using since the start of O3 in April 2019) and "the fit to the 2019-11-11 data" is small (i.e. the percent difference is unity). In short -- it's the perennial question: "anything that shows evidence that we need to update the model?" or, asked differently, "any evidence that the time dependent correction factors are not doing their job?" Summary: All continues to be well-understood and accurate at the level of concern needed with the O3 ETMX actuator. But, beneath that level of concern, there are interesting things happening. The attached .pdfs are the plots that show the fits to the 2019-11-11 data. Below is a table highlighting the results which help answer the above question: UIM PUM TST Reference Model 7.67e-08 (N/ct) 6.036e-10 (N/ct) 4.727e-12 (N/ct) MCMC Fit 8.038e-08 6.086e-10 4.780e-12 kappa via MCMC 1.0478 1.00824 1.01124 kappa via CALCS 0.997 1.013 1.012 Systematic Error 0.048% -0.0047% 0.00075% (MCMC-CALCS)/MCMC I was at first a bit alarmed by the 4% systematic error in the UIM, but the MCMC fit appears to be poor -- spoiled by the data point at 10 Hz -- so I've resumed a state of zen. Also, we've shown that a 5% scale factor systematic error on the UIM has no significant impact on the overall response function systematic error above 10 Hz, so even if the systematic error was real, I'd still brush it off. The previous time this table was made was for the 2019-10-28 measurement. See LHO aLOG 52973. Then, all actuation fits agreed with TDCFs within 0.5%, so I argue that the TDCFs are tracking reality well. Finally, the image attachment is a 1 month trend of the time-dependent correction factors, just because they turned out to be interesting -- but not consequential -- when I looked. The interesting points: (1) The high duty cycle from this month allows us to *clearly* resolve a six-hour period, 0.2% amplitude fluctuation in the ESD actuation strength. I attribute this to the first time LHO has demonstrably seen the impact of differential reaction-chain to main-chain distances changing due to tidal control. This change in chain-to-chain distance means the gap between reaction mass (the ESD pattern) and the test mass changes, thus impacting the ESD actuation strength. LLO regularly sees this because they drive all of their tidal control to *only* the main chain at the TOP mass level, as opposed to LHO which drives tidal to the UIM stage. This actuation strength fluctuation is real, noticeable, but at the 0.2% level, it doesn't really matter, especially since we're correcting for it. (2) The actuation strength as drifted down in 1 month, from 1.018 to its current value of 1.008. Again, inconsequential, because we correct for it, but I wonder what effect is causing this slow drift. I can think of three things that would evolve on that time scale: vacuum pressure, residual charge, and ambient temperature. The former two won't have changed over the course of the month, but trending further back in time reveals that that was a ~1% increase in drive strength between the end of O3A and the start of O3B, and the strength appears to just be returning to that level. The ambient temperature has changed for the colder over this month... (3) There has been significant step function in the PUM *and* UIM actuation strength, at the ~1% level after the 2019-11-26 the maintenance day break. The only activity that day at the end stations was powering up and leaving on the NCAL, setting up a PEM instrumentation array around it, and Robert making scattered light investigations. I could believe that the former two might have some electrical effect in the strength... but it's shear speculation. I'll be gathering a new set of data this Wednesday to continue the assessment, and confirm that we see this ~1% increase in the sweeps as well. I suspect it will.
In the above entry, I confusingly say
I've trended the [... TDCFs ...] to see if systematic error between [ ... the model of the actuation strength ...] and [the fit to this new data] is small (i.e. the percent difference is unity).
I meant either
- "the percent difference is close to zero" where percent difference is 100*[({new data fit} - model) / model] or
- "the ratio is unity," where the ratio is {new data fit} / model
and in my haste convolved the two.
And then, I totally botched the reporting of the percent difference, adding the *label* for %, without actually multiplying by 100.
For example, for the TST stage TDCF, the percent difference is
100 * (1.01124 - 1.012) / 1.012 = -0.075, and thus "0.075%"
and the ratio is just
1.01124 / 1.012 = 0.9992
Sorry for the confusion! It's too late for me to edit the entry proper, so please forgive the error and interpret with the corresponding grain of salt.