Sheila is driving down to EX to fix.
ETMX SUS screen doesn't seem to have any indicator that something's not OK.
Guardian error message: SUS ETMX saturation.
ER8 Day 6. No restarts reported.
ER8 day 5. No restarts reported.
8/23 OWL Shift: 7:00-15:00UTC (00:00-8:00PDT), all times posted in UTC
Summary:
Handed an H1 in NOMINAL_LOW_NOISE, but it was marked for Commissioning, so Jim took it to Undisturbed (6:56utc). But then after he mentioned the temporary fix for the hobbling PRM, decided to take it out of Undisturbed (7:10utc with 50Mpc range) to run Evan's python script. This took us up 10Mpc, and went back to Undisturbed at (7:19utc) with a range of 60Mpc.
Shift Activities:
PRM NOTE: If H1 drops out of lock, one will want to see if it acuires without PRM's LL. If it does not, then one will want to revert the matrix for PRM (see Evan's aLog or python script for matrix elements). The script is currently saved on the Desktop of the Ops Workstation and can be run in a terminal (it takes about 1 min to zero out the LL elements).
Since we had some empty space on the nuc wall monitors, I put up a few items: (I assume once we can run DMT Viewer on these monitors we'll put seismic plots on them.)
Another fairly quiet night. IFO was locked when I came in, Evan was poking around and LLO wasn't up, so we mostly stayed in commissioning.
At about 4:30 UTC Evan ran a script circumvent a busted OSEM on PRM, which jumped our range 10 Mpc.
~5:00 UTC a 5.2 earthquake in El Salvador knock us out for a about an hour.
Smooth sailing since.
Jim, Evan
We have grown tired of the glitching in the PRM M3 LL OSEM, so here is a script that ramps it off in full lock. It gets rid of the glitching and allows us to recover 60ish Mpc range.
Also included is a screenshot of the usual Euler/OSEM matrix for PRM.
From detchar, here are some glitchgrams to show just how well this works. The PRM M3 LL OSEM was ramped off at 3:43 UTC, and again at 7:13 UTC in a different lock (times gotten by check EUL2OSEM matrix elements). Two glitchgrams are attached which shows that the excess glitchiness goes away as soon as the LL quadrant is disabled. This is fantastic because these are one of our top most worrisome glitch classes from ER7.
Hey @DetChar, can you make a glitch-gram of the H1:SUS-PRM_M3_NOISEMON_LL_DQ? Evan's gunna make a spectragram to see if it contains the same frequency content as the glitch grams (of DARM and the one you'll make). This "on/off" test of PRM M3 LL, at least shows that the frequency content of the glitching is below 50 [Hz]; if the content is similar in spectragram, we can use that -- a spectragram is much easier to make on the floor and/or at least here on site while the channel is being investigated. At this point, the entire drive chain is suspect, and we're not really sure where to start. I worry that starting without a more pointed target, it means we'll be looking for hours, slamming a sledge hammer blindly everywhere, and only come up with more questions. For example, as you know, these NoiseMons can be tricky. This particular PRM M3 LL NoiseMon has passed what tests that have been done on it (see LHO aLOG 17890), but the test is only a "which one of these doesn't look like the other" kind of test, not anything concrete.
Jeff and I looked at a time series trend of the LL noisemon when the interferometer was not locked, in order to give a baseline for diagnostics.
During a quiet time, it seems the peak-peak of the noisemon is about 30 counts, which [accounting for the ADC gain (216 ct / 40 V)] is something like 20 mV pp.
During a noisy time, the peak-peak can go as high as 100 counts, which is something like 60 mV pp.
@Jeff - A glitchgram would not be terribly enlightening. Normalized spectrograms actually show these glitches very clearly, and even standard spectrograms are fine. These glitches only show up in DARM to about 70 Hz, but they're in PRCL up to 150 Hz (first plot). They're getting fed back to PRM, among other things, so all four quadrants' drive signals look like PRCL. The second plot is the normalized spectrogram of LL MASTER, and it's the same as PRCL. There's also something near Nyquist in the plot, but I think it's just spectral leakage in the spectrogram. The characteristic of the LL noisemon (third plot), in contrast to the other noisemon, is that the glitches go up to 1 kHz. They happen at the same time as the glitches in MASTER, so below 150 Hz this doesn't tell us anything. But the higher-frequency content indicates that something before the noisemon is creating excess noise. And since the excess noise goes away as soon as the LL drive is zeroed, it's not just a problem in the noisemon. The noisemon stops showing any glitches once the drive is zeroed, which may be a useful clue. Is it possible to drive a single line in MASTER and see what the noisemon shows? The first three plots were all normalized spectrograms. The last two are standard spectrograms to show that these glitches do show up there. I used 0.25 sec FFTs with overlap of 0.9.
I took a look at the integrated DARM 0-2000 Hz spectrum from 50 hours of 30-minute FScans SFTs taken over the last few days. A couple of sample plots are shown below, and a more extensive set of plots is attached in a zip file. The alphabetic labels on narrow lines conform to those defined in this earlier pre-ER7 report. Highlights and lowlights: * The 0.1698-Hz comb seen before is gone. * The 3.9994-Hz comb seen before is gone. * The 36.9725-Hz comb seen before is gone. * The 16-Hz comb seen in ER7 is still prevalent throughout the spectrum. The 64-Hz harmonics are no longer marked separately, since they don't seem as special as they once did. * There is a new 1-Hz comb with a 0.5-Hz offset, that becomes visible at about 16.5 Hz and peters out around 69.5 Hz (for this integration time). This comb seems likely connected to there being strong digital lines at 0.5 Hz and 1 Hz. There are some new single lines marked here and there (with 'x'), plus some new calibration lines. I have not removed singles marked previously, in case they reappear with deeper integrations to be done after more ER8 data is available. Figure 1 - 0-2000 Hz Figure 2 - 20-100 Hz (note the new 1-Hz comb marked with 'O' for 'One'
We ran the coherence tool on the first week of ER8 data at Hanford, and calculated the coherence between h(t) and numerous auxiliary channel. For the 1 mHz resolution, the results are here: https://ldas-jobs.ligo-wa.caltech.edu/~eric.coughlin/ER7/LineSearch/H1_COH_1123891217_1124582417_SHORT_1_webpage/ We are looking into some of the lines that Keith Riles observed and summarized here https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=20790 Specifically, we concentrated on "There is a new 1-Hz comb with a 0.5-Hz offset, that becomes visible at about 16.5 Hz and peters out around 69.5 Hz (for this integration time). This comb seems likely connected to there being strong digital lines at 0.5 Hz and 1 Hz." Here are the channels where we see this structure in the coherence, and a few observations as well for some of the channels H1:SUS-ITMY_L2_NOISEMON_UR_DQ_data There is a 16.5 Hz line here, and others off by 0.5 Hz at low frequencies. H1:SUS-ITMY_L2_NOISEMON_UL_DQ H1:SUS-ITMY_L2_NOISEMON_LR_DQ H1:SUS-ITMY_L2_MASTER_OUT_UR_DQ H1:SUS-ITMY_L2_MASTER_OUT_UL_DQ H1:SUS-ITMY_L2_MASTER_OUT_LR_DQ H1:SUS-ITMY_L2_MASTER_OUT_LL_DQ H1:SUS-ITMX_L2_NOISEMON_UR_DQ H1:SUS-ITMX_L2_NOISEMON_UL_DQ H1:SUS-ITMX_L2_NOISEMON_LL_DQ Coherence at 10.0, 12.0, 12.5, 14.0, 15.5, 18.0 (big), 21.0, 24.0 30.0 Hz etc H1:SUS-ITMX_L2_MASTER_OUT_UR_DQ H1:SUS-ITMX_L2_MASTER_OUT_UL_DQ H1:SUS-ITMX_L2_MASTER_OUT_LR_DQ H1:SUS-ITMX_L2_MASTER_OUT_LL_DQ H1:SUS-ETMY_L3_MASTER_OUT_UR_DQ H1:SUS-ETMY_L3_MASTER_OUT_UL_DQ H1:SUS-ETMY_L3_MASTER_OUT_LR_DQ H1:SUS-ETMY_L3_MASTER_OUT_LL_DQ H1:SUS-ETMY_L2_NOISEMON_UR_DQ H1:SUS-ETMY_L2_NOISEMON_UL_DQ H1:SUS-ETMY_L2_MASTER_OUT_UR_DQ H1:SUS-ETMY_L2_MASTER_OUT_UL_DQ H1:SUS-ETMY_L2_MASTER_OUT_LR_DQ H1:SUS-ETMY_L2_MASTER_OUT_LL_DQ H1:SUS-ETMX_L3_MASTER_OUT_UR_DQ H1:SUS-ETMX_L3_MASTER_OUT_UL_DQ H1:SUS-ETMX_L3_MASTER_OUT_LR_DQ H1:SUS-ETMX_L3_MASTER_OUT_LL_DQ H1:SUS-ETMX_L2_NOISEMON_UR_DQ H1:SUS-ETMX_L2_NOISEMON_UL_DQ5 H1:SUS-ETMX_L2_NOISEMON_LR_DQ H1:SUS-ETMX_L2_NOISEMON_LL_DQ H1:SUS-ETMX_L2_MASTER_OUT_UR_DQ H1:SUS-ETMX_L2_MASTER_OUT_UL_DQ H1:SUS-ETMX_L2_MASTER_OUT_LR_DQ H1:SUS-ETMX_L2_MASTER_OUT_LL_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_Z_DQ 16.5 Hz is here. And more 0.5 Hz harmonics after. Not super strong, but they are there. H1:PEM-CS_TILT_LVEA_VERTEX_Y_DQ Not clear for this channel. A real mess of lines at low frequencies. 25.5 Hz is distinct. Also 12.5, 26.5, 30.5, 32.5 Hz too. H1:PEM-CS_TILT_LVEA_VERTEX_X_DQ Also a mess at 16.5 Hz, but the line is there. There are lines at 12.5, 14.5, 32.0, 40.0 Hz H1:PEM-CS_MAG_LVEA_OUTPUTOPTICS_X_DQ Starts at 14.5 Hz, and the lines then appear consistently every 0.5Hz. H1:PEM-CS_MAG_LVEA_OUTPUTOPTICS_QUAD_SUM_DQ H1:PEM-CS_MAG_EBAY_SUSRACK_Z_DQ Seems to start at 14.5 Hz, and then this 0.5 Hz harmonic continues to appear noticeably and consistently. H1:PEM-CS_MAG_EBAY_SUSRACK_Y_DQ 6.0, 7.0, 7.5, 8.0, 8.5, 12.0 Hz, then some small but observable coherences continue at 15.0, 15.5, 16.0, 16.5 Hz. 18.0 Hz and 21.0, 21.5, 22.5, 23.5, 24.0, 30.0, 35.0 Hz are big again. H1:PEM-CS_MAG_EBAY_SUSRACK_X_DQ Some example coherence plots are given. Nelson, Michael Coughlin, Eric Coughlin, Pat Meyers
Since LLO had been down in this afternoon, I took this oportunity to renew the dtt templates that are for measuring the transfer function of each stage of ETMY. I tuned the envelop parameters while the interferometer was locked at 23 W in the NOMINAL_LOW_NOISE state. The attached are the resultant templates. The frequency range is newly adjusted to [10 200] Hz with 21 frequency data points as planned. Each one takes several minutes to complete the measurement. According to the results I got, we can get a quite good coherence for all the relevant stages (i.e. L1, L2 and L3 stages) at all the frequency points with a coherence of more than 0.99. On the other hand, the template for measuring the DARM closed loop in this particular frequency band may need a bit more tuning because the coherence was not as great as the ones for the ETMY transfer functions.
These measurements have been committed to the CalSVN repo, under /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-22/ 2015-08-22_H1SUSETMY_L1toDARM_FullLock.xml 2015-08-22_H1SUSETMY_L2toDARM_FullLock.xml 2015-08-22_H1SUSETMY_L3toDARM_LVLN_LPON_FullLock.xml
Daniel, Dan, Evan
We took another measurement of 45 MHz oscillator frequency noise into DARM.
The conclusions are as follows: (1) like the previous measurement, the transfer function is flat in terms of frequency noise coupling into DCPD current, but (2) the coupling is about 7×10−7 mA/Hz, which is 5 to 6 times lower than the previous measurement. Assuming a flat 45 MHz phase noise of 4×10−7 rad/Hz1/2, we expect a contribution in the DCPD sum that is 3×10−10 Hz/Hz1/2 and rising like f, which is not enough to explain the excess we see (and isn't the right shape).
In the previous measurement, we had some trouble with the calibration (namely, the voltage-to-frequency coefficient on the IFR front panel didn't match what we measured with a mixer), and also we were shaking the 9 MHz and 45 MHz sideband phases simultaneously. So I am more inclined to believe this measurement.
The attachments show the calibrated transfer function (both in terms of phase noise and frequency noise).
Locked about 5 hours into day shift - a continuation from owl shift.
LLO down and commissioners arrive, so went to commissioning mode.
Excitation broke the lock.
IFO relocked with only a tweak to the BS alignment for DRMI.
Commissioning continues, but this lock does not have an extra stage of whitening on the OMC - see note below.
---
OMC_Lock has a new stage ADD_WHITENING
This is added after the IFO reaches Nominal_Low_Noise, BUT
H1:OMC-DCPD_A_IN1 and H1:OMC-DCPD_B_IN1 need to be less than 3000 counts peak to peak, or adding the extra whitening will saturate... and bad will happen like lock loss or increased noise
There was a DMT glitch around 2015-08-22 21:00:00 Z. Our previous range channel disappeared, and the seismic FOMs dropped out for 10 minutes or so.
As Cheryl says, in order to turn on an extra stage of whitening, OMC-DCPD_A_IN1 and OMC-DCPD_B_IN1 must each have an ac peak-to-peak less than 3000 ct. Otherwise, turning on more whitening will saturate the ADCs. If the peak-to-peak is more than 3000 ct, probably it is due to some violin mode(s), which must first be damped.
Probably it is a good idea to wait until nominal low noise to activate extra whitening.
Commissioners arrived and LLO is down, so took IFO out of Observing/Undisturbed Mode for commissioning work.
IFO has been in lock 10+ hours.
H1 has been locked for 10 hours as of 20:00 UTC with a steady range of a bit more than 60. The detector has experienced one or two momentary disruptions per hour over the last seven hours. Since 17:00 when I arrived, the voice inside has indicated ETMY when these have occurred. Cheryl had initiated an oplev strip tool earlier this morning; the H1:SUS-ETMY_L3_OPLEV_SUM_OUTPUT trace has moved higher throughout the last couple of hours. SDF: OMC Table (5) and Calcs Table (29) are red. Vacuum: Several LX and LY ion pump status readouts are linking red. And CP-1 is reading 23. Dust: EX dust is alarming white.
8/22 OWL Shift: 7:00-15:00UTC (00:00-8:00PDT), all times posted in UTC
Arrived to Commissioning on a locked H1, and L1 down for last 12+hrs. Environemental conditions were nominal with wind dipping below 20mph & all seismic bands also quiet.
Want to reiterate that for ER8, we are deeming 1pm - 10pm (with L1 down) as the Commissioning window. And I'd say if you need a measurement done during the 1-10pm window and H1 is locked, break lock in H1 and do your measurement if L1 is already down.
Additionally, before Evan, Dan, Stefan left, I made sure to have them clean up any differences on SDF. Most were cleared up except for the OMC.
After the Commissioning work, H1 has been locked the rest of the shift with nice range ~65Mpc & a few ETMy glitches dropping range. DARM looks nice. At times we seem to run below the reference from 10-20Hz. There is also a bump at about 330Hz which is above reference. H1 was much less glitchy for this OWL shift vs yesterday.
Shift Activities:
SUMMARY: Arrived to an H1 locked and in Commissioning, and L1 down for last 12+hrs. There was some commissioning work and then H1 was taken to Observing Mode at 65Mpc. Environemental conditions were nominal with wind dipping below 20mph & all seismic bands also quiet.
Commissioning Activity
Had a discussion about WP5442 with commissioners and since L1 was down and there was a measurement which needed to be done before O1, gave them a few hours to run a PLL measurement (they had it from roughly 7-10UTC).
H1 Back To Observing
Once they were done and H1 was back up to Low Noise, a roll mode was noticed aroudn ~41Hz (Dan/Evan mentioned it's a triple). It rung down after about 10min, but it was huge at the onset.
While checking items before going to Observing Mode, noticed a CFC bit for H1OMC on the CDS Overview. This was due to Evan & a diagnostic for LSC CARM (which we don't use). I hit Load Coefficients to clear this bit.
At this point we were hovering at about 55Mpc. Since we had low violin modes, transitioned from READY_FOR_HANDOFF to ADD_WHITENING on the OMC_LOCK guardian (and then went right back to READY_FOR_HANDOFF). This took the range up to 65Mpc.
The range then went to a cool 65Mpc and the DARM spectrum looked very nice (compared to the previous night)--spectrum was very close/better than the reference.
SDF Overview had some DIFFERENCES, but Evan/Dan cleared most of them. The only differences left were for:
OMC (5 diffs)
CALCS (29 diffs)
According to Mark Barton's Mathematica model, 40.4 Hz is the HSTS roll mode.
Jenne, Dan, control room people
Attached are some additional plots of the glitches that Sheila posted about earlier today. These are the same 'DRMI' glitches which we observed back in ER7 -- or, at least, they are similar enough that I can't tell the difference.
In our most recent lock stretches, started around 0400 UTC, the glitches have gone away. The mystery continues.
The excess noise is loudest in POP_A_RF9_I_ERR, which is what we use for PRCL in low noise. We also see noise in POPA_45_I_ERR (used for SRCL), and not so much in the Q-phases. See the first plot. So, they're not an issue with the PD or the demod electronics. It seems like real length noise, except the noise is very flat. It's hard to imagine a glitchy actuator making flat noise in the PRC and SRC lengths.
FWIW the noise is also visible in POPAIR, with the same pattern as POPA. We also see the excess noise in REFLA_45_I and Q (second plot).
Since these glitches appeared in ER7 we have implemented the offloading of the PRM and SRM M3 stages, so the drives to the last stage are mostly around zero. We looked for any correlation with zero-crossings in the SRM and PRM M3 coils (third and fourth plots). No smoking gun.
Jenne, Dan
Maybe the PRM M3 coil driver is bad? Or something else in the analog electronics...
The gliches in the DRMI signals are correlated with bursts of noise in the PRM M3 NOISEMONs. Compare the first plot (a glitching time) with the second plot (a quiet time).
This isn't surprising - if the PRM drive is compensating for a burst of noise in the PRCL error signal, we'd expect to see something in the coils. But the NOISEMON readbacks for PRM M3 are different for each coil, and LL is the worst offender at the glitchy times. See the third plot. This plot includes a comparison with the SRM M3 NOISEMON, which also shows a difference between glitchy and quiet times, but the coil NOISEMONs agree.
The variation between the coils is certainly due to something in the analog electronics. In the fourth plot we show the PRM M3 MASTER_OUT signals. The drive to the optic from the digital side is the same for all the coils.
It could be that the PRM M3 NOISEMONs are just wrong, but the correlation with the glitches is suspicious.
Not sure if this is enough to justify swapping the coil driver.
The fifth plot is a time-series of the PRM NOISEMONs at the time of a glitch. The y-axis has been scaled the same for all channels to highlight the large value of the LL readback. Note that the LR channel has two dropouts, this happens frequently (there are LR dropouts in both of the first two plots) but is not correlated to the DRMI glitches.
Agreed that the PRM M3 LL driver looks like it's broken. The excess noise in all of the ISC channels is a shelf up to about 70 Hz, and that's what is being sent out to the PRM drive. But in just the LL noisemon, the noise bursts actually go up past 1000 Hz. In UR and UL, the noisemons look just like the drive signal. The attached PDF lines up the two spectrograms to show this. Unless the noisemon is very nonlinear, LL is generating excess noise. Is there a transfer function somewhere of PRM M3 to different ISC degrees of freedom that would tell us whether this could be causing all the noise we see? The second page shows that the noise from LL is probably not major-carry glitches. They don't line up with zero crossings in the drive. The third page shows that the LR noisemon is broken. It looks like it's got ADC overflows or some other kind of saturation. I'll try to check it for ADC overflows.
For completeness, I checked if the glitches show up in the ASC signals that are sensitive to the PRC alignment. They do: the plot attached shows that during glitch times (dashed references) the REFL WFS have excess noise in the same band as the LSC sensors.
So, it's not pure length motion -- consistent with the picture of a single glitchy quadrant on PRM M3.
Dan and Jenne were wondering if this happened in ER7. I took a time when we saw similar mysterious DRMI glitches (June 7 19 UTC) and made some spectra and spectrograms. It looks like the same thing happening, but less severely. The noisemon in UR has the same shape as the drive signal, but LL clearly has a different shape and more excess noise at high frequency. Maybe some other Detcharians can look into this in more detail.
We were locked most of the day until ~21:00, out of lock for around an hour, and came back up relatively easily ~22:00. Another successful PRMI-->DRMI transition trick seemed to help expedite the process.
Although I failed to note specific times, Kyle and Gerardo were working at Y-2-8 most of the day, and Jodi visited the mid stations for property tagging work. Neither of these seemed to effect the IFO performance noticably.
Winds have been in the 20-30mph range for the past 4 hours, with a few gusts to ~40mph.
What's the PRMI -> DRMI transition trick?
Using the calibrated DRMI channels created and described by Kiwamu in entry 18742, I grabbed data from the lock of August 3, 2015, starting at 04:20:00 UTC. The attached 4 page PDF shows spectra of the open loop and residual displacement noise for SRCL, MICH and PRCL. The 4th page shows the coherence of SRCL with the other 2 degrees of freedom.
| Degree of freedom | Residual rms | Shot noise level |
|---|---|---|
| SRCL | 8 pm | 1.3 x 10-15 m/rtHz |
| MICH | 3 pm | 1.5 x 10-16 m/rtHz |
| PRCL | 0.8 pm | 4 x 10-17 m/rtHz |
The SRCL spectra has a curious shape: it comes down quickly with frequency to 10 Hz, then is fairly flat from 10 Hz to 50 Hz, then falls by a factor of 5 or so to the (presumed) shot noise level that is reached above 100 Hz. What is this noise shelf between 10 Hz and 80 Hz? That is exactly the region where the SRCL noise coupling to DARM is troublesome. Our usual approach is to send a SRCL correction path to DARM to reduce the coupling, but this spectrum shows that there should also be some gain to be had by reducing this noise shelf.
The last page of the PDF shows the coherence between SRCL and MICH & PRCL, and it indicates that the SRCL noise shelf could be coupling from PRCL noise -- the coherence is fairly high in this band, though not unity. This suggests that the DRMI signals could use some of the demodulator phase and input matrix tuning that Rana has recently done on L1, reported in LLO log entry 19540.
To complete this log entry, it would be useful if someone at LHO could add the open loop transfer functions for each loop (models), and other pertinent info such as DC photocurrents for these detectors and the input matrix coefficients.
The shelf appears to be gain peaking in SRCL. We have an 80 LPF to get rid of SRCL control noise in the bucket, but it makes the control noise from 30 to 60 Hz a bit worse.
I had the filter off between 2015-08-23 00:36:00 Z and 00:39:00 Z. The attachment shows the error and control signals with filter off (dashed) versus filter on (solid).
Evan, is the shelf in Peter's open loop spectrum there because of OLTF model without LPF? Otherwise, we still need to investigate.
