david.barker@LIGO.ORG - posted 14:17, Thursday 10 September 2015 (21377)
CDS model and DAQ restart report, Wednesday 9th September 2015
ER8 Day 24. No restarts reported.
ER8 Day 24. No restarts reported.
We are currently not locked while a few measurements are being run and while there is hammer drilling going on on the beam tube enclosures. The plan is to proceed back to locking in a few hours.
The 3 SSD drives that failed yesterday for the Raid on h1tw0 had been replaced. Jim will work on mount process for minute trend.
Sheila, Evan
Since we have recently gotten rid of the excess oscillator AM noise in DARM, we wanted to look at the residual of the DCPD sum and null. This test has been done at LLO several times (most recently here).
Unlike the previous look at this noise, we decided to follow the LLO technique and notch the DARM loop by 20 dB between 100 Hz and 700 Hz. In order to do this, we had to reduce the DARM ugf to 20 Hz or so. Unfortunately, this rang up the quad bounce modes. Rather than recommission the bounce mode damping, we just turned it off and collected the sum/null data in a few 20-minute intervals, and then applied damping inbetween using the usual DARM loop shape. In total we collected a little over an hour's worth of data, with the usual 20 mA of current on the sum. The attached plot shows the data processed into 1-second ASDs.
The dc current of the sum was 20.0 mA, meaning we expect a shot noise of 8.00×10−8 mA/Hz1/2. On the other hand, the ratio of sum/shot and null/shot (attached) indicates that we might be too low by a few percent. It's a bit complicated, since it relies on a combination of the frontend calibration (Koji's preamp and whitening compensation filters) and a post facto calibration (Kiwamu's combination of the IOP inversion, the supernyquist preamp poles, the AA response, and the mystery 10.7 kHz pole).
Why is the coherence between A and B not one at the jitter peaks (300-400 Hz)?
You band-stopped the DARM loop in that region, so there should be no correlation induced by DARM. So I would expect that the coherence is one, if the peaks are coming through intensity noise on the laser beam (possibly converted from jitter by the OMC). Instead there is low coherence, which seems to me not compatible with the shot noise level: this is visible in Evan's plot too, since the uncorrelated noise is closer to the value of the sum than to the null...
Two hypothesis
I think the numbers hang together. Let SAA = S00 + SA'A' and SBB = S00 + SB'B', where S00 is the PSD of the correlated component of the noise between the DCPDs, and SA'A' and SB'B' are the PSDs of the uncorrelated noises on each PD.
From last night's data, at 315 Hz we have SAA = SBB = (1.14×10−7 mA/Hz1/2)2. Looking nearby at 540 Hz, where the DCPDs seem to be shot-noise dominated, we can estimate SA'A' = SB'B' = (0.577×10−7 mA/Hz1/2)2, so therefore S00 = (0.983×10−7 mA/Hz1/2)2.
For the CSD, we have SAB = S00.
Therefore, the coherence we expect is γ2 = |SAB|2/(SAA×SBB) = 0.55.
Or in terms of the PSDs of sum (S++) and null (S−−), the coherence should be γ2 = [(S++ − S−−)/(S++ + S−−)]2.
I agree with your computations. But why is your projection of the residual showing the peaks? it should line up with the shot noise if that's what is limiting the coherence.
New ion pump OK
As a periodic evaluation of condition, we'll be trending the Pump Station Pressures to watch for changes in the Pump Performance.
The attached plots show the pressure trends from the pump stations at the three buildings for the past 30 days.
There are no obvious steps indicating a pump stating cavitation--I don't really expect this out of the blue. But if the system is taken down (Pump spun down) for any reason and then brought up, that is when cavitation has occurred and the pump pressures change. This will be obvious for the corner station as the other pumps will compensate for the cavitating pump. Hmmm, if this happens at an end station though...I expect the VFD output will have to increase to make up for this if if can. I've only seen this at the corner so... Anyway, I include the VFD output for the ends to look for this.
Observations:
Corner Station: Bizarrely, the first pressure out of the Pumps all show almost perfectly flat tends--look at the scales; I can't explain this. All the other pressures show a somewhat regular glitch up; this may be dailyish but seem a bit irregular.
EndY: Nothing new here. Pressure and VFD output steady. The long documented daily and weekly glitches still present and the daily temperature swings evident on the VOUT.
EndX: Same story except there is no regular glitches and the building must insulate things from the temperature fluctuation as there is no daily signal in the VOUT.
LLO still down, and Kyle needs to unplug the aux cart down at EX.
While LLO is down, the cleaning crew needs to use the roll up door and head into the garb room breifly. Should be right back in as soon as they're done.
Back to Observing Mode.
After the earth settled back down, I had no trouble relocking the IFO. The OMC issue persists but our range has been stable. I also had to clear SDF again after relocking, but I imagine this will be resolved when the transition to the Observe.snap state is completed.
12:49 locked in Observing Mode.
8:08 locked Low Noise
8:36 set to Observing after clearing a bunch of SDF diffs (Betsy had told me to set them to Not Monitored since they are still in the process of creating the observe.snap). Sheila helped clear some that couldn't be Not Monitored due to the 'too many decimal places' issue.
9:30 I noticed that the OMC had a red CFC block on the CDS overview. The overflow was incrementing rapidly. I tried to reset it anyways, but to no avail. I then tried calling several people, none answered. Finally I called Corey to let him know about the situation. Eventually, I got a couple of calls back suggesting that I let it ride since we were already in Observing. Looking at a few trends, I saw that this began during the locking sequence. Perhaps the CDS overview had a red block in it, but I failed to notice (at least it didn't prevent me from going to Observing).
10:30 lockloss due to 5.9 EQ in Alaska.
During the ~2.5 hours lock stretch, our range has been steadily trending downward, perhaps related to the OMC issue (?).
As far as the overflows are concerned, A1 and A2 overflow in OMC is OK as the only channels in A1 and A2 used by OMC are the ones used for acquisition/transition.
If you look at H1OMC_GDS_TP screen, there are four potential source of overflow, A0, A1, A2 and D0 representing ADC card 0, 1, 2, and DAC card 0. In the attached, you can see that ADC2 is overflowing.
If you press "A2" button, you'll see that the channel overflowing is C_DIFF_PLL_CTRL_INMON (second attachment), and that this channel is the only one in ADC2 that is used by the OMC frontend.
This channel is only used before RF lock and always rails after full lock. It's safe to ignore this in observation mode as the channel is not used in full lock.
Likewise, if A1 overflows in observation mode, that's fine as the channels in ADC1 used by OMC (ASAIR_A_RF45_I and Q) are only used before DC lock.
If A0 or D0 was constantly overflowing, that would have been a concern.
To follow up, here's Kiwamu's email regarding the outstanding OMC filter file that was modified last night causing additional confusion and his clearing of it this morning:
"I loaded the coefficients. The CFC bit is now back to green.
kiwamu"
Upon some closer inspection, I'm not totally convinced that the lockloss at 10:30:46 UTC was due to the earthquake.
Attached is a plot for a single seismometer (located at End-X), but all 5 STS-2s that are mounted on the floor look the same. The top row is x-axis data, the middle row is y-axis and the bottom row is z. The columns from left to right are in increasing order of BLRMS bands. The lockloss is at about 0 seconds (actually -0.9ish), and the plots are 10 minutes of data, centered around the lockloss. We don't see the earthquake until about 150 seconds after the lockloss. This is consistent with the anticipated P-phase arrival time from Terramon - 10:33:12 UTC, but much earlier than the S-phase time (10:38:25 UTC) or the R-phase time (10:43:56 UTC).
I don't know yet what did cause the lockloss, but I don't think it was the Alaskan earthquake.
Summary
EPICS values used for calculation of DARM time-varying parameters (DCC T1500377) in GDS pipeline and with PCALMON data were updated on 10-Sep-2015, 01:03:38 PDT (10-Sep-2015, 08:03:38 UTC) using H1 DARM model for ER8/O1. We will analyze DARM time-varying parameters (sometimes generalized as "kappas") calculated in GDS pipeline in GDS frames from nominal low-noise lock stretches after this change.
Previously values written in these EPICS records were calculated based on ER7 model (see LHO alog 20452), now we used ER8/O1 model with parameter file "H1DARMparams_1124954397.m", the summary of this param file residuals were given in LHO alog 21318. There is an additional phase of 136.7 degrees that was fudged into H1:CAL-CS_TDEP_REF_INVA_CLGRATIO_TST_(REAL|IMAG) that should provide kappa_tst with a reasonable phase value (ideally the phase of the kappa_tst should be 0), this was done so that we can test overall performance of GDS pipeline when calculating kappas.
These are not the final values for ER8/O1, currently H1 DARM OLGTF model is under testing using different DARM OLGTF and PCAL2DARMTF measurements and possibly will go through minor adjustments.
New values were accepted in SDF_OVERVIEW (however some of the values still does show up in "diffs", that's because SDF_OVERVIEW cannot handle very small values).
Details
Earlier we had a hard time getting reasonable values for kappas, in Maddie's GDS code as well as recent Greg Mendell and Sudarshan's Matlab code using PCALMON data (see LHO alog 21102). One of the reasons why calculations wouldn't fully agree with actual parameters was that EPICS records had values calculated from ER7 model, which we know is not up to date anymore. But I belive that even after we got a fairly good approximated ER8 model there's probably something missing in EPICS actuation coefficient for x_{tst} calibration line. We're still investigating the source of this discrepancy.
The script that extracts values needed for calculation of kappas from DARM OLGTF model was committed to calibration SVN:
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/CAL_EPICS/writeER8model_TDEP_EPICS.m
Output files (raw epics values and more verbose log) were committed to the same directory, detailed log is also attached to this alog.
The EPICS values correspond to the model file and the parameter file in calibration SVN at rev. 1374:
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMOLGTFmodel_ER8.m
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMparams_1124954397.m
This morning Elli and I spent some time on phaseing of AS WFS, and again this evening we picked it up again. More details to follow, but here are some highlights:
We can use a consistent phase on all 4 quadrants to get reasonable signals.
In PRMI we can use 40 degrees for all quadrants on ASA36 to get most of the static signal into the I phase, and 90 degrees on AS 36 B. In this situation the Q phase has a good signal for the BS, with the transfer function from BS pit drive (1000 cnts, 12 Hz) to each quadrant having all the right phase relationships.
In DRMI 90 degrees still seems like a good setting for keeping the static signals in the I phase for AS B, while AS A it seems like 20 degrees is a better phase, except for quadrant 4 where it seemed like the phase should be 70. I assume that this could be caused by a misalingment, and should not be a real since all the phases were consistent in PRMI.
We closed the loops on AS B in DRMI, which worked fine, and locked. The phases of the transfer function to each quadrant stayed consistent in DRMI, full lock, and at 10 Watts. I left the excitation on the BS as I powered up, starting around 7:08 UTC sept 10th. Durring this time all the quadrants on AS 36 A had an R phase of 20 degrees, all the R phases for AS B 36 were 90, and the yaw matrix for both wfs was normal (it included all quadrants.) After powering up to 10 Watts we saw the usual runaway where POP90 grows. We can perhaps look at this data to see if our sensing matrix was changing durring some thermal process.
After several minutes at 10 Watts, I tried to recover a good operating point by moving the offset in AS_C, which did not work and broke the lock.
Everything is reverted and we are going back to locking with our old arangement for AS WFS.
One possible cause of the DAC overflows we've been seeing (the ETMY saturation warnings) is something at high frequency using up the range of the drive. We only record the last output to the ESD DAC at 2048 Hz, so we don't see anything happening above a kHz. It's possible some high frequency line or feature bumps up against 2^17 counts and that makes the DAC overflow, which then causes a much larger glitch - once there's an overflow the whole thing will go crazy. To try to investigate this, the simplest thing to do is to record a channel like SUS-ETMY_L3_MASTER_OUT_LL_DQ at the highest rate possible, or something equivalent that monitors the actual signal sent to the ESD DAC. Once an ETMY overflow happens, save the data for several seconds around the time somewhere that people offsite can access it, preferably in a common format like ASCII or frames. Some time without an overflow would be good for comparison. Then we can see if there's something using up the drive range, or if it's just something sudden from somewhere else in the DARM loop. A few instances of overflows would be nice. If this is not possible, there's another tack we can take but it's more involved, so I'd prefer this.
If it's any help, we do record H1:SUS-ETMY_L3_ISCINF_L_IN1_DQ at the full rate (16 kHz). In full lock, this is the only signal (other than the calibration line) that is sent to the ESD. With an adequate knowledge of the filters between this channel and the ESD master outs, it should (in theory) be possible to reconstruct the master out signals at the full rate. That was the original motivation for recording this channel at full rate, anyway.
Yes, using ISCINF was the other tack that I meant. I checked the MEDM screen when I knew the detector was locked. Here's what I see in the chain: ETMY_L3_ISCINF_L - seems to be empty ETMY_L3_LOCK_L - FM1,3,5,6,8,9,10; gain is 1.25 ETMY_L3_DRIVEALIGN - L2L is only thing used, just a gain of -30 ETMY_L3_EULER2ESD matrix - factor of 0.25 to each OSEM ESD linearization - bypassed, so I think we can ignore it ETMY_L3_ESDOUTF_LL - FM6,7; gain is 1. (I think the four coils get nearly the same drive, so any quadrant is fine). The needed filters are available from the web SVN in this text file. So there's only two filter banks to apply, and a few gains. I think this is doable, though it seems like it would just be easier to record SUS-ETMY_L3_MASTER_OUT_LL_DQ at 16384 Hz for a few hours of lock. Maybe one or two of the LSC Fellows could take on this more complicated method.
I managed to capture the full data for SUS-ETMY_L3_MASTER_OUT_LL around the time of the most recent glitch. I put it on my account on caltech at /home/jordan.palamos/detchar/ETMY_glitch.txt and also at https://lhocds.ligo-wa.caltech.edu/exports/jordan.palamos/.
The start time is 1125946072.8
Dan, Daniel, Evan
The addition of a 9.1 MHz bandpass on the OCXO output has removed the broadband excess noise between DCPD sum and null. The dashed lines in the figure show the sum and the null as they were three days ago (2015-08-31 7:00:00 Z), while the solid lines show the sum and the null after the filter was inserted.
Since at least June (probably longer), we've had a broadband excess noise between the sum and null DCPD streams. Stefan et al. identified this as 45.5 MHz oscillator noise a few weeks ago (20182).
In parallel, we switched the 9 MHz generation from an IFR to the OCXO (19648), and we installed Daniel's RFAM driver / active suppression circuit (20392), but the excess noise remained (20403). For a while we suspected that this was 45.5 MHz phase noise (and hence not supressed by the RFAM stabilization), but the shape and magnitude of the oscillator phase noise coupling (20783) were not enough to explain the observed noise in the DCPDs, under the assumption that the OCXO phase noise is flat at high frequencies (20582). For that matter, the shape and magnitude of the oscillator amplitude noise coupling were also not enough to explain the observed noise in the DCPDs, assuming a linear coupling from the RFAM (as sensed by the EOM driver's OOL detector) (20559).
Daniel et al. looked at the 45.5 MHz spectrum directly out of the harmonic generator in CER, and found that most of the noise is actually offset from the 45.5 MHz carrier by 1 MHz or so (20930), which is above the bandwidth of the RFAM suppression circuit. This suggested that the noise we were seeing in the DCPDs could be downconvered from several megahertz into the audio band.
Yesterday there was a flurry of work by Keita, Fil, Rich, et al. to find the source of this excess noise on the 45.5 MHz (21094 et seq.). Eventually we found circumstantial evidence that this excess noise was caused by baseband noise out of the 9.1 MHz OCXO.
Tonight we installed a 9.1 MHz bandpass filter on the OCXO output. This has removed the huge 1 MHz sidebands on the 45.5 MHz signal, and it also seems to have greatly lowered the coherence between DCPD A and DCPD B above a few hundred hertz.
The chain from OCXO to filter to distribution amplifier currently involves some BNC, since we could not find the right combination of threaded connectors to connect the filter to the amplifier. This should be rectified.
Also, it appears that our sum is lower than our null in a few places (400 Hz in particular), which deserves some investigation.
"NULL>SUM" problem is just DARM loop. You're talking about 10%-ish difference, and DARM OLTF gain is still 0.1-0.2 at 400Hz.
See attached.
I don't know how to obtain official DARM OLTF model, so I just took 2015-08-29_H1_DARM_OLGTF_7to1200Hz_tf.txt in
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/DARMOLGTFs/
The coherence for this OLTF measurement was much larger than 0.95 for the entire frequency range shown on the plot.
On the bottom is |1+OLTF|. I interpolated this to the frequency spacing of SUM and NULL spectra, and plotted SUM*|1+OLTF|, SUM, and NULL at the top.
Note that DARM OLTF template measures -1*OLTF.
Nice work. After O1 we can figure things out now you have narrowed it down.
Nice work!
Great job! Following up on the discussion during the commissioning meeting today, at LLO Evan's equivalent plot of the coherence between the two OMC PDs is already below 10^-3 (below 3 kHz).
Fil and I replaced the BNC cable with an SMA/N cable.
