All day next Tuesday will be scheduled for maintenance Model changes planned for next Tuesday Sheila and Richard will install a low pass filter box between the IO piezo controller and periscope PCAL was calibrated at end X yesterday PCAL may have a bad AOM at end X There is a request to label the analog camera images
model restarts logged for Thu 12/Feb/2015
2015_02_12 00:56 h1fw0
2015_02_12 15:33 h1fw1
2015_02_12 17:14 h1broadcast0
two unexpected restarts (first h1fw0 unexpected restart since 11 days ago). Broadcaster reconfigure for DMT channel addition. Conlog frequently changing channels report attached.
Dan, Jeff, Sheila
Tonight we had some more fun noise hunting. Turning the power up to 8 Watts was easy, but didn't improve the noise so much since we are mostly not shot noise limited.
This gave us a lower noise floor, we sat here from 5:28:10 UTC Feb 13th to 5:41:29 UTC, with the intent bit undisturbed.
We then made a single destructive attempt to reduce the ESD bias voltage, we'll have to come back to this. :)
On the next lock we decided to try increasing the input power. This was suprisingly easy, we have now been sitting at 8 Watts input power since 7:03:43 UTC. We have been adjusting the BS alignment by hand every 10-15 minutes or at 10 Watts. We have watched the AS36Q WFS signals while we do this, and can see that they are not always correct (zero does not always give us the best AS90). BS yaw dramatically improves the intensity noise coupling. It seems as though the IFO would stay locked like this all night if we sat here and adjusted the BS once in a while. We are leaving it locked and undisturbed.
In the attached plot, the green trace is earlier today, The blue trace is with 1 stage of DCPD whitening on and the ISS second loop on, which is injecting noise at around 700 Hz. The red trace is now, with 8 Watts input power, BS YAW oplev damping gain reduced, and the ISS second loop off. You can see that we are not really shot noise limited, as increasin the power by a factor of 2.8 only gives us a small improvement at high frequencies. Something is clearly wrong with the calibration.
The attached plot illustrates eight hours of progress today. (Times are approximate.)
At 3:30pm we were using the in-vacuum POP photodiode for MICH, PRCL, and SRCL.
At 9:30pm the violin damping ninjas had allowed us to enable one stage of whitening on the DCPDs. The ISS second loop was on for this lock stretch. During this lock the intent bit was set to OBSERVATION for about ten minutes (see times above), and ISC_LOCK Guardian was in the 'DC READOUT' state.
At 11:30pm we had increased the power from 2.8W --> 7.8W. But, we forgot to turn on the ISS second loop. Tweaking the BS alignment reduced the beam jitter coupling at 100-300Hz and around 700Hz. There is a lot of coherence above 1kHz with the LSC DOFs, we believe that we are not dark noise or shot noise limited in this region.
As we increased the power we changed the DARM offset, gain, and OMC scale factor & gain to maintain good phase margin. At 7.8W we could not incerase the DARM offset to provide ~16mA in the carrier mode at the OMC, the DCPDs would saturate from the violin mode. The last lock stretch had a DARM offset of 2e-5 and a DCPD sum of ~12mA.
We noticed the 1/f^3 noise between 30 and 70Hz was breathing on a several-second timescale. The noise here is somewhat coherent with MICH, we wonder if it is also due to ESD DAC noise or DAC zero crossing glitches. We have left the IFO locked in 'DC READOUT' with the intent bit set to observe. DetChar, please examine the data for glitches associated with major-carry transitions.
Glitch investigation: Keep in mind that there are about a dozen IPC erros per second on the controls signal sent to the ETMs.
The interferometer unlocked around 9:25UTC, so it stayed locked for more than 2 hours at 8 W. The plot shows a 14 hour trend from yesterday, with several hours of lock overall. Most of the unlocks were commissioning-related. (P.S.: still very quiet seismic environment).
Shot noise dominates the noise floor in kHz range, there was something wrong about the power scaling and/or scaling with DARM offset.
The first attachment shows that, in calibrated spectrum, the Pcal line in 7.8W (red) was a factor of 1.6 larger than in 2.8W (blue).
There should have been something wrong about power scaling somewhere, and if you want to make a fair comparison between 7.8W and 2.8W, 7.8W should be brought down by a factor of 1.6.
If you take this into account, 7.8W noise floor would be about a factor of 1.7 smaller than 2.8W for, say, f>7000Hz. (The top panel in the second attachment shows that 7.8W data is 5% or so smaller than 2.8W, and 1.6/0.95 = 1.7.)
sqrt(7.8W/2.8W)=1.7.
Further, in the same plot, the bottom and the middle panel shows that both in 7.8W (red, blue) and 2.8W (green, brown) the noise floor of DCPD SUM is very close to the NULL stream except for many structures, and PDB/PDA coherence is very low except these structures. (Also, in the middle pane of the second attachment, the black trace is the dark noise with high Z and 1 stage whitening, which is a factor of 5 smaller than 7.8W noise.)
Therefore we're confident that the noise floor for kHz range is already dominated by the shot noise.
Overall scaling is another problem which should be fixed separately.
I examined the sidebands of the OMC ASC dither lines today. The excess noise around these lines is disappointingly broad, roughly +/-2Hz. There are sidebands around the dithers at +/-0.18, 0.45, 0.49, and 1.5Hz.
The attached figure compares the sidebands of the first dither line (middle plot) to the low-frequency DARM noise (top plot). The bottom plot is the noise around the violin mode.
Sheila, Dan, Elli, Kiwamu, Lisa, Peter, Robert
Here is a calibrated DARM spectrum from today's DC-readout lock, using the new CAL-CS channel described in LHO#16698.
According to GWINC, the BNS inspiral range of this spectrum is 8 Mpc.
The calibration must be wrong, ignore this for now.
H1BROADCAST.ini was modifed to add the signal H1:DAQ-BCST0_CHANS_SAVED to the DMT. The DAQ broadcaster was restarted.
07:00 Karen and Cris in LVEA 08:06 Jeff B. turning off dust monitor 16 08:16 Bubba to LVEA to work on LN2 piping for LTS 08:17 Jeff B. done 08:29 Corey to squeezer bay for 3IFO work 08:35 Rick to end Y to check on Thomas and Sudarshan 08:39 Bubba done 09:12 Jim W. working on SEI BSCs 09:17 Alastair putting 3IFO equipment in cabinets in LVEA 09:47 Corey out of squeezer bay 09:48 Jim W. done 09:54 Robert to IOT2 and PSL enclosure 10:02 Travis back from end Y 10:05 Corey and Andres to LVEA 10:09 Corey back 10:13 Andres back 13:39 Filiberto to end X, not VEA, to read serial numbers and pick up tools 14:18 Filiberto done
Added 889 channels reflecting model changes from yesterday.
Several people reported that the DARM calibrated signals from the OAF model did not look healthy in the past two or three days. Looking at the data in frequency domain, I confirmed that they have behaved funny. Cosulting with Jeff K, we decided to gradually migrate the functionality from OAF to CAL-CS model (alog 16669) because CAL-CS is going to be the official calibration place in future anyways. We are still in the process of moving from OAF to CAL-CS, but a calibrated DARM signal is now available in CAL-CS. The funny behavior seen in OAF is not seen in CAL-CS so far. The calibration accuracy is not yet examined.
(Bad OAF data)
Looking at the data in frequency domain, indeed they looked funny -- the spectral shape of, for example, OAF-CAL_DARM_DQ was suspiciousely featureless with the noise floor around 100 Hz much higher than it should be by a couple of orders of magnitude. Also it did not show a roll-off shape at 5-ish kHz for some reason. A confusing fact is that it sometimes looks behaving correctly and sometimes doesn't. I did not make a further investigation because we decided to move to CAL-CS.
(CAL-CS)
I copied the DARM-related filters that I had in OAF over to CAL-CS. So it is right now completely a duplication of what we had in OAF. I did not move the DRMI, CARM, or IMC filters yet.
Here is a brief summary of how the current calibration was done.
The current one relies on the reponse of the ALS DIFF VCO. Since we knew the VCO response in Hz/V, we were able to calibrate the ALS DIFF sensor into meters using the dx/ L = d(nu)/nu relatation. Then, by knowing the UGF of ALS DIFF, we calibrated the ESD coefficient in meters/counts. The last step is to calibrate the DARM optical gain by measuring the UGF in the final DARM loop.
I checked in the matlab code, which I used for the calibration, into svn. It is Runs/S7/Common/MatlabTools/LSC_DARM_calibration.m in the calibration SVN. For more details, please refere to the code.
H1:CAL-DELTAL_EXTERNAL_DQ is the new calibrated DARM signal. The signal is digitally whitened by five zeros at 1 Hz and five poles at 100 Hz.
The scripts you need are attached here. The main file is bruco.py
When you call the script, there are some parameters that must be set:
# Command line arguments (with default values)
# --ifo=L1 interferometer prefix # --channel=OAF-CAL_YARM_DQ name of the main channel
# --gpsb=1087975458 starting time
# --lenght=180 amount of data to use (in seconds)
# --outfs=8192 sampling frequency of the output results (coherence will be computed up to outfs/2 if possible)
# --minfs=512 skip all channels with samplig frequency smaller than this
# --naver=100 number of averages to compute the coherence
# --dir=bruco_1087975458 output directory
# --top=100 for each frequency, save to cohtab.txt and idxtab.txt this maximum number of coherence channels
# --webtop=20 show this number of coherence channels per frequency, in the web page summary
Here is an example, that I'm running right now with the last night lock:
./bruco.py --channel=LSC-DARM_IN1_DQ --gpsb=1107760396 --lenght=600 --outfs=4096 --naver=300 --dir=/home/gabriele.vajente/public_html/bruco_1107760396 --top=100 --webtop=20 --minfs=32 --ifo=H1
Beware that the processing of all channels takes many hours, when I run it on one machine in the Caltech cluster.
There might be some options to tune in the script, to set where the data is saved. Mainly in lines 106-109. The script is now configured for the Caltech cluster, but I imagine it should be easy to point it to different places where the data is.
As suggested by Dan and Lisa, we might try to speed the script up by running it in parallel with condor or reducing significatly the list of channels. Now, bruco automatically selects 2390 channels.
Elli, Evan
After Peter and Kiwamu's deep insights into the in-vacuum POP signal chain (LHO#16691), we've transitioned control of the DRMI DOFs from POPAIR to POP with DARM controlled on RF.
First I drove a 1000 ct, 89.1 Hz line into the PRM in order to phase POP. I set the demod phase for POP9 to 90°, making the Q signal a factor of 50 smaller than the I signal. For POP45, I set the phase to 66°, making the Q signal 25 times smaller than the I signal.
With the line still on, I looked at the TFs of the POPAIR/POP signals. For 9I, 45Q, and 45I, POPAIR is weaker than POP by a factor of 0.15 ct/ct, with 0° of phase difference. So I took the POPAIR matrix elements, rescaled them by 0.15, put them into the corresponding POP matrix elements, zeroed the POPAIR matrix elements, and then transitioned.
Then Elli and I remeasured the OLTFs of PRCL, MICH, and SRCL. The PRCL gain was a bit low, so we increased it by 10%. The SRCL gain was a bit high, so we decreased it by 20%. Now the UGFs should be roughly the same as for POPAIR.
The new matrix elements are as follows:
This last matrix element is just a rescaling of the matrix element for POPAIR9I → SRCL (which itself is just a rescaling of REFLAIR27I → SRCL), so it should be retuned to optimize the PRCL/SRCL subtraction.
This works fine with DC readout.
Also, it is fine to transition directly from the REFLAIR 3f signals to the POP 1f in-vac signals. This is now written into the guardian.
Hanford tells us that they will not be hauling container loads on Friday 2/13 either on day or swing. Their operations will remain quiet through the end of Monday 2/16.
Thomas updated the SUS Drift Mon to the a time just at the end of last nights 2hr lock (2/12/15 07:35:00 UTC, 1107761716 GPS).
model restarts logged for Wed 11/Feb/2015
2015_02_11 11:28 h1fw1
2015_02_11 12:23 h1susetmy
2015_02_11 12:31 h1fw1
2015_02_11 13:16 h1susetmx
2015_02_11 13:20 h1susitmx
2015_02_11 13:20 h1susitmy
2015_02_11 13:27 h1broadcast0
2015_02_11 13:27 h1dc0
2015_02_11 13:27 h1fw0
2015_02_11 13:27 h1fw1
2015_02_11 13:27 h1nds0
2015_02_11 13:27 h1nds1
2015_02_11 13:37 h1calcs
2015_02_11 13:40 h1calcs
2015_02_11 13:42 h1broadcast0
2015_02_11 13:42 h1dc0
2015_02_11 13:42 h1fw0
2015_02_11 13:42 h1fw1
2015_02_11 13:42 h1nds0
2015_02_11 13:42 h1nds1
2015_02_11 14:27 h1fw1
2015_02_11 16:52 h1calcs
2015_02_11 17:07 h1iopoaf0
2015_02_11 17:07 h1oaf
2015_02_11 17:07 h1pemcs
2015_02_11 17:08 h1iopoaf0
2015_02_11 17:08 h1oaf
2015_02_11 17:08 h1pemcs
2015_02_11 17:09 h1calcs
2015_02_11 17:09 h1odcmaster
2015_02_11 17:09 h1tcscs
2015_02_11 17:11 h1broadcast0
2015_02_11 17:11 h1dc0
2015_02_11 17:11 h1fw0
2015_02_11 17:11 h1fw1
2015_02_11 17:11 h1nds0
2015_02_11 17:11 h1nds1
three unexpected h1fw1 restarts. New QUAD SUS code. Install of the h1calcs model. Several related DAQ restarts. Power cycle of h1oaf0 to remove ADC noise from PEM.
Evan, Dan, Jeff, Peter, Daniel, Lisa2+ hours lock on DC readout, this time for real!
* Feb 12, 5:33 UTC lock on DC readout, engaged ISS loops shortly afterwards * Feb 12, 7:00 UTC improved low frequency noise by turning off ETMs optical lever damping * Feb 12, 7:35 UTC still lock on DC readout, starting alignment tests (intentionally changing BS alignment; also reduced the BS optical lever damping gain by a factor of 10 * Feb 12, 7:48 UTC unlocked by closing the AS beam diverter Positive news of the day - Thanks to the improved bounce mode damping , we can now reliably damp the bounce mode, and it is not a limiting problem anymore; - ETMX and ETMY optical lever PITCH damping loops have been reduced by a factor of 100; it turned out that these loops were responsible for the huge excess of noise that we saw yesterday below 40 Hz, and that was causing the OMC transmission to shake painfully. Now the OMC is quiet and happy (the Guardian has been updated to reflect this change). - We turned off the QPD alignment of the OMC, and replace it with dither alignment; - We turned on the ISS first and second loops: the positive news is that we improved the high frequency noise by a factor of 10. Negative news of the day - While we have improved noise at low and high frequency, we now have a new bump of noise around 100 Hz which was not there yesterday . - Our wonderful BS WFS loops, so effective yesterday, were not working today. However, today the BS alignment was not critical, and we have barely had to touch the BS anyway, but we need to figure out what happened. Right now we commented out the BS WFS in the locking sequence; - Yesterday the locking sequence was very reliable, and worked 5 times in a raw..today it has been less robust, despite similar environmental conditions.. - The RF of in-vac POP is somehow broken - no signal there. It is probably not a big deal right now at low power. - We also closed the OMC REFL beam diverter (level 2 of "Valera's levels of awesome": it works, but it doesn't do anything)
Attached are some plots of coherences and OMC noise.
Fig 1: Coherence between DARM and LSC, ISS - the excess noise at 70-400Hz, much worse than last night (grey), is coherent with the common DOFs (PRCL, SRCL, and CARM [not plotted]). Also the ISS 2nd loop out of loop PD.
Fig 2: DARM and ASC - there is coherence between IMC yaw WFS error signals and DARM, implying that the noise is due to some beam jitter coupling
Fig 3: OMC noise - note that we are shot noise / dark noise limited above ~1kHz
In figure 1, the grey trace is yesterday's almost-two-hour lock, the green is today before the oplev damping was turned downand without ISS second loop, the blue is at the end of the lock. There was a substantial improvement in many bands due to the ISS 1st loop and loosening up the oplevs. The second ISS loop didn't change much, although we did notice it reduced the intensity noise on, for example, ASC-POP_A_SUM.
The beam jitter into the IMC, as measured by DOF1_{P,Y}, hasn't changed from last night. The working hypothesis is that the alignment has drifted into a place where beam jitter couples more strongly into the length DOFs. We checked many PEM channels for coherence with DARM in case, for example, someone had left a fan on in the LVEA, but found no coherence with environmental channels.
A plot of POP vs. POPAIR demodulated signals is attached. Evidently, there is nothing meaningful in the in-vacuum signals, even though there is about 1.7 mW of dc power. This is roughly what we expect; with 2.8 W incident on the IMC, an IMC transmission of 90%, a power-recyling gain of 33 W/W, 250 ppm of transmission through PR2, and 10% transmission through mirror M12 on HAM1, we get 2.1 mW.
Also attached are some short videos of the AS port and the OMC transmission during full lock.
The Box is Checked, and the Advanced LIGO Project has delivered on its prime instrument performance goals. Congratulations to the entire aLIGO team for making this happen! and thanks again to the Hanford All-nighters.
Congratulations and happy noise hunting.
Once again, congratulations from your LLO colleagues! Allow yourselves a moment to enjoy this sweet success before having good luck with the noise hunting…
Fantastic! Noise hunting licenses for everyone! Curious as to what the wind conditions were like.
Dave -- wind conditions were around ~10 [mph]. Microseism was ~5e-1 [m]. Pretty quite night! May we have MANY more. Just like tilt meters.
PeterF, Kiwamu
We found that the RF cables from POP_A were unplugged at the front panel of the demodulation board by the PSL enclosure. We hooked them back in. Confirmed that the demodulated signal showed some interferometer fringes.
J. Kissel ETMX UIM Driver's rocker switch tripped last night at 8:44:30 UTC (Feb 06 2015 00:44:30 PST), which cause all OSEM output on the L1 / UIM stage (and the sensor signals as well) to fail. I've turned on the rocker switch, and the stage appears functional. Other points: ------------- There is no smoking gun for this chassis power outage. There are not enough diagnostic tools in the frames to look at all possible suspects (power surge to the chassis [no power monitors], chassis overheating [no temperature sensors], an inconsequential component on say -- a monitor board -- smoking [only one of the four monitor signals for each channel are stored], too large a current request [the NOISEMON is stored, not the FASTIMON]). Output request of DAC (from any ISC) does not appear to coincide with trip, but difficult to tell with dataviewer. There is an Longitudinal request to drive really hard, but it's unclear whether it's a the cause of or a result of the driver's power being shut off. Output request is roughly 42 [mA] (according to FAST_I_MON) on all four coils leading up to trip, switch trips on 3 [A]. Even the sudden, very large output request does not cause any current larger than the 42 [mA] -- as reported by dataviewer / EPICs. Reqrettably, only the severely-high-passed-at-5-[Hz] noise monitor signals are stored in the frames, but I attach the time series of one of the coils anyways to indicate the exact time. It shows a nice smooth ramp down at the time of the chassis power down. Serial number of this box is S0900303. I attach pictures of the box after I'd turned it back on. When I first approached the box, the front "power" LED lights were OFF, and the rocker switch was in the opposite (powered off) position. I'm not sure if there is good way to make this power shut-down control-room visible. I did notice that all the coil driver monitor signals were frozen at -16 [ct], and that the OSEM sensor speed dials were zero for this stage and this stage alone, which is what immediately clued me in that it was a flaw with the entire coil-driver chassis or satellite amplifier. This last happened (according to collective analog CDS crew's memory) on H1 SUS ITMX UIM driver, some time ago, 19 November 2013. See LHO aLOG 8635.
What's going on here?
Adding more to Daniel's comment -- the problem in the screen shot he shows is ONLY a problem with the reported decimation filter output for the ETMX L1 LOCK L bank. He had repeated the same test on the ETMX L1 LOCK P bank, and saw no difference between the OUTPUT and OUT16. It's not an issue progressing forward because no one and nothing uses the decimation channel but for diagnostics, but it's this kinda of stuff that makes users immediately suspicious of the entire front end code as a whole (regardless of how serious the problem actually is) and demand "REBOOT! REBOOT!" without really identifying the specific problem. Again, all signs point to that this is UNRELATED to the coil driver switching off from a current overload. This being said -- I agree the decimation display problem with *this* filter bank *is* weird, and we'll try to debug further.
I have seen this too. Good to know its not just in my mind.
A couple more observations from bizarro land:
As indicated quickly by Daniel (LHO aLOG 16604), I restarted the front-end process for h1susetmy Tuesday morning, and saw no change in this behavior. Since this bug seems to be extremely low impact, we should toss it into the CDS bugzilla for exploration of the flaw offline on a test stand to see if we can reproduce the error and hopefully debug. Also -- just because we like to blame everything new, it might be worth a check to see if this bug appeared after the RCG 2.9 upgrade, or after Daniel's Tidal Servo installation (see LHO aLOG 15601, or from his new Integrator filter module (LHO aLOG 15560), which is installed and feeding to/through the UIM L filter bank.
