Thomas updated the SUS Drift Mon to the a time just at the end of last nights 2hr lock (2/12/15 07:35:00 UTC, 1107761716 GPS).
model restarts logged for Wed 11/Feb/2015
2015_02_11 11:28 h1fw1
2015_02_11 12:23 h1susetmy
2015_02_11 12:31 h1fw1
2015_02_11 13:16 h1susetmx
2015_02_11 13:20 h1susitmx
2015_02_11 13:20 h1susitmy
2015_02_11 13:27 h1broadcast0
2015_02_11 13:27 h1dc0
2015_02_11 13:27 h1fw0
2015_02_11 13:27 h1fw1
2015_02_11 13:27 h1nds0
2015_02_11 13:27 h1nds1
2015_02_11 13:37 h1calcs
2015_02_11 13:40 h1calcs
2015_02_11 13:42 h1broadcast0
2015_02_11 13:42 h1dc0
2015_02_11 13:42 h1fw0
2015_02_11 13:42 h1fw1
2015_02_11 13:42 h1nds0
2015_02_11 13:42 h1nds1
2015_02_11 14:27 h1fw1
2015_02_11 16:52 h1calcs
2015_02_11 17:07 h1iopoaf0
2015_02_11 17:07 h1oaf
2015_02_11 17:07 h1pemcs
2015_02_11 17:08 h1iopoaf0
2015_02_11 17:08 h1oaf
2015_02_11 17:08 h1pemcs
2015_02_11 17:09 h1calcs
2015_02_11 17:09 h1odcmaster
2015_02_11 17:09 h1tcscs
2015_02_11 17:11 h1broadcast0
2015_02_11 17:11 h1dc0
2015_02_11 17:11 h1fw0
2015_02_11 17:11 h1fw1
2015_02_11 17:11 h1nds0
2015_02_11 17:11 h1nds1
three unexpected h1fw1 restarts. New QUAD SUS code. Install of the h1calcs model. Several related DAQ restarts. Power cycle of h1oaf0 to remove ADC noise from PEM.
I ran my brute force coherence script for the almost 2 hours lock of two days ago (the coherence for last night 2+ hours is running, you guys are locking too often!). Basically this script computes the coherence of the LSC-DARM_IN1 signal with *ALL* available channels. Some channels are supposed to be coherent (for example DARM_OUT), so I have a list of excluded channels. For the moment being I used the same list as for LLO, then I'll see if I need to fine tune it.
The full report is available here:
https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_1107680416/
Some preliminary comments:
Basically, so far there is nothing new with respect to what Dan reported.
The attached plots are taken direclty from the report, and they show the coherence with some of the channels I discussed above. The top panel shows the coherence, in log scale for both x and y axis. The dashed red line in this panel corresponds to the level of coherence you expect for completely uncorrelated signals, given the resolution and the number of averages. Any coherence at this level or below is completely accidental. The bottom panel shows the spectrum of DARM signal and the noise projection for the auxisliary channel, based on the coherence. It gives a good indication of how significant the coherence is, but of course must be taken "cum grano salis".
The data was transferred to the new hardware per WP 5044. There is a gap in the data from around 8:15 am to 12:10 pm on Feb. 11.
After we broke the 2+ hour lock, I got greedy and tried to re-lock the IFO myself (at least to RF), after having been shown the ropes by Sheila and Evan earlier in the day. - I could get past finding the IR steps in the ALS COMM guardian when the COMM vco failed or needed some hand holding, - Could get past tweaking of the H1:ALS-C_DIFF_PLL_CTRL_OFFSET when ALS DIFF couldn't find IR, - tried tweaking the Beam Splitter alignment while stuck in the DRMI lock acquisition, and - managed to survive a few quick breaks of the DRMI during the initial 1F to 3F transition ~ 10 seconds after catching a DRMI lock - Can get all the way through the CARM offset reduction, but on of the first steps of the "RF DARM" state breaks the lock consistently. The last example of which I'd stayed for was at Feb 12 2015, 09:23:57 UTC I feel like I'm playing a 1980's Atari game, and can't get past a mid-level boss! Ah well. Good night all! I've left the ISC LOCK guardian in the requested state "RF_DARM."
Today I measured the balancing of the OMC DCPDs, using an excitation at 103Hz into the OMC length servo, and measuring the height of the second harmonic in the nullstream channel. The DCPDs are only, as far as I can tell, 1% misbalanced. In the attached plot data are in red, the blue is a by-eye fit. I have set the balance to be 50.5% PDA, 49.5% PDB (1% towards PDA on the slider).
Evan, Dan, Jeff, Peter, Daniel, Lisa2+ hours lock on DC readout, this time for real!
* Feb 12, 5:33 UTC lock on DC readout, engaged ISS loops shortly afterwards * Feb 12, 7:00 UTC improved low frequency noise by turning off ETMs optical lever damping * Feb 12, 7:35 UTC still lock on DC readout, starting alignment tests (intentionally changing BS alignment; also reduced the BS optical lever damping gain by a factor of 10 * Feb 12, 7:48 UTC unlocked by closing the AS beam diverter Positive news of the day - Thanks to the improved bounce mode damping , we can now reliably damp the bounce mode, and it is not a limiting problem anymore; - ETMX and ETMY optical lever PITCH damping loops have been reduced by a factor of 100; it turned out that these loops were responsible for the huge excess of noise that we saw yesterday below 40 Hz, and that was causing the OMC transmission to shake painfully. Now the OMC is quiet and happy (the Guardian has been updated to reflect this change). - We turned off the QPD alignment of the OMC, and replace it with dither alignment; - We turned on the ISS first and second loops: the positive news is that we improved the high frequency noise by a factor of 10. Negative news of the day - While we have improved noise at low and high frequency, we now have a new bump of noise around 100 Hz which was not there yesterday . - Our wonderful BS WFS loops, so effective yesterday, were not working today. However, today the BS alignment was not critical, and we have barely had to touch the BS anyway, but we need to figure out what happened. Right now we commented out the BS WFS in the locking sequence; - Yesterday the locking sequence was very reliable, and worked 5 times in a raw..today it has been less robust, despite similar environmental conditions.. - The RF of in-vac POP is somehow broken - no signal there. It is probably not a big deal right now at low power. - We also closed the OMC REFL beam diverter (level 2 of "Valera's levels of awesome": it works, but it doesn't do anything)
Attached are some plots of coherences and OMC noise.
Fig 1: Coherence between DARM and LSC, ISS - the excess noise at 70-400Hz, much worse than last night (grey), is coherent with the common DOFs (PRCL, SRCL, and CARM [not plotted]). Also the ISS 2nd loop out of loop PD.
Fig 2: DARM and ASC - there is coherence between IMC yaw WFS error signals and DARM, implying that the noise is due to some beam jitter coupling
Fig 3: OMC noise - note that we are shot noise / dark noise limited above ~1kHz
In figure 1, the grey trace is yesterday's almost-two-hour lock, the green is today before the oplev damping was turned downand without ISS second loop, the blue is at the end of the lock. There was a substantial improvement in many bands due to the ISS 1st loop and loosening up the oplevs. The second ISS loop didn't change much, although we did notice it reduced the intensity noise on, for example, ASC-POP_A_SUM.
The beam jitter into the IMC, as measured by DOF1_{P,Y}, hasn't changed from last night. The working hypothesis is that the alignment has drifted into a place where beam jitter couples more strongly into the length DOFs. We checked many PEM channels for coherence with DARM in case, for example, someone had left a fan on in the LVEA, but found no coherence with environmental channels.
A plot of POP vs. POPAIR demodulated signals is attached. Evidently, there is nothing meaningful in the in-vacuum signals, even though there is about 1.7 mW of dc power. This is roughly what we expect; with 2.8 W incident on the IMC, an IMC transmission of 90%, a power-recyling gain of 33 W/W, 250 ppm of transmission through PR2, and 10% transmission through mirror M12 on HAM1, we get 2.1 mW.
Also attached are some short videos of the AS port and the OMC transmission during full lock.
The Box is Checked, and the Advanced LIGO Project has delivered on its prime instrument performance goals. Congratulations to the entire aLIGO team for making this happen! and thanks again to the Hanford All-nighters.
Congratulations and happy noise hunting.
Once again, congratulations from your LLO colleagues! Allow yourselves a moment to enjoy this sweet success before having good luck with the noise hunting…
Fantastic! Noise hunting licenses for everyone! Curious as to what the wind conditions were like.
Dave -- wind conditions were around ~10 [mph]. Microseism was ~5e-1 [m]. Pretty quite night! May we have MANY more. Just like tilt meters.
PeterF, Kiwamu
We found that the RF cables from POP_A were unplugged at the front panel of the demodulation board by the PSL enclosure. We hooked them back in. Confirmed that the demodulated signal showed some interferometer fringes.
J. Kissel, S. Dwyer
After both Keita and Sheila had recited the H1 SUS ETMY's measured highest vertical, "bounce," mode frequency was 9.7305+/-0.0002 [Hz], I was glancing around at the various notching done in the control filters for H1 SUS ETMY. I'd found that the optical lever's pitch loop had a bandstop filter of
ellip("BandStop",4,1,60,9.7,9.9)ellip("BandStop",4,1,40,13.4,14.2)gain(1.25893)
i.e. just missing the ETMY frequency. Thus, we adjusted the filter to better match 9.7 [Hz], i.e.
ellip("BandStop",4,1,60,9.6,9.8)ellip("BandStop",4,1,40,13.4,14.2)gain(1.25893)
This reduces the gain of the loop at 9.73 [Hz] by a factor of 76.9231 (37.7 dB).
This was done well before locking activity go started this evening. The pitch loop remains on. As with the DARM DAMP filters, this change was individually loaded into the bank.
S. Dwyer, J. Kissel, K. Kawabe After installation of the infrastructure (LHO aLOG 16655), and copying of LLO's filters (and adjusting for the specific frequency of 9.7305 [Hz]; LHO aLOG 16658), we tried damping the H1 SUS ETMY's highest vertical mode (a.k.a. "bounce" mode). For the first attempt, the IFO was locked only using ALS diff, with Sheila in the driver's seat. At the time, the mode had been rung up to ~7e-12 (DARM) [m/rtHz] @ 9.7 [Hz]. We had tried a few configurations of the filter bank, and only adjust the gain. We'd found how to ring *up* the mode with a positive gain, with the +60 [deg] (FM2) and bp9.73 (FM4) filters engaged -- then flipped the gain sign (i.e. flipped the phase 180 [deg]), and immediately could see reduction. We had the gain as high as -64, using ~50% of the DAC range, after which took about ~10 [min] for the mode to cool down to the ALS DIFF noise floor of ~1e-12 [m/rtHz] @ 9.7 [Hz]. After a lock loss of two, we were able to get as high the IFO guardian state "RESONANCE," with CARM controlled using digitally normalized RELFAIR9, and DARM has been transitioned to AS45 Q. At this point we saw the mode was still quite run up, so we again turned on the DARM DAMP V filter -- same filter combo, and we could see just as quick a reduction with a gain of -64. This time however, we were using much less of the DAC range, so I went up to a gain of -100, and the mode was quickly damped to the RESONANCE noise floor of ~1e-13 [m/rtHz] @ 9.7 [Hz] within a minute or three. With these two victories, I'm reasonably confident that this will be our ticket to future bounce-free success. Design strings: FM1 "+60dg" zpk([0],[2.16667+i*12.8182;2.16667-i*12.8182],1,"n")gain(0.0523988) FM2 "-60dg" zpk([0],[1.21667+i*7.1979;1.21667-i*7.1979],1,"n")gain(0.0911865) FM4 "bp9.73" butter("BandPass", 4, 9.3, 10.4)gain(120, "dB") with all filters set to an input switching of "zero history" and output switching of "immediately." Attached is a bode plot of the final good set of filters together, FM2 and FM4. Note that these filters were loaded individually from each bank, Keita did *not* load the the whole foton file.
Why local damping does not work:
Before the DARM bounce damping was implemented, I started playing with the BOSEM damping at the top stage, and concluded that it will not work even though the bottom stage bounce mode is clearly visible in the top BOSEM.
The reason for this is that the feedback only sees the top to top transfer fuction.
This TF at the bottom bounce resonant frequency (9.7305 something something Hz) is not that different from that at off-the-resonance proximity (e.g. 9.7Hz). I confirmed this by various things like injecting band limited white noise, injecting sine wave tuned to the resonance as good as possible (9.7305 something level), injecting sine wave at proximity frequency, feeding back with a band pass filter and turning up the gain until it does something.
This means that, since the coupling from the top to the bottom is small, the top mass starts oscillating at proximity before the feed back can do something significant to the bottom motion.
The decay time (1/e) for the bottom bounce mode, measured by the top BOSEMs, was measured to be 13000 to 14000 seconds (Q of 4E5 or so).
I was able to reduce this to 8000 to 9000 seconds by top mass local feedback, which is useless.
Why DARM to top mass damping works:
The feedback loop sees the top to the bottom TF, which has a very sharp peak (as in Q of 4E5 sharp) at the resonance. Therefore you can touch the resonance without touching anything else.
Note that DARM to top mass bounce damping affects the calibration, so there will be a calibration hole at 9.7Hz if the damping filter is on.
When the bounce motion measured by DARM was on the order of 10^-11m, we needed to use the full range of DAC to damp the motion using DARM bounce damping path.
I have measured the violin mode using the most recent lock data (February 10th ~5pm). Due to the lack of precisions in the existing information and the mismatch in values I was not able to identify any of the peak except for the ITMX BR (Back Right) at 499.9 Hz. The violin mode found in the recent lock data are as follow:
499.9, 505.58, 505.71, 505.80, 505.85, 505.92, 506.92, 507.16, 507.20, 507.36, 508.01, 508.15, 508.20, 508.85 Hz
The (fundamental harmonic) violin mode is expected to be somewhere between 500Hz and 520 Hz. Not every line is present - possibly due to high noise floor.
The violin mode has been previously identified as follow:
| ITMY (alog11184) | Frequency (Hz) | Bin Width (Hz) |
| BL | 501.5 | 0.25 |
| BR | 501.3 | 0.25 |
| FL | 504.2 | 0.25 |
| FR | 502.8 | 0.25 |
| ETMY (alog9359) | ||
| BL | 508 | 0.125 |
| BR | 508.1 | 0.125 |
| FL | 507.9 | 0.125 |
| FR | 507.6 | 0.125 |
| ITMX (alog11044) | ||
| BL | 501.2 | 0.06 |
| BR | 499.9 | 0.06 |
| FL | 502.2 | 0.06 |
| FR | 500.8 | 0.06 |
| ETMX (alog6858) | ||
| BL | 505 | ? |
| BR | 506.5 | ? |
| FL | 506.5 | ? |
| FR | 505 | ? |
Positions of the wire:
BL = Back Left
BR = Back Right
FL = Front Left
FR = Front Right
Where's left and where's right you ask? I'm trying to find the answer myself!
The designations back, front, left and right refer to looking at the reflective coating of the optic - so looking at the mirror from inside the arm. Don't forget that each mode will be split by about 0.07 Hz (based on LLO fibers) with relative amplitudes dependent on orientation of the fiber axes.
J. Kissel ETMX UIM Driver's rocker switch tripped last night at 8:44:30 UTC (Feb 06 2015 00:44:30 PST), which cause all OSEM output on the L1 / UIM stage (and the sensor signals as well) to fail. I've turned on the rocker switch, and the stage appears functional. Other points: ------------- There is no smoking gun for this chassis power outage. There are not enough diagnostic tools in the frames to look at all possible suspects (power surge to the chassis [no power monitors], chassis overheating [no temperature sensors], an inconsequential component on say -- a monitor board -- smoking [only one of the four monitor signals for each channel are stored], too large a current request [the NOISEMON is stored, not the FASTIMON]). Output request of DAC (from any ISC) does not appear to coincide with trip, but difficult to tell with dataviewer. There is an Longitudinal request to drive really hard, but it's unclear whether it's a the cause of or a result of the driver's power being shut off. Output request is roughly 42 [mA] (according to FAST_I_MON) on all four coils leading up to trip, switch trips on 3 [A]. Even the sudden, very large output request does not cause any current larger than the 42 [mA] -- as reported by dataviewer / EPICs. Reqrettably, only the severely-high-passed-at-5-[Hz] noise monitor signals are stored in the frames, but I attach the time series of one of the coils anyways to indicate the exact time. It shows a nice smooth ramp down at the time of the chassis power down. Serial number of this box is S0900303. I attach pictures of the box after I'd turned it back on. When I first approached the box, the front "power" LED lights were OFF, and the rocker switch was in the opposite (powered off) position. I'm not sure if there is good way to make this power shut-down control-room visible. I did notice that all the coil driver monitor signals were frozen at -16 [ct], and that the OSEM sensor speed dials were zero for this stage and this stage alone, which is what immediately clued me in that it was a flaw with the entire coil-driver chassis or satellite amplifier. This last happened (according to collective analog CDS crew's memory) on H1 SUS ITMX UIM driver, some time ago, 19 November 2013. See LHO aLOG 8635.
What's going on here?
Adding more to Daniel's comment -- the problem in the screen shot he shows is ONLY a problem with the reported decimation filter output for the ETMX L1 LOCK L bank. He had repeated the same test on the ETMX L1 LOCK P bank, and saw no difference between the OUTPUT and OUT16. It's not an issue progressing forward because no one and nothing uses the decimation channel but for diagnostics, but it's this kinda of stuff that makes users immediately suspicious of the entire front end code as a whole (regardless of how serious the problem actually is) and demand "REBOOT! REBOOT!" without really identifying the specific problem. Again, all signs point to that this is UNRELATED to the coil driver switching off from a current overload. This being said -- I agree the decimation display problem with *this* filter bank *is* weird, and we'll try to debug further.
I have seen this too. Good to know its not just in my mind.
A couple more observations from bizarro land:
As indicated quickly by Daniel (LHO aLOG 16604), I restarted the front-end process for h1susetmy Tuesday morning, and saw no change in this behavior. Since this bug seems to be extremely low impact, we should toss it into the CDS bugzilla for exploration of the flaw offline on a test stand to see if we can reproduce the error and hopefully debug. Also -- just because we like to blame everything new, it might be worth a check to see if this bug appeared after the RCG 2.9 upgrade, or after Daniel's Tidal Servo installation (see LHO aLOG 15601, or from his new Integrator filter module (LHO aLOG 15560), which is installed and feeding to/through the UIM L filter bank.