We had four lock losses from nominal low noise state this afternoon. At least three of them seem to be caused by a suddend excursion in DARM correction, which leads to a saturation of the ESD. The reason is yet unclear.
I alrady noticed yesterday that the DARM noise at the periscope peaks (200-400 Hz) was high at the beginning of the lock and then reduced over time.
The first attached plot shows a BLRMS of DARM around those peaks, starting right after reaching the low noise state. There is a clear reduction of the noise over time. The second plot shows that on a similar time scale, the OMC alignment output signal changed, mostly ANG_Y.
This seems to confirm the idea that input beam jitter at the periscope peaks is converted into intensity noise by an OMC misalignment, which changes over time.
To confirm this, I move the OMC angular loops during full lock, adding offsets of few tens of counts to the POS and ANG loops. The third plot shows the steps in the control signals. I was able to reduce the BLRMS by adding an offset of -40 to the ANG_Y loop. The fourth plot compares the DARM spectrum with (red) and without (blue) the ANG_Y offset. I should have increased the offset more. Unfortunately, I noticed that the output was hitting the limit of 300 and I stupidly increased it to 1000: but since the loop was integrating, I broke the lock since I suddenly increased the output from -300 to -1000.
However, the experiment confirms that the noise at the periscope peaks changes in amplitude with the OMC alignment, which is not optimal.
I am not 100% sure what value Gabriele meant to leave, but I have accepted in SDF (Sheila and I are in process of clearing a bunch of diffs in SDF) the value of -30 for ANG_Y for the OMC. I'll check with Gabriele about what value he meant to leave (his alog seems to indicate -40, but we found it at -30).
Since we're using the QPD loops to align the OMC, it's probably better to record any change in the alignment in the QPD offsets. I forget the channel names at the moment, but these are the offsets in the OMC QPD channels (not the same channels in the ASC models). If the offsets are stored in the ANG and POS loops, they will have to be turned off if/when we switch to the dither alignment. If they are recorded in the QPD filter banks it is one less thing to think about.
To summarize the OMC alignment: the QPD offsets have been tuned so the OMC is well-aligned in the low power state. In this state, the dither error signals should be zero. We know that as the power is increased, the QPD offsets are no longer a good alignment, especially in pitch -- this is according to the dither error signals. We suspect the misalignment is due to some junk light that shifts the nominal alignment position on the QPDs. Unfortunately, the misalignment is large enough that engaging the dither loops in the high power state saturates the drive to the OMC SUS. This is why we have stuck with the QPD alignment for now...we should find a solution before O1 that allows us to use the dither loops. The last time the alignment scheme had any attention was in late May.
Needless to say, do remember to check the drives to the OMC SUS OSEMs when changing the alignment settings, they may saturate!
The offsets I left (-30) is better than 0, but not the optimal one yet. It's better to check the OMC alignment again
Why do I get a No Data report from DTT and dataviewer when I ask for H1:ODC-MASTER_CHANNEL_OUT_DQ ? (dataviewer knows about second trend, but not full data...)
Log:
Times in UTC (PST)
1743 (1043) Nominal Low Noise
1752 (1052) Robert heading to Yarm injecting dust
1810 (1110) Lockloss (A bit of seismic activity maybe?)
1828 (1128) Robert baack
1922 (1222) Nominal Low Noise
1925 (1225) Robert off to inject again
1929 (1229) Lockloss (Nothing obvious from the lockloss tool, seismic was low, wind is low, I'm not sure what caused it)
1958 (1258) Nominal Low Noise
2029 (1329) Lockloss (Terramon predicted an earthquake to arrive at 1335 PST, maybe it was off a bit)
2208 (1508) Nominal Low Noise
2214 (1514) Robert to Y arm for dust injections
2244 (1544) Lockloss
When I got here this morning, Nutsinee was reporting that the ISC_LOCK Guardian would get stuck in a few places. Here is what I found and my solutions to them:
ENGAGE_ASC_PART2
Issue: If this state is requested it would get stuck due to self.skipflag returning True. The next time that the run() method would execute, it would jump to the next condition, which would not allow for further movement in the state path when requested.
Solution: I added an extra condition to check if the self.skipflag is True, then return True. This allows for foward movement after a request is put in, while still looking for what it needs to.
ENGAGE_ASC_PART3
Issue: Not all of the logic would run from the run() method, and would get stuck in this state. There are several steps in this method that wait for a timer, do something, then set another timer and wait, before doing more. Problem was that while it would wait for the timer to run out, it would still run through the first step of the logic which included setting the timer. So the timer was set again and again.
Solution: Added a checkpoint flag (self.cp1) so the timer is not set and reset.
ENGAGE_ASC_PART(all)
Issue: No lockloss check decorator in its run() methods!!! Very dangerous, especially since these are requestable states.
Solution: Added the decorator
I hope we are not pushing the commissioners too hard and they are getting sleep...
I have tested this and it has gone through a few times, but now I'm losing lock when it get to DC_READOUT. On to the next mystery!
All time in UTC
07:00 IFO locking. Wind below 20 mph.
07:23 Dale came to notify that the Star Party is over.
08:04 Lock loss. 6.6M earthquake in Solomon Islands.
10:21 Another 5.0M earthquake in Indonesia.
11:00 Started relocking. DRMI won't lock after 10 minutes. I did the SRM misalignment and adjust the beam splitter. Everything went smoothly.
Trouble with ENGAGE_ASC_PART3 for the rest of the night. Please refer to alog20548.
15:00 TJ arrived and tried to lock. He ran into the same problem with ENGAGE_ASC_PART3 and lost lock at REFL_IN_VACUO.
Found a few issues with the logic in the new ENGAGE_ASC states. Currently fixing them and will post and alog when I know they work.
Guardian just spent 40+ minutes in ENGAGE_ASC_PART3 state. DSOFT and CSOFT kept switiching back and forth between "ON:INPUT" and "OFF:FM2". I set the ISC_LOCK to manual and skipped to the next step, lost lock shortly of course. Now I'm stuck at ENGAGE_ASC_PART2 with Guardian message "POP A has a large offset. Align by hand." but abs(ASC-POP_A_PIT_IN1) and abs(ASC-POP_A_YAW_IN1) is already less than 1 (conditions for the Guardian to move on to the next state). Still trying to figure out what to do.
I checked to make sure that the conditions to get pass ENGAGE_ASC_PART2 were met, set ISC_LOCK to manual and moved to ENGAGE_ASC_PART3. DSOFT and CSOFT were in the conditions they were supposed to be (input on, correct gains) but Guardian seems to have trouble moving on (the same loop as before). I paused it and manually copied-pasted the code to Guardian terminal. Then manually select REFL_IN_VACUO. Soon lost lock at PARK_ALS_VCO.
Well, I tried.....
The low frequency sensitivity (below 50 Hz) was coherent with CHARD PIT and YAW. I tried to retune the A2L, first using Hang's script: I had to fix a bug in the function call (number of steps passed as double instead of integer). The script ran, but made the coupling much worse. So I tried to measure by hand the couplings, injecting lines at 21 Hz. A the end of the day I got a performance similar to what I could get reverting to the A2L coefficients of one day ago. So I reverted to the old coefficients.
At 0:53 LT I reduced CHARD PIT and YAW gains by 20 dB, and as expecetd the noise got better. However, the low frequency is non stationary.
The first plot shows the situation at the beginning of a full power, low noise lock. The ISS second loop is closed. After the re-tuningof the IMC offsets, the periscope peaks are no more visible in either the in-loop or out-of-loop ISS second loop signals. This is very good.
However, the first plot shows that the peaks were, at that time, still visible in the DARM spectrum, together with some coherence with the periscope accelerometer. So, there is residual beam jitter that does not generate intensity noise anymore, but that is transmitted to the IFO input and couples to the OMC transmitted power.
The second plot shows the situation after a while we were locked in low noise. The peaks are no more visible in DARM: so something in the IFO drifted in the right direction (probably some thermally induced drift) and now jitter is no more converted by an IFO misalignment into intensity noise. This is something interesting to investigate in the future over long lock stretches.
OMC alignment? This will convert jitter to DARM...
ALL TIMES IN UTC
22:01 Kiwamu out to LVEA to measure ALS Diff PLL electronics next to the PSL enclosure.
22:03 Gabriele has taken the mode cleaner for measurements.
22:35 Kiwamu out of LVEA
23:33 Sigg, Stefan and Robert out to the PSL to connect a cable for RIN strain measurements WP#5434. Robert will be monitoring the environmental entry procedure from outside the enclosure.
23:45 Kiwamu back out to the floor to measure next to the PSL again
01:00 Sigg, Stefan and Robert back from PSL. temps monitored by me in dataviewer. Temp dropped by a few degrees as expected 73F to 69F. Everything remains normal.
01:40 we’ll try locking now. Winds around 35mph
02:15 TCAC members start arriving for tonight’s outreach event
03:39 IFO fully locked at NOMINAL_LOW_NOISE
03:40 Lockloss. Will attempt relock at DC_READOUT
Dale called to inform me that the Star Party is over and that some late comers may show up.
Summary:
The beginning of the night was slow with high winds. IFO hadn’t been locked all day because of the high winds.After a couple of hours I began locking and had to contend with a troublesome Y-Arm.
Input_Align gave me trouble so Sheila and I tried increasing gains and decreasing trigger times to no avail. So we moved on with the other alignments.
The IFO came into full Nominal_Low_Noise lock for a few minutes. Lockloss most likely due to poor initial alignment done while the wind was still about 30mph.
We re-did Initial Alignment.
subsequent attempts to lock resulted in loss, mostly at Engage_ASC. Sheila edited guardian in a way that I believe breaks up the engaging of the ASC loops into different times rather than all at once.
Shift ends in about 30 minutes. IFO locked at Nominal_Low_Noise/22.1W for about 45 minutes now @ 60mpc. Wind has calmed to about 20mph.
Ed, Gabriele, Sheila
We had three locklosses in a row while engaging ASC. I've rearranged the ASC_ENGAGE state in the gaurdian, breaking it into three separate states.
ENGAGE_ASC part1 does the set up, turns on CHARD, checks the recylcing gain. If the recycling gain is OK, it engages PRC2 and INP1, other wise it gives a message to align by hand and will not move on. This logic was in the old state, but it did not stop to wait for by hand alingment. Once the recyclcing gain is high enough, the guardian will engage PR2 and INP1, so the user may want to pause the gaurdian if aligning by hand.
ENGAGE_ASC_PART2 increases the gain for the SRC loops, this part used to be done after engaging PRC1 but we tested this and it is fine. Then it checks the POP offsets, and if they are small enough it engages the PRC1 loop. Otherwise it gives the user a message again, and keeps checking the offsets. It will move on if the drop below the threshold, so if you are aligning by hand you may want to pause the guardian.
ENGAGE_ASC_PART3 engages the SOFT loops. There is a place hoder here to check the offsets and not engage if they are too large. I put this in because sometimes we are loosing lock when engaging these loops, however based on the last lockloss it wasn't obvious how to set a threshold, so currently there is no check, they will always be turned on. As we collect some data while running next week we can look at these locklosses and set an appropriate threshold.
The main motivation for doing this is because the checks that we had previously written weren't really being used (the guardian would move on, just leaving the PRC loops off, but this might not be obvious to the user). A secondary benefit is fewer blocking calls so that we will check for locklosses in places where we used to miss them.
We tested this once, but did parts of the PART2 state by pasting into the command line. The only untested change is turning on the ITM offloading to the top mass in when we first turn on DHARD. Now we are both turning on the offloading and setting it to the full gain in the state DARM_WFS. (lines 1052 to 1056 used to be near the end of the engage ASC state). #guardbomb
The other thing to note is that the OFFLOAD_ALIGNMENT state generators are no longer requestable. I've been slowly trying to reduce the number of requestable states so that we have options that we need in the drop down menu, and fewer options that shouldn't be used.
Our X arm green WFS took extremely long to converge (up to 8 minutes.) While it was too windy to lock this afternoon gabriele and I increased the gain of ALS-X_WFS_DOF_1 and 2 by a factor of 3, now things should be faster. A factor of 10 gain was too much, the loops became unstable.
We probably have not been waiting long enough for these loops to converge while doing inital alingment, hopefully things will go better now that they are a little higher gain. However, it is still a good idea to watch the control signals to see when the converge. Ed has put templates called XARM_GREEN_WFS and YARM_GREEN_WFS in ops/Templates/StripTools that can be used for this.
J. Kissel, J. Driggers, C. Cahillane, K. Izumi I've taken the data that we took last Sunday of the ALS DIFF PLL Open Loop Gain TF and Boosts (from LHO aLOG 20363), built a model of the loop. Given the precision of the measurement, I can make a model the reporoduces the boost filters and the PLL OLGTF to within 1% and 1 [deg]. The most important outcome of this is the ability to characterize the frequency dependence of the low-pass filter just before the VCO. The nominally z:p = 40:1.6 VCO Filter is actually a z:p = 1.05:40 Hz filter. This means that, in using the ALS DILL PLL CTRL signal, prior estimates of overall actuation strength of the QUADs (i.e. LHO aLOH 18711) was over estimated by (1.6-1.05)/1.6 = 34%. This potentially explains some of the discrepancy we saw when comparing the three methos, PCAL, Free Swinging Michelson, and ALS DIFF VCO in LHO aLOG 18767. Of course, as you know, we plan remeasure all methods and make the comparison again. Details: ------------------------------------ Here're the final answer numbers of the model: Traditionally Quoted "Nominal" Values This Model's Fit Values Phase Frequency Discriminator (PFD) z:p = (none):(0Hz) z:p = (none):(0Hz, 450kHz) Voltage Contolled Oscillator (VCO) z:p = 40Hz:1.6Hz z:p = 40Hz:1.05Hz Phase Locking Loop (PLL) Control Common Gain 26 [dB] 25.5 [dB] Phase Locking Loop (PLL) Control Boost Filter 1 z:p:k = 20kHz:2kHz:0dB z:p:k = 1.9kHz:1.89kHz:0dB Phase Locking Loop (PLL) Control Boost Filter 2 z:p:k = 2kHz:40Hz:0dB z:p:k = 1.95kHz:38.55Hz:-0.2dB ------ Attached are figures demonstrating the progression and precision of the model. pg1 : The final answer, showing the model of the ALS DIFF PLL Open Loop Gain TF from 0.1 Hz to 1e6 Hz. pg2 : PLL Control Boost Filter 1; a comparison between measurement, model with fit parameters, and nominal parameters pg3 : PLL Control Boost Filter 2; a comparison between measurement, model with fit parameters, and nominal parameters pg4 : PLL OLGTF, with out the boosts engaged pg5 : PLL OLGTF, with boosts engaged pg6 : A comparison between fit residuals and nominal residuals with measurement. pg7 : A confirmation that the closed loop gain, G / 1+G, of the ALS DIFF PLL is indeed 1.0 to better than 0.1% out to 1 kHz. ------ Perhaps it is of interest for the scrutinous to focus on pg 6, where I compare the residuals, and it'll illustrate my motivations for the fit paramaters, and my quoting that we trust their values to such high precision. Question 1: Why d'you think the magnitude uncertainty is less that 1%, when clearly the boost OLG measurement residual falls by 15% with frequency below 1 kHz, and the unboosted measurement residual falls by 10% with frequency below 100 Hz? Recall that at these frequencies, below 1 kHz and 100 Hz for the boosted and unboosted measurements, respectively there is a rediculous amount of loop suppression. As such, given that the phase residual holds up well to fitted poles, and has little affect on the magnitude residual, I suspect that the measurement magnitude drops with frequency as the suppression increases due to very-small, non-linear, in-loop voltage affects. Question 2: Why do you think there's an 450 kHz pole in the PFD when you've only measured up to 100 kHz? I played around with the very-high-frequency response for a while. Of course, a pole is the only high-frequency tool one has to affect the magnitude residual, and 450 kHz was a good balance between the fitted time delay, and the frequency of the pole. Both the 800 [ns] and 450 kHz pole are "plausible." Inside the PFD, there is a fast chip, that the data sheet claims is good up to 100 kHz, but that doesn't mean much, because you don't know what rolloff filter was chosen. 800 [ns] would correspond to a few hundred meters of cable length difference, so it's not that. But if I see a consistent ~230 [ns] delay in the indepednent measurements of the boost filters, it's not crazy to think that the whole phase-locking-loop accrues 800 [ns]. ------ The analysis script that built this model is commited to the CAL SVN here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER8/H1/Scripts/ALSDIFF/ALSDIFFModel_PreER8.m and the plots live here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER8/H1/Results/ALSDIFF/2015-08-09_H1ALSDIFF_PLL*.pdf
Sheesh -- you'd figure I'd get the most important sentence in the aLOG right. Of course, I mean to say: The nominally z:p = 40:1.6 VCO Filter is actually a z:p = 40:1.05 Hz filter. The pole is at 1.05 Hz. Thanks for the catch, Matt. Further, in the table, I make another typo: Traditionally Quoted "Nominal" Values This Model's Fit Values Phase Frequency Discriminator (PFD) z:p = (none):(0Hz) z:p = (none):(0Hz, 450kHz) Voltage Contolled Oscillator (VCO) z:p = 40Hz:1.6Hz z:p = 40Hz:1.05Hz Phase Locking Loop (PLL) Control Common Gain 26 [dB] 25.5 [dB] Phase Locking Loop (PLL) Control Boost Filter 1 z:p:k = 20kHz:2kHz:0dB z:p:k = 19kHz:1.89kHz:0dB Phase Locking Loop (PLL) Control Boost Filter 2 z:p:k = 2kHz:40Hz:0dB z:p:k = 1.95kHz:38.55Hz:-0.2dB Thanks for that catch, Bram. This is what I get for writing such an entry at 8p on a Friday, and then immediately leaving for an off-the-grid public outreach event for the weekend! Are we any where closer to being able to edit aLOGs past 24 hours?
In order to determine how sensitive we are to activities at the site, and to help set rules for the O1 run, I injected human disturbances as recorded in the table below. DetChar, can you run these times to see if there are events detected in DARM for these activities? My timing should be good to within a second.
Injection |
Time of first injection Aug. 12 UTC |
Injection spacing (seconds) |
Injection session duration and number of injection (seconds, number) |
Truck (3808) horn at OSB recieving |
15:09:00 |
5 |
30, 6 |
Truck (3808) horn at 2k electronics room building |
15:09:00 |
5 |
30, 6 |
Truck (3808) sudden braking near 2k electronics building |
15:16:00 |
5 |
30, 6 |
Quick human moves in the optics lab next to the input arm |
15:23:00 |
5 |
60, 12 |
Human jumps in the change room |
15:25:00 |
5 |
60, 12 |
Silver van horn, 2k electronics building area |
17:04:00 |
5 |
30, 6 |
Silver van sudden brakes 2k electronics building area |
17:05:00 |
5 |
30, 6 |
Silver van horn, high-bay area |
17:08:00 |
5 |
30, 6 |
Siler van brakes high-bay area |
17:10:00 |
5 |
30, 6 |
Car door slams, patio area |
17:13:00 |
5 |
30, 6 |
Corolla car horn, parking area |
17:16:00 |
5 |
30, 6 |
Daniel, Stefan Puzzled by Evan's elog on the still persisting broadband noise (alog 20524), we hooked up the pre-EOM driver 45MHz RIN monitor again (plot 1) - Interestingly, the out-of-loop RIN monitor is still coherent with the pre-EOM driver input (just 38 dB lower), although there is a sign flip below ~2.5kHz (plot 2 shows the transfer function).
Robert, Daniel, Stefan Installed cable from PSL rack to EOM driver test excitation. This will allow us to measure the (driven) 45MHz oscillator RIN transfer function once the winds die down. We hooked the drive up to EXTRA_AO_2, and verified the drive works. We can measure the TF during the next lock. (Work permit 5434)
There is some suspicion that motion of the HAM1 table is contributing to CHARD noise. I learned today that LLO is using some inertial isolation on their HAM1 HEPI platform (i.e. they are blending L4Cs & IPSs, and use that signal for feedback control). I've started looking in to doing this here, but I wanted to see what kind of gains we could maybe expect from such a scheme. The two attached plots are (first image) LH0's HAM1 L4Cs versus our ground, and (second image) LLO's HAM1 L4C's versus their ground. A few points:
1. X at low frequencies at both sites sucks. I suspected sensor correction, my third image is an on/off comparison. X is a lot better with it off (blue is sensor correction off, red is on, green and brown are the grounds). We are getting hammered by wind right now but the performance is equally bad without sensor correction). I remember turning HAM1 sensor correction on a while ago, and did a measurement where I thought it was making an improvement at the time, so I'm not sure what happened here. Y is not as bad, but I've turned it off as well.
2. At 1 hz LHO is doing almost as well as LLO because our ground is quieter. Yay us.
3. The real improvement would come between 1-10hz, where LLO is moving less by ~1.5 orders of magnitude. This will mean installing blends, which is an easy copy paste with foton. Or if LLO was nice they'd send a mat file with a zpk. It wil also mean that we would need to redesign the isolation loops for a UGF above where we want isolation. Right now they are probably 2hz loops, I would guess that LLO's are probably 10-15.
The 2 actions I want to do before 01 are:
1. turn off X/Y sensor correction... which is done. I'll look at the filters and see if I can find whats wrong, but for now I would expect to leave this off. Z sensor correction seems to work fine.
2. Install the LLO blends and see if they make things better at least a little better around 1 hz, and not make things worse at low frequency. I would like to do this Monday.
After that I can look at redesigning the isolation loops, or maybe just copy LLO's, but I would worry that the higher frequency parts of the plant may be different.