model restarts logged for Sun 23/Nov/2014
2014_11_23 02:27 h1fw1
one unexpected restart. Conlog frequently changing channels log attached.
On the way to better initial alignment automation, there are some new scripts for TMS alignment in:
/opt/rtcds/userapps/release/asc/h1/scripts
they are alignTmsxDither.py and alignTmsyDither.py which use baffleAlign.py. These are still a work in progress. After testing is done, we will need to incorporate them into the IAS guardians.
The usual scripts (e.g., alignDitherFull.py) should be as before, and backup copies are in old_2014-11-24 just in case.
Matt, Dan
After Alexa gave us a quick alignment tutorial, we started by fiddling with the dither alignment of the X-arm. The PD thresholds for dithering across the baffle PDs were set to very small values, so small that the dither would declare victory before it even started. Matt has been working on the initial alignment, I'll let him say more about it.
We noticed that the green spot on the TMS-Y mirror, visible through ETMY, is very low. There are large offsets in the EY green QPDs that steer the beam to what looks like a bad place on the mirror. Zeroing these offsets (which is where they were for most of the last six months) would have forced us to begin a painful realignment procedure, so we left them as we found them. Historically it looks like people have set these values to maximize the buildup in the arms, but it's hard to believe that the optimal beam position is so far off-center.
With the arms well-aligned for green we aligned the corner optics - IM4, PR2, PRM, BS, SRM. The scripts for this went quickly, about five minutes.
Then we locked the DRMI, with the arms misaligned. Locking tonight was fast and robust; the DRMI would lock within a few minutes (rarely more than 5 minutes), and would remain locked for tens of minutes. There were many unexplained lock losses, so things are not as stable as we'd like, but we didn't see any mode hopping and the sideband buildup was consistently good. Over three hours of DRMI locking we didn't observe a significant drift in the alignment.
We went through the DRMI guardian and commented out lots of gain/switch settings that weren't needed, mostly for the DC centering. The script is now pretty fast, it goes from lock acquisition to ASC loops on in less than 30 seconds. We did not observe any unexpected long pauses. The duration of the script is dominated by ramping the SRM and PRM M2 gains, this takes 9 seconds total (4.5sec each).
We re-enabled the PRM and SRM2 offloading to M2. We found the SRM offloading was a little flaky and sometimes broke the lock. To make this a little gentler, we reduced the initial gain ramp to 1, and ramped the rest of the way (to 4.5) after the WFS turned on.
We measured the LSC OLTFs and found that MICH and PRCL didn't have much phase margin (~25deg each). We reduced the gain for these loops so that they both have >30deg. The MICH gain is now 3 (was 5) and PRCL is 15 (was 22).
Afterwards I mucked around with the OMC, I found a set of good misaligned values for ITMX and ITMY that minimized the RIN at the dark port when the other ITM was aligned (presumably this steers ghost beams away from the OMC). Following Gabriele's instructions I used the picos to center the beam on IMC WFS A and added some offsets to the IMC ASC loops; this reduced the intensity noise measured by the ISS second loops PDs by a factor of three or so. I tried to engage the ISS second loop but failed. The script posted here did not appear to work; I tried to follow these directions but the loop was unstable.
Plots attached are MICH, PRCL, SRCL loops before we reduced the gain to get some phase back in MICH and PRCL.
FWIW the misalignment settings for ITMX/Y that inject the least amount of RIN at the dark port (with a single bounce off the other ITM) are:
| ITMX PIT | ITMX YAW | ITMY PIT | ITMY YAW | |
| Aligned | 64 | -1.7 | 222 | -140 |
| Nominal misaligned | 0 | -62 | 0 | 0 |
| Better misaligned | -10 | -380 | 282 | -450 |
I didn't observe any change in the RIN spectrum when moving the PRM, SRM, or ETMs. This doesn't solve the problem - the OMC TRANS spectrum is still a little ratty - but it seems to be dominated by RIN noise that is coming from the input beam, which is a job for the ISS.
The ISS second loop scripts are not 100% fail proof. If they fail once, try a couple of times more. Normally the switch one script works more than 50% of the times.
Well, Gabriele is right -- running the ISS secondloop script with Sudrashan this afternoon was problem-free.
For posterity, I've been using Sheila's DRMI OLTF templates for measuring loop gains: /ligo/home/sheila.dwyer/DRMI/DRMI_MICH_OLG.xml and related.
no restarts reported.
conlog frequently changing channels list attached.
Stefan, Matt, Evan
This afternoon and this evening we made some more attempts at locking DRMI3f+arms, with the goal of implementing a handoff from ALS to sqrt(TRX+TRY).
Below are some issues that we’ve noticed.
ALS:
DRMI:
ISC_DRMI guardian seems to get stuck on the OFFLOAD_DRMI state. It engages the M2 locking filters in PRM and SRM, but then does not move on. For the time being I’ve short-circuited the run portion of the state so that it directly returns DRMI_1F_OFFLOADED.ISC_DRMI guardian.Alignment:
Other:
Good UTC time for DRMI3f+arms starts 2014-11-22 06:48:00
I think Evan means "2014-11-23 06:48:00". About the L2P on ETMX L1, it is not just the transient that appears to be problematic. This is a quick look at the first hour of data. ETMX gets badly misaligned once corrections are sent to L1: the green power decreases down to half its maximum level (first plot). The second plot shows corrections sent to L1, while the third plot shows the induced pitch misalignment as seen by the optical lever. Around t = 30min it looks like a realignment attempt happened.
We don't want to cut out the DRMI offloading. When we locked DRMI we only actuate on M3 of PRM and SRM, but we need to offload to M2 to avoid saturating M3. So we need to figure out why the guardian is getting stuck there (maybe there is a hidden sleep some where). Another thing we could do is engage the wfs first and then offload; maybe that would be fine.
Also there haven't been any "new" decoupling filters installed in ETMX L2P L1 stage since Arnaud installed these (alog 11832). Jeff/Sheila/I had only been investing ETMY L2P more recently because that seemed to be the bigger issue.
This is a 5 hour trend with more advanced tools.
(Alexa, Evan, Stefan, Matt)
The ETMX bounce mode has been limiting us in locking ALS DIFF, and causes us to lose lock when the ESD saturates. The first attachment shows the coil outputs of the stages, and there is a clear peak in the ESDs (note this screen shot was taken with a much lower gain in DIFF and with the boosts off; this peak causes saturation when the DIFF gain is nominal). The second attachment shows the bounce mode appearing in the oplev spectrum. You can see that this mode starts to appear Friday and gets worse into today. Thursday night the mode looks fine, which explains why we were able to lock DIFF Thursday night. At some point Friday afternoon we turned off the ETMX L2 oplev damping; however, we see this mode ringing up before this was switched off.
We managed to damp this bounce mode using the optical lever signal. This has the advantage of not needing anything to be locked, so it is very robust, which is important since it took about 2 hours to damp. The first image shows the L2_OLDAMP_P filter configuration we were using, and a time series of the resulting coil signal. The second image is a trend of the OLDAMP_P_OUT signal, which shows the damping in action (the envelope of the signal slowly decreases with time as the mode rings down, and then jumps up each time we increase the gain). After a couple of hours of damping, we were able to reduce the mode amplitude by a factor of 10, and then return to locking. Note - the bounce mode frequency is 9.775 Hz
I want to give some more detail on the filters we used for the bounce mode damping: We used the ETMX_L2_OLDAMP_P filter, and had only FM1 (0:0.5) and FM6 (Pass 9.8) engaged. The OLDAMP pitch gain was -1.
no restarts reported.
conlog frequently changing channel report attached.
Relieving the arm VCO signals doesn’t work once the COMM and DIFF loops are engaged. (The DIFF loop will fight against the HEPI feedback for example.) Therefore I made a similar HEPI offload servo for the COMM loop.
I use MC_F as the error signal and feed back to the common end station HEPIs.
guardian code:
class HEPI_RELIEF(GuardState):
@fault_checker
def main(self):
servo_gain = -.3
offset_ramp_time = 2
offset_chan_1 = 'HPI-ETMX_ISCINF_LONG'
offset_chan_2 = 'HPI-ETMY_ISCINF_LONG'
readback_chan = 'IMC-F_OUTPUT'
ezca.write(offset_chan_1+'_TRAMP',offset_ramp_time)
ezca.write(offset_chan_2+'_TRAMP',offset_ramp_time)
self.hepi_servo = cdu.Servo(ezca,offset_chan_1+'_OFFSET',readback=readback_chan,gain=servo_gain,control2=offset_chan_2+'_OFFSET',gain2=1);
@fault_checker
def run(self):
self.hepi_servo.step()
return True
attached file shows servo with wrong sign, then sign flipped. Then an unrelated lockloss.
So right now when just the arms are locked on ALS, they each feed to their own hepi. At some point the ALS_COMM guardian turns on the HEPI feedback and the arms turn off their feedback.
One last thing to mention: These offsets will walk away over time and should be zeroed periodically. It should probably be written into the DOWN state of one of the guardians.
Jeff, Dan, Alexa, Evan, Matt
To illustrate the need-for / effect-of the feedback to HEPI from ALS_COMM, I attach two plots.
The first shows the start of an IR lock in the arms. The ALS_DIFF path is turned on about 1/3rd of the way through. Once the differential degree of freedom is out of the arms, the tidal drift begins to dominate the ALS green control signals, and the VCOs run away. The rail at 7V in end-X is where we lose the lock.
The IMC control signals
In the second image, I plot a series of locks around the time when Nic implemented his ezca servo code. After some missteps for gain tuning the offloading of the COMM signal to HEPIkeeps the VCO in range.
Nic's code uses MC_F, but for a permanent digital implementation we think that MC_L is a better choice. The current plan is to add a PCIE channel to the LSC front-end module to send MC_L (the input of the LSC-MC filter) bank to the endstation HEPIs.
(Sheila, Evan, Dan, Kiwamu, Stefan, Alexa)
We were able to lock DRMI + arms off resonance a few times tonight, but the alignment was not great and we would mode hop and lose lock. The guardian is slow to engage the asc wfs, which usually help save the alignment and get the ASAIR build-up high. We need to take a look at our guardian script to get them to engage faster. We did not get a chance to try bringing the offset in and transitioning to TRX+TRY.
The other problem is that DRMI + arms off resonance takes a while to lock (around 20 min), and ALS DIFF is not staying locked for much longer than that. Nic and Sheila worked on a HEPI offloading script (see Nic's alog). However, by the time we got this up and running, ALS DIFF was not locking robustly. For one it seemed we had rung up the bounce mode at 9.8 Hz, which makes locking much more difficult. Another thing we noticed is that the ETMX oplev is glitching.
Of course, as I write this alog, ALS DIFF has been locked for 8 minutes ... unclear what changed.
Nic and Sheila ran Jim's script to run the overnight sensor correction test.
In these two weeks I optimized few times the IMC angular offsets, in order to reduce the coupling of beam jitter to intensity noise. Attached the MATLAB script I used to decide the optimal offsets. It grabs a period of data (of the order of some tens of minutes) and plot some scatter plots of how the band-limited RMS of intensity noise (measured with IM4_TRANS) depends on the IMC angular DOFs. The script parameters are set at the beginning, and the attached one shows the typical values I used.
The typical result is shown in the attached picture. In this case one can see thay intensity noise tends to be samller for positive values of DOF_1_P and negative values of DOF_3_P. My conclusion would be that we need to add an offset of about -20 on DOF_1_P and about 2e-3 to DOF_3_P. I didn't want to interfere with the ongoing locking activities, so I didn't add those offsets now.
We noticed that at least three times tonight, DIFF lost lock because of glitches in the ALS (Y) refl control signals. The times were 0:18:12 UTC, 1:35:00 UTC, and 1:47:29 UTC, we didn't investigate all of the DIFF lock losses we had tonight so there could have been other examples. The glitches were both more frequent and larger in the Y arm, but there are also glitches present in the X arm (they are independent glitches in the 2 arms). We see the glitches even without ALS DIFF and COMM locked, they aren't in the optical levers, the PZT signals, or a number of other things we looked at.
Evan and I drove down the to Y end to look on a fast scope at the glitches. By the time we got down there, the glitches had largely stopped.
New SR3 offset: [452.9, -155.3].
There could be +-120-ish measurement error for YAW due to difficulty in finding the PR2 baffle right edge, so later if somebody gets suspicious about the clipping finer YAW scan of SR3 might be a good idea. This problem is not present for PIT.
New SR2 offset: [2800, 800].
Tolerance is +-100, but the center value strongly depends on the SR3 offset.
Since Doug is not available for finding the SR3 oplev beam with new offset, and since we use SR3 oplev for damping, we reverted SR3 and SR2 back to the original angle.
Tomorrow morning SR3 oplev will be done with Doug.
Plots will be posted later.
SR2 [2963.7, 2728.0]. SR3 [430.3, 142.6].
Centered AS_C using SR2 offset: [2917. 7, 2794.0].
At this point ASC_SUM was [1576.29, 1576.87] (measured twice, each is 10sec average number).
Position 1: [3042.9, -438.4].
Position 2: [-2137.1, -408.0].
Center = (Pos1+Pos2)/2 = [452.9, -423.2].
Then moved the beam to the right edge.
Position 3: [452.9, -1248.4+-100]. Barely touching.
Position 4: [452.9, -1848.4+-500]. Totally blocked.
For whatever reason YAW is much more ambigous than PIT to my eyes, and right edge is more difficult than top and bottom.
Right = (pos3+pos4)/2 = [452.9, -1548.4+-510].
Center-Right = 1125.2+-510 in the slider counts. This is supposed to be 36.75mm.
SR3-SR2-SRM angle is 1.67deg, and the baffle-SR2 distance seems like about 46cm (eyeballing HAM4 systems drawing), so the beam distance at the baffle is 17.5mm.
New beam position = 17.5mm/2 =8.75mm to the left of the center.
Y=-423.2 + 8.75mm/36.75mm * (1125.2+-510) = -155.3 +- 120;
NEW SR3 slider = [452.9, -155.3+-120];
At this point SR2 was moved to [2708.7, 814.0] to center AS_C, and the SUM was found to be [1596.99, 1597.84, 1596.48] (three measurements), this is about 1.3% increase from the initial number (but note below about the interference pattern caused by either some etalon effeft of a ghost beam from somewhere).
Scanned SR2 in YAW, then in PIT, to find the Faraday edge while centering AS_C using Pico.
Today I made a finer scan than previously done (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=15023) and what I originally thought to be a ghost beam looked more like an etalon type interference that mostly depend on the beam angle on AS_C. It was mostly in YAW but you can also see it in PIT, and the pattern is very repeatable.
Anyway, my objective here is just to place the beam at the center of the Faraday aperture, and I chose this:
New SR2 slider = [2800, 800].
The peak-to-valley in YAW data is as large as 1.5% or so. Does this mean that SR3 scan before/after comparison is useless? No.
During SR3 scan, the beam was shifted by 300 counts in YAW, or about 10mm on SR2. The beam pointing on AS_C was fixed using SR2, so the beam angle on AS_C changed by about 10mm/18m or so, i.e. about 600urad or so.
OTOH, the full scale of the SR2 YAW scan corresponds to the Faraday aperture of 20mm, and that's roughly the beam shift on the pico. The beam position on AS_C was fixed using pico, and pico-AS_C distance is about 14 inches, so the beam angle change over the entire SR2 YAW scan is 20mm/14" = 60mrad or so. This is three orders of magnitude larger than SR3 scan angle change.
If we replotted it as the function of beam angle on AS_C (which I didn't), and put both the SR2 scan and SR3 scan data on the same plot, two data points representing the before/after SR3 scan would be basically on one vertical line in the plot. If this is indeed an etalon type thing on the diode surface, basically SR3 before/after the beam would see the same interference. That's my theory anyway.
Here is a collection of suspicious CPS trips on ITMX. This is a problem only on this chamber, which has just cropped up in the last few weeks. It needs to be investigated.
another example
Still there, still a problem....
I made a few plots today. The spectrum of the CPS right before and right after the trip, and the zoom-in time series plot around the time of the trip. I notice that the time that the CPS signal goes down does not match the GPS time indicated the trip exactly (off by a few hundred milliseconds). While making the spectrum I also notice the peak near 4Hz in the "beforetrips" spectrum plot. So I went and look at the LHO summary page and found that there's a small gaussian-looking bump around 4Hz that seems to appear everyday. Probably not very important but just wanted to point that out. Also, please note that V2, H2, and H1 signals are almost perfectly overlapped (that's why you only see two colors). I'll start looking into other related channels that *should* have seen the signals. Although I believe this might be an electronics issue.
