model restarts logged for Mon 17/Nov/2014
2014_11_17 14:27 h1fw1
one unexpected restart
(Sheila, Evan, Keita, Nic, Kiwamu, Alexa)
We locked the IR to the x-arm and adjusted PR3 until we achieved the good IM4 alignment specificed in alog 15109. We adusted the HAM3 picometers to restore the ALS green trans beam on ISCT1; I used the camera reference position that we had previously found (alog 15072) and ensured a high build up in TRX. We then aligned MICH on the dark fringe and PRM using PRX lock. Not surprisingly, we had to move PRM a lot. SRM was also aligned, with SR2 and SR3 in the old configuration. Attached is a screen shot with the current alignment values. The POP sled was aligned as noted in Keita's alog 15116.
We were able to lock PRMI on the sideband, and acheived 260 cnts build up. Our best build up had been between 260-399 cnts as seen by a trend Keita had taken over the span of 3 weeks. We feel okay about our current build up.
Sheila and Evan marked a new IM4 trans beam on the wall in the LVEA. It seemed the beam hadn't moved all that much from the old position (from June); maybe a few millimeters.
We adjusted the ALS alignment on ISCT1; we got 0 dBm for COMM beatnote, and 6 dBm for DIFF beanote.
ALS DIFF locked right away. The UGF was 7 Hz, with 70 deg phase margin. The in loop RMS was 0.2e-3 um/srtHz at 0.1Hz. This was with feedback to ETMX only. We then tried feeding back to both ETMX and ETMY L1 stage with Jeff's new EY L1 L2P filters engaged (FM6, FM7); however this did not work. The L2P filters output a huge value 1e+20 and we saturated the DAC immediatly. FM7 has a step response of 10^18 ! Also, are we supposed to leave Arnuads invP2P filter engaged? We must have rung up a mode around 10 Hz. Nic and Kwamu fixed this (see their alog 15118).
We locked DRMI 1f with the ASC wfs engaged. POPAIR18 measured 343cnts, and ASAIR90 measured 5731. It appeared to be the best DRMI "moment of [Sheila's] life." DRM1 1f lock time Nov 18, 2014 6:07 UTC. We transitioned to 3f flawlessly at ~ Nov 18, 2014 6:12 UTC. We lost lock at Nov 18, 2014 6:14 UTC.
Then we tried locking DRMI with the arms 500 Hz IR off resonance. At first, the DRMI alignment did not look as good; when we engaged the wfs the build up increased dramatically but we would lose lock when the build up got high. We deduced that the gain of the loops had to decrease with the precense of the arms. We reduced MICH gain from 5 to 4 to obtain a 15 Hz UGF. PRCL we reduced from 22 to 11. At some point the alignment looked good and the lock was stable for a bit, but we lost lock trying to measure the SRCL loop gain. We got tired...
"We lost locked at Nov 18, 2014 6:14 UTC" GPS 1100326478 Something bad happens to MICH (-0.1s in the plots, different zooms).
Sorry for the typo's -- deux*, lost lock**
Thanks for the plots Lisa. After this lock loss, we had another stable lock that only broke because we took it down. But it seems like we should take a look at MICH.
Also, Jeff told me that BOTH EY L1 L2P FM7 (UIM_InvP2P) AND P2P FM1 (InvP2P), should be turned ON. This is the configuration we used last night when we tested his new filters, but we had been unsure whether this was correct or not.
(kiwamu, nic)
There was a very large ~10Hz spike in the ALS DIFF control signal. The signal was also visible in the optical lever signals of both ETMs, and they looked coherent with each other
We were only feeding ALS DIFF to ETMX, so Kiwamu hypothesised that this indicated that the source of the motion was ETMY and the DIFF loop was impressing motion onto ETMX.
We waited for a long time for the mode to damp itself, but it was very persistent.
We weren’t sure what caused the mode to ring up, it may have been the bad L2P filters that caused the DAC to saturate. The only drive signals on ETMY were OSEM and optical lever damping. So we also put a notch in the OL damping loop in case that was ringing it up.
Finally we decided to just damp the mode using the “DARM ERR DAMP FILTERS” on the SUS screen (thanks LLO!). Using the H1:SUS-ETMY_L2_DAMP_MODE1_INMON filter bank and with the output matrix feeding to ETMY L2 pitch, we put a narrow bandpass around 9.73Hz, as well as a velocity damping filter. Then we just varied the output gain until we could see the mode changing amplitude. Then we flipped the sign and further increased the gain until the mode was clearly ringing down. It had a time constant of about a minute. Once we had a good gain, we dumped the gain into a filter module.
Now the correct gain in H1:SUS-ETMY_L2_DAMP_MODE1_GAIN to damp the mode is -1.0.
Bounce mode coupling (to the length) is a factor of 80 larger at EY than at EX due to local gravity VS IFO z-axis angle (alog 14876).
Unlike roll, bounce mode should have a predictable, consistent phase relation to the length, i.e. bounce up = longer arm.
You can see the bounce mode rung up by looking at top mass V (attached) for EY, but not for EX.
This should mean that the bounce could be damped by a factor of few using the top mass (which might or might not be able to stop the ringing up).
Also, if we have a path to route the LSC signal to the top V, we should be able to damp the mode easier (as in easy filter design) than feeding back to L2 stage.
Just in case my elog wasn't clear.
We found the mode rung up while locking ALS DIFF. We were not sure of the reason for it to be rung up.
We were able to damp it using the DARM ERR damping filters that feed to L2, these had not been used in the past. We commissioned them for the first time. Our initial guess of the sign was wrong (50/50 chance) so we flipped it to damp the mode.
Once having the new filters and gain, the mode was damping. We don't yet know if the coupling is stable (requiring retuning) over long time scales. I think LLO has seen that these are stable.
The error signal is DARM ERR so it can only be turned on when on ALS DIFF or full lock. With ALS DIFF, the gain that gives good damping is -1.0.
After IM4 alignment was moved, POP sled QPDs were recentered using a straight shot beam.
POPA and POPB SUM are about 0.33 and 0.30 right now. There is some clipping going on in the sled path (we observed these SUM went to 0.38 and 0.34 respectively with different IM alignment), but that was expected from my original IM4 scan (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=14567).
POPA pico counts are [X, Y] = [-27491, 6688] (was originally [0,0]).
POPB pico counts are [-23206, -8455] (originally [0, -300]).
Alignment tips:
When POPB QPD is to the right relative to POPA QPD, both of the PICOs are moved in positive X (that is, the pico X count increases for both).
When POPB is higher than POPA, POPA pico is moved UP (that is, pico Y count increases) while POPB pico is moved down.
Sheila, Evan
We have recentered the PR3 oplev. The controller is in the AOS cabinet by the PSL.
After the fix reported earlier, the contamination of ISS signals by the PRM reflection is smaller than before, but still not neglegible, see the attached plot (blue PRM misaligned, red PRM aligned).
We should investigate more to see if this residual noise is still entering from the same path or not. Maybe it would be a good idea to dump the beamp directly before the first mirror of the periscope.
With the correction (update) of the L2C2L matrices planned for possibly tomorrow, I went through the controller development (required) using generic loops Hugo designed (SEI log 489.) These are 5hz UUG controllers. The vertical dofs looked okay but X Y & RZ looked a bit marginal around the UUG--see first attachment. This plot shows the 5hz generic controller for X; the gain peaking is higher than comfortable and the phase margin is marginal. The controllers for Y & RZ are marginally less marginal. I made a new controller set with 2Hz UUG and and reduced the boost zero from 1 to 0.7hz on X Y & RZ. I'll keep the 5hz controllers for the vertical dofs and go with these more conservative filters for the horizontal dofs.
The pdf attachment shows all eight controllers I plan to install tomorrow if allowed.
The second image attachment is the only controller I found saved in the svn. It looks pretty wild but it doesn't quite match the foton filter so I don't know if it really represents the controllers we've been using. Still it sort of represents the wild west in our controller development evolution.
08:00 LVEA is LASER HAZARD
08:15 Re-Started Operator2 iMac (Mid and End cameras)
08:30 Morning meeting
08:59 Fil and co. to EX and then EY
09:15 Karen to End station
09:16 Doug ino LVEA to preview OpLev work
09:19 Jeremy, Mitch, Betsy and Travis into LVEA W bay
09:36 Cris to EX
10:02 Bubba to MX to get palette jack
10:12 Karen leaving EY
11:00 Cris back from EX cleaning
12:15 Fil et al leaving EX
12:50 Mitch and Jeremy out to W Bay w/comm appr
13:46 someone on site to take water samples?
A second round of testing on the all-metal 3IFO QUAD unit Q7 is attached. After we saw a few cross coupling terms and struggled with coherence in the prior measurement set, we made a few adjustments to the QUAD mechanics. However, readjusting the lacing cabling and retuning the sensor mounting tablecloth seemed to not help much. There is still L and R bleeding into P on the reaction chain. We were able to improve the coherence by building a fort around the test stand however. The main chain looks to be in pretty good shape though.
The first attachement below is of Q7 with damping on and off.
The attachment below is of the Q7 (pink traces) undamped TFs compared to the 3IFO Q6 and Q8 units.
Screenshots of damping filters for DC offset diagnosis.
Spectra for all stages of Q7 attached. Note that the L1 and L2 stage OSEMs are not aligned and centered.
(Keita, Alexa)
We moved IM2 in pitch to correct for the 60urad deviation that Kiwamu and Keita saw on Friday (alog 15081). Originally, IM2 P, Y = 17000, 5154 with M1_DAMP P, Y = 580, -51. This was adjusted to IM2 P, Y = 19200, 5154 with M1_DAMP_ P,Y = 637, -45.
We then repeated an IM4 scan on PR2. The new good alignment values are: IM4 P, Y = 6670.5, -2735.0
The scan procedure:
We moved IM4 in pitch until we saw the beam on the PR2 baffle. This allowed us to center the beam in yaw; we found that the good center position in yaw was -881.8 -- this was consistent for both top and bottom of the baffle. Then we found the pitch values for which the beam was just barely clipping the baffle:
| IM4 Pitch | IM4 Yaw | |
| Barely Top of PR2 Baffle | -24329.5 ± 2000 | -881.8 |
| Barely Bottom of PR2 Baffle | 37670.5 ± 1000 | -881.8 |
This average gave us the pitch position of 6670.5.
It turned out that yaw was bit more tricky. For whatever reason, the "barely clipping" procedure in yaw did not work; it gave us a center value of -3881. So instead we looked at the PR3 camera (with ITMs very misaligned), and determined when the beam disppeared in yaw:
| IM4 Pitch | IM4 Yaw | |
| Left on PR2 baffle, No beam on PR3 | 6670.5 | 6418.2 ± 100 |
| Right on PR2 baffle, No beam on PR3 | 6670.5 | -11681.8 ± 500 |
This difference amounted to 18,100 counts. We then looked at the PR2 baffle lower left and lower right beam positions, and found that this yaw seperation for the PR2 baffle diameter made sense.
As noted in Keita's alog 15063, the diameter of the baffle is 65.4mm, and the beam must be aligned to 26mm from the right edge (or equivalently 6.7mm from the center). So we used
yaw = center + radius in counts/ radius in mm * distance from center
yaw = -881.8 + (6.7 mm / 32.7 mm) * (-9050) = -2735.0
Keita pointed out to me that as we move the beam in yaw to clip on the left side of the PR2 baffle we will actually first see a back reflected beam from PR2. This probably explains why our yaw measurements for the "barely clipping" scheme was incorrect; we were probably measuring the back reflected beam from PR2 and not the beam from IM4.
The calibrations on PR3 angles are wrong. In the attachecd screen shot you can see that someone moved the alignment slider -72 urad in yaw, the OSEMs sees a shift of -62 urad, and the optical lever records a shift of -26 urad
This morning the power on IM4 was about half of what it should be, even though all other readings of IMC ower were good. It turned out that the IM4 whitening gain was reduced by 6 db on Saturday night at about 9.15 LT (2014-11-16 05.15 UTC). We don't know why that happenened and if something else was also changed at the same time. For the moment being, I changed the gain back to the previous value.
This was my fault - apologies for not putting in the log. The IM4_trans quadrants were saturating on Saturday evening, during the time when the CARM offset was being scanned to lock the arms; when I saw this I reduced the gain on IM4_trans by 6dB to bring the signals away from the rails. (The IMC WFS were saturating too, but since they are in-loop I didn't adjust them.)
Sheila, Dan, Evan
Yesterday and today we have tried to get the interferometer into a state where we can lock ALS with a CARM offset, lock DRMI on 3f, and then reduce the offset while staying locked. The lesson learned this weekend is that it is taking too long to get DRMI and ALS locked so that offset reduction can begin.
On the ALS side:
On the DRMI side:
ALS+DRMI, other:
For the record, upon further query Evan says "[The ALS DIFF used during the efforts documented by this entry] was with ETMX only. DARM feedback to ETMY was off." (I'd asked because I wasn't sure whether my new H1 ETMY UIM L2P decoupling filter from LHO aLOG 15049 had been tried / tested.)
Patrick, Dave
We have been scouring the houly conlog reports of EPICS PVs which change more than 500 times an hour for a reoccurrence of the rapid (many times a second) setting of the LSC ramped mux matrix which was observed last weekend. Unfortunately we have not seen anything yet, we will continue our vigilance.
FYI: here is the report from the last hour. (Remember that a 16Hz front end slow channel, if constanly changing, will give 16*3600 = 57,600 changes per hour.)
Process variables in conlog that have changed more than 500 time(s) in the last 3600 second(s):
H1:SUS-ETMY_L1_LOCK_L_RSET 57541
H1:SUS-ETMX_L1_LOCK_L_RSET 57541
H1:ALS-Y_CAM_ITM_SUM 2361
H1:ALS-Y_CAM_ITM_YAW_POS 2361
H1:ALS-Y_CAM_ITM_PIT_POS 2361
The L1 RSET frequency seems bad to me. I would interperet this to mean that the Guardian is resetting the filter history on that filter bank 16 times a second. Even if this filter bank is unused it indicates some rogue/bad Guardian script.
Rana is correct. Sheila, Alexa and Nic found the DOWN state of the ALS_DIFF guardian node was rapidly clearing the history of the ETM[X,Y]_L1_LOCK_L filters. This started at 8pm last Tuesday (11/11) and was ongoing until it was fixed a couple of hours ago. This highlighted a guardian logging issue with this node which Jamie is looking into.
J. Kissel I've measured new transfer functions necessary redesign of the H1SUSETMY UIM (L1) DRIVEALIGN frequency-dependent decoupling filters for L2P and L2Y. For the impatient, the last page of the last attachement shows a comparison between the proposed new design and the old, currently installed / in-use design. I have *not* installed this filter because I'd like to discuss the results with commissioners tomorrow morning (and it'll take me some time to resurrect quack to get these installed). In fact, this may argue for us to indeed just copy over ETMX's filter. Details: ----------------------------------------- I attach the measurement results (see 2014-11-13_H1SUSETMY_L1_drivealignfilters_rawdata.pdf), but the raw templates can be found here: /ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMY/SAGL1/Results/Data/ 2014-11-13_H1SUSETMY_L1_L2PY_WhiteNoise_ReactionChainTest.xml 2014-11-13_H1SUSETMY_L1_L2PY_WhiteNoise.xml 2014-11-13_H1SUSETMY_L1_P2PY_WhiteNoise.xml 2014-11-13_H1SUSETMY_L1_Y2PY_WhiteNoise.xml Not that I drove through the TEST filter banks (instead of having to turn off all the existing filters in the LOCK and DRIVEALIGN banks), and made sure that optical lever damping was OFF, and local damping was the LLO design as described in LHO aLOG 14959. I focus this entry solely on L2P, but the data exists for an L2Y filter if ever needed. in the first four pages, Pink curves are the current measurement, and black curves are what were used in the original design of the previous filter by Arnaud (see LHO aLOG 11832). The encouraging fact is that the data supports exactly what we'd model from the change in damping loop filter design (see green traces on pg 5 of first attachement to LHO aLOG 14959). As an aside -- just because Betsy made me think of it while chasing down slider values -- what effect does reaction chain alignment have on these off-diagonal, lower-stage transfer functions? One might guess that if the L2P cross-coupling isn't something fundamental but a result of dirt alignment affects between the coils on the reaction chain and the magnets on the main, then one would see a change in the L2P transfer function with different alignments of the reaction chain. The good news is that is doesn't have any effect -- see the last page of 2014-11-13_H1SUSETMY_L1_drivealignfilters_rawdata.pdf. I compare the same transfer function for 4 different reaction chain alignments. They still have the old dead-reckoned calibration factors installed, so we know they're not exactly [urads] but close enough for the scale of these offsets used; they use up about 1/2 the DAC range. As far as the design goes, I used the same transfer function fitting routine and filter generation software suite that (I think) Arnaud used, namely the function /ligo/svncommon/SusSVN/sus/trunk/QUAD/Common/FilterDesign/Scripts/Length2Angle_decoupling.m which is essentially a wrapper around the function, /ligo/svncommon/SusSVN/sus/trunk/Common/MatlabTools/happyVectfit.m which Keita's wrapper around the function, /ligo/svncommon/SusSVN/sus/trunk/Common/MatlabTools/vectfit3.m All tweaking of input parameters to the fitting routine, and plots were made using the overarching script /ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMY/SAGL1/design_H1SUSETMY_L1_L2P_20141113.m Notable key additions to the design software -- a computation of the final filter's impulse response, as well as a comparison to the previous design. As I know Arnaud had struggled, I struggled to find a fit with which I was really happy, namely because we all *know* that the L2P transfer function should fall off as some large power of frequency (again, see the models of these TFs on pg 5 of LHO aLOG 14959), but I can't find a set of parameters happyVectFit / vectfit3.m that allows for such a roll-off at high frequency. Indeed, the height of the final flat part of the L2P fit transfer function directly determines the high-frequency gain of the decoupling filter, and directly impacts the impulse response time. I settled on a filter that does a poor job at reproducing the > 2 [Hz] behavior of the L2P measurement, BUT I think it's OK because (a) the small magnitude of the high-frequency causes a significant drop in impulse response time, and (b) At the UIM, we're typically driving below 2 [Hz] -- especially if the crossover remains at 0.9 [Hz] as quoted in LHO aLOG 15025, so the accuracy of inversion is not so critical. Though I didn't find any design aLOGs, this seems to have been the philosophy behind the ETMX UIM L2P filter because it ends up looking strikingly similar. I should say also that having to tune the uncertainty tolerance, the order (i.e. number of poles) in the filter, the tolerance for when to throw out adjacent poles and zeros, the frequency range over which to fit -- all free parameters into happyVectFit.m -- as one has to do to get what one wants in the end really destroys the advantage of automating the fitting process. If only there was a better way...
I've installed the filter into H1:SUS-ETMY_L1_DRIVEALIGN_L2P filter bank, in FMs 5&6 via autoquack.
HOWEVER
(1) I have not tested the filter performance with transfer functions or step response
(2) I've split the L2P and (invP2P * rolloff) into two filter banks. Even with the rolloff, the (invP2P * rolloff) filter has more zeros than poles, so foton has added a whole bunch of high-frequency poles to compensate. Though the *total* filter's -- L2P * (invP2P * rolloff) -- number of poles is equal to the number of zeros, I think the high-dynamic range, zero pole, cancellation between the two filters isn't working out. This is resulting in a HUGE calculated impulse response, which may or may not be real.
(3) This is my first time using quack to install Matlab filters into foton in a *really* long time, so I'm nervous of its functionality.
I attach a series of comparisons with ETMX's filter. The bode plot, in the frequency band we care about (pgs 1 and 2) seems reasonable. Both ETMX and ETMY have some high-frequency garbage in their impulse response, but the overall amplitude of the impulse response in ETMY is many orders of magnitude larger than that of ETMX (!!!). Further, taking the transfer function out well beyond the Nyquist frequency, we see that they behave significantly differently at the Nyquist frequency (~7 [kHz]).
As such, I'm still entirely unsure that this new filter gunna work, and may require further tweaking.
Anyways, these are installed in
FM5 "UIM_L2P"
FM6 "UIM_invP2P"
and should be turned on TOGETHER, with no other filters engaged in the module, if one would like to be bold and try it over the weekend.
P.S. in case you've forgotten, when you autoquack in a filter who's name has a space in it, and you try to load the new filter coefficients, Foton will hate you. You will, at least, get a nasty-gram on the GDS_TP screen's filter message, ${IFO}:FEC-${DCUID}_MSG saying "bad filter coeff definition on line 259 in .txt," pointing you to the exact line of the problem. So with Jim's confirmation that "Oh yeah, you DEFINITELY don't wanna do that." I changed spaces to underscores and all worked fine.
These filters did indeed run away and saturate the DAC. (The EPICS values were even showing “-Inf,” I’d never seen that before). I don’t think we can get away with more zeros than poles in a filter module, even if they are compensated by some other module.
I took the two filters (FM6 and 7) and viewed them in the foton “Multiple” mode, then I took the output in the “Alternate” box of foton and copied that into FM10.
When I tried this, instead of a huge mess I got a nice step response according to foton, and the frequency response was the same as the two original filters. As seen below (new is red, old is blue):
However, stupid foton wouldn’t save the filter (but gave no error message). In the end, I had to take this “Alternate” output and split it up into two filters, being careful that each had more poles than zeros. This allowed me to save the foton file and load the filters into the frontend. I tested that the impulse response behaved nicely by putting an offset into the filter bank.
The two filters are in FM5 and FM10. Their names reflect my frustration.
K. Venkateswara, J. Warner
The BRS damper (see 14388) has been malfunctioning since yesterday afternoon. It's controller is an open loop system that assumes it is in the centered position when it is first started. Occasionally it can get restarted accidentally when the insulation box is bumped (probably due to a short somewhere, but I'm not sure where). It looks like this happened yesterday around noon.
This requires a manual re-centering of the turn-table for the feedback system to work as designed. I've attached the directions to do this. Jim will do this later today or tomorrow.
I checked on the BRS this morning, and found the code frozen. I followed Krishna's fairly straightforward document up to the point of opening the box and found the masses 90 degrees out of position (NW/SE instead of SW/NE), I was unsure of what to do, so I closed up, restarted the code and called Krishna to consult.
Afterwards, I went back to EX this afternoon, and re-aligned the masses on the damper. I did this by first modifying the code as described in Krishna's document, then opening the large insulating panel on the west side of the BRS enclosure, and just turned the turn table until the masses were approximately SW/NE. The turn table moved quite easily, I was expecting more resistance.It was dark down there, so it was hard to get a good picture, but I've attached an image taken from the west side of the BRS enclosure, showing how I left the masses. I then went back to the rack, re-commenting the appropriate code and restarting. Haven't had a chance to check on the BRS since then, but it looked okay from what I could see on the laptop readouts.
Looking back at the data, it seems that the changes made in the morning, resulted in a positive feedabck loop leading to large amplitude oscillations in the BRS. The changes made in the afternoon reversed it but it took ~5-6 hours to damp. However, the software appears to have crashed again later. Tomorrow, I'll investigate for intermittent shorts or other grounding issues.
