After our modifications to the DRMI LSC thresholds and loop gains on Thursday night, we had several short locks. Yesterday, we saw the same good locking behavior again. We found that the reason for the short lock duration was a typo in our trigger matrix: the PRCL boosts weren't getting turned on. After switching those and tweaking some filter shapes, the locking is fast and the locks last a long time.
The first attached plot shows the minute trend of our 4 hour lock last night. The 6 WFS loops are engaged during this time, as well as the DC centering. Near the end of the lock, there is a SRC mode hop and it stays locked like that for 1 hour. There are 20% fluctuations in the SPOP18 and SASY90. The control signal plots in the left hand column are scaled so that the full y-scale is approximately the full suspension DAC range (there is a divide by 4 from the LSC to the SUS DAC outputs). The MICH and PRCL loops are pretty calm, but there are several large transients in the SRCL loop which, I believe, corresponds to the TEM00 -> TEM01/10 mode hop.
The second PDF shows the error and control signal spectra for the DRMI LSC loops during a non-hopping epoch from last night.
PRCL: The error signal is white, so not much to be gained by changing the loop shape. The control signal is dominated by 0.05 - 0.5 Hz as is expected from the ISI performance.
MICH: The error signal is dominated by sub-Hz stuff, so we could squash it better with a 1:0.1 zero:pole stage if we found it necessary. The MICH control signal is dominated more by the sub 0.1 Hz motion than anythin else in the DRMI. Usually we think this is the seismic amplification produced by the seismic isolation system.
SRCL: Similar story to PRCL, but more noise from <0.1 Hz than 0.1-0.5 Hz.
In all spectra, the large line at 131 Hz is a calibration line used for making sensing matrix and demod phase measurements.
After the lock loss, it never got it itself together. The ASC loops stayed railed and kept it from being aligned enough to lock.
Also, the Guardian had a filter writing problem and so it stopped restoring the correct IFO state after the DOWN state
(it was trying to set a filter which was under the control of the LSC filter module trigger logic and stopped because it didn't
get the correct readback. This fault condition should be added to Guardian to make the log file more informative and we
should fix this conflict in the LSC state defs).
After clearing all of that stuff and bypassing the Guardian temporarily, the Michelson, PRMI and DRMI all locked fine.
Our DOWN state triggering during DRMI lock acquisition is probably the main holdup in acquisition times now. Watching the Guardian log files I see that it often just attempts acquisition for several seconds before it then begins a ~30 second period of switching buttons on and off until its ready to lock again. We would be better off adjusting these state triggers such that it stays in the 'trying to lock' state all the time. It should only do any of the DOWN stuff if its gone up into M2 offloading or WFS boost turn ons. We may be able to get a 2-3x speedup by trimming these downtimes.
I had tried reducing the guardian "DOWN" time by sending the guardian to "LOCK_DRMI_1F" if the locking failed as it entered the wfs centering or offloading. This seemed to work on Friday, but there were/are still certain instances where the guardian goes to the "DOWN" state. I have attached an example from the log. Both "OFFLOAD_DRMI" and "ENGAGE_DRMI_WFS_CENTERING" are now suppose to return to "LOCK_DRMI_1F", but it seems that when there is a transition between these two states, it jumps to the "DOWN" state. Jamie and I will take a look at this on Monday.
Sheila and I looked at the Guardian script this morning; I had missed an assert function that would take ISC_DRMI to the "DOWN" state, I have changed this to the "LOCK_DRMI_1F", so hopefully this fixes the problem ....
model restarts logged for Fri 24/Oct/2014
2014_10_24 01:43 h1fw0
2014_10_24 21:43 h1fw0
both unexpected restarts
As reported in the previous alog (alog 14602), I analyzed several locking events in which the LSC controllers attempted to lock the DRMI.
All the data were taken before we made the improvement (alog 14602) in order to study how to make locking faster. The idea was to collect many cases such that we can try to understand what is going on in lock acquisition.
In summary:
Analysis of class 1 events
Class 1 events were defined to be the ones which showed high POP RF18 vaules (> 100 uW) for more than 0.2 sec and showed high AS RF90 values of more than 2000 counts when POP RF18 was also high.
The plot shown below is a x-y projectin of POP18 and AS90 to show how these events evolved in this 2-D space:
Each event has different color from some others. As shown, a typical tack of class 1 events is to go down toward lower right in the plot and then jump up to upper right relatively quickly. This means SRCL is the last degree of freedom which comes into the linear range.
Also I attach a collection of some relavent LSC signals in time series as attachement 4. As seen in the plot, the SRCL feedback saturate most of the time at the SRM DAC. This should be mitigated by some means in order to do an efficient SRCL control for locking. Also the upper right panel of the plot shows how long time SRCL stayed within the linear range. As shown, the staying time is typically order of 10 msec which is faster than the time scale of the SRCL control (~ 50 msec for the typical UGF of 20 Hz) or comparable. So this is not great.
Interestingly, most of the events showed a dip in the POPRF18 signal (upper left panel) roughly 10 msec after POP18 rose up. This may indicate some kind of misalignment as the LSC controller push the optics.
Class 2 events
They are the events in which we were not lucky enough to have the SRCL in the vicinity of the linear range. The plot below is the 2-D representation of how they evolve:
The typical track looks different from that of class 1. They stay in the lower right part of the plot most of the time.
Rana pointed out some rookie mistakes in the filter banks for the QPD centering in HAM6 (AS WFS and OMC QPDs). We changed the DC3_P,Y loop and the OMC ASC loops to fix the following:
A comparison of the old and new filters is attached; the old version is in blue. We lose phase at 1-10Hz but it shouldn't be a problem for these simple loops. The DC4_P,Y filters still need to be changed, but the DRMI is under guardian control and sometimes the loops are turning on, so I'll wait for a quiet moment to fix them. (Also the guardian filter settings are all wrong.) Need to remember to add a 60Hz comb to these loops, too.
Alexa, Rana, Evan
This afternoon, we spent some time improving the ASC for PRMI.
We balanced the amplitude response of the segments on REFL_A_RF9. We let PRX swing freely, watched the fringes on each of the four segments, and then adjusted the gains until the responses had more-or-less equal amplitude. This gave a noticeable improvement in the POPAIR_B_RF18 buildup with PRMI locked on 9 MHz.
We wanted to increase the bandwidth of the PRCL loops to about 1 Hz, so that we could then implement MICH loops with a bandwidth of about 100 mHz. To this end, we did the following adjustments:
We then balanced the amplitude response of the segments on AS_A_RF36, usig a similar method with free-swinging SRY (and we double-checked using free-swinging Michelson).
With the PRCL loops active, we engaged the MICH loops. Everything seems to work fine for about 1 minutes, until the MICH error signals started to run away.
IMPORTANT: for both MICH and PRCL ASC loops, we now feedback to M2 stage instead of M1.
We updated the ISC_CONFIGS guardian script to include a PRMI ASC state. This handles the ASC in/output matrix, the ASC filter, and the suspension feedback to the M2 stage instead of M1. The DOWN state, restores the suspension feedback to M1. Right now, I have also copied and pasted this to the DRMI ASCs, since we expect to use something similar. The old code is commented so we can always revert.
PRMI was locked on the SB with both MICH and PRCL ASC loops turned on from October 25th 2014 00:54:10 to 0:55:30 UTC.
We should measure the full sensing matrix between REFL/AS and PRM/BS, and use it to decouple these loops. Then apply this to DRMI. For DRMI we should include a SRCL loop, and possibly also feedback PRCL to PR2.
Gains (for reference)
ASC-MICH_Y_GAIN = 0.003, ASC-MICH_P_GAIN = 0.003, ASC-PRC1_P_GAIN = -0.03, ASC-PRC1_Y_GAIN = 0.1
We closed the PRCL, MICH ASC loops with DRMI locked. The configuration is the same as above except with the following gains: PRCL1_P_GAIN = -0.006, PRCL_Y_GAIN = 0.001, MICH_P_GAIN = 0.003. MICH_Y_GAIN = 0.003.
We have now closed the SRCL loop with DRMI locked. We are using ASC_AS_A_RF36_I as our signal, and acutate on SRM. We have FM1, FM2, FM6 engaged, and feedback to SRM_M2_LOCK_P/Y, which has the f^2 filter installed a described above. The gains are (P,Y) = (0.3.,1.2). I have attached a screen shot. The DRMI guardian reflects these changes.
With these loops closed, we've now had a stable DRMI for over 2.5 hours. POP18 and SASY90 are both pretty steady throughout with the 6 WFS loops running. We had a coupled HOM hops in the SRC, but they happened for a few seconds and didn't cause lock loss. Some were associated with us turning on excitations or fooling around with the ASC feedback gains.
Measured the LSC loop gains (on the 1f REFLAIR PDs). UGFs are 75, 10, and 50 Hz for PRCL, MICH, and SRCL respectively.
We are leaving it in DRMI with ASC engaged. Please feel free to do some DetChar. Hopefully our Guardian settings and Limits will keep things safe when the lock breaks.
8:00 LVEA is LASER HAZARD with Limited Access 8:20 ISS Diffracted Power was set to the proper values range. 9:00 Quick visit to the LVEA (with commissioner’s authorization)- Hugh/Mitchell. 9:10 Back from LVEA – Hugh/Mitchell 9:30 Testing PCAL channels at EndX VEA – Patrick/Shivaraj 9:35 Optics alignment on ISCT1 table by HAM1 – Kiwamu 9:44 Heading to EndY – Kyle 10:05 Returned from the LVEA – Kiwamu 11:09 Back from EndX – Patrick,Shivaraj 11:42 Kyle is back from EndY 11:45 Heading into the LVEA (checked with commissioners) – Betsy 13:51 Heading into the LVEA (checked with commissioners) – Travis/Krishna 13:58 Back from LVEA – Travis/Krishna
Attached is ASD comparing HEPI L4Cs on ETMY & ETMX. I'm still working out the calibration but for comparison it is something. It is pretty subtle mostly but ETMX is quieter than ETMY. Not so subtle in a few places: 1 hz ...
Summary of results:
Details:
In response to Sheila's alog 14416 indicating high winds, DetChar looked at the transient motion behavior of ETMX and ITMY during high local wind speeds and compared it to quieter night time (low wind speed and BLRMS motion) from two days before.
The ETG (Event Trigger Generator, or single-ifo burst pipeline) Omicron was run on a two hour stretch of both windy and quiet times for a subset of DOFs on all stages (ground motion, HEPI, ISI ST1, ISI ST2). Omicron searched for transients (bursts of excess signal power, distinct in time and frequency) between 0.1 and 64 Hz.
Take away: If we find the transient motion events in optic table motion associated with high wind speeds couple to DARM, feed forward may be a good option to mitigate this effect.
Kiwamu, Alexa,
This morning, we moved IM4, PR2 and PR3 in order to reduce the amount of the clipping that Keita found the other day (alog 14567).
I moved PR2 and PR3 approximately by +150 and +15 urad horizontally which are the angles that we tried the day before yesterday. As a result, this increased the recycling gain for the 45 MHz sidebands from 21 (alog 14532) to 25.
We may want to go further in these angles for more improvements, but first I need to go through some careful analysis to see if all the observed numbers makes sense.
(some details)
Before starting the project, we realigned the X arm using the green light. The idea is to use the X arm as a reference for recovering good alignment for the infrared light around the corner station. Also we checked the location of the ALS X green light in the GigE camera which can serve as a reference when we touch a picomotor. According to the centroid fitting, the ALS X beam was at (X.Y) = (406.5 pix, 239.7 pix).
After moving PR2 and PR3 to 784 and -162 urad respectivly in the YAW bias sliders, as had been observed, we lost the POP beam at ISCT1. So we moved the picomorot in HAM3 by (dX, dY) = (900, 0) counts using the ALS X green light as a guidance. After picomotoring, we confirmed that there was no clipping according to the camera images. Note that it seemed that this operation somehow introduced vertical displacement slightly in the POP and ALS paths. We did not correct the vertical displacement as it was not considerably large. We locked the infrared laser to the X arm in order to automatically fine-tune the angle of IM4 and PR2 using the REFL WFSs. At this point, we went to ISCT1 and realigned the POP path. In addition to the YAW displacement that we had to correct on ISCT1, the bottom periscope mirror of the POP beam was clipping the beam a little bit. So we steered the top periscope mirror to correct it. Note that the spot position on the top periscope mirror seemed fine and therefore we did not have to move the location of it. We did not correct the ALS path, but it looks like at least the X green light can hit the TRX diode.
We did an OMC scan to measure the power recycling gain. I repeated the same process as we did on this past Monday (alog 14532). The highest DCSUM I obtained for the 45 MHz sidebands was about 7.5 mA when the PRMI was locked. Note that the PRMI alignment was optimized by maximizing POPAIR_RF18 manually. A single shot measurement for the 45 MHz sidebands gave us DCSUM of 0.27 mA. Using the equation in alog 14532, we estimated the recycling gain to be 25.
After moving IM4, PR2 and PR3, the PRC build-up observed by POP18 increased roughly from 170 to 200 uW. Good.
(How much did we shift the beam on PR2 ?)
Actually, this is unclear to me at the moment. I need to look into this issue more carefully. Basically, we seem to have a contradiction between two expected shifts, one is a value derived from the IM4 angle and the other from the PR2 angle.
If we believe that the IM4 bias is accurate, we must have moved its angle by more than +500 urad. This means that the spot positin on PR2 must have moved by ~ 2 x 500 urad x 15 meter = 15 mm toward the positive y-direction, which sounds too large. On the other hand, if we use the PR2 angle instead, my ABCD analysis suggests an expected shift of 0.5 mm on PR2 toward the positive y-direction. So they don't match up at all. Something is wrong. Since both expected shifts are based on the suspension biases, we probably should check the accuracy of them.
R. Schofield, K. Venkateswara
The BRS damper was working incorrectly over the last few days. As described in 14388, the vibration from the damper occasionally drives up the beam-balance's fundamental mode. This lead to incosistent damping. On Wednesday night, Robert Schofield and I tried adding some foam under the turn-table platform, which seemed to help. The BRS damped down correctly that evening. The next day, it failed to damp again. I noticed that the foam had gotten squished almost completely, thus not providing much isolation.
This morning, I made three small U - shaped springs out of steel shims and laid them under the platform, along with some foam for damping. The damper seems to be working correctly for now. I will monitor it over time and check to see if this is a suitable long term solution.
ERDF expects to operate the pit through the day today. Hanford container hauling will occur on day shift and swing shift today. There's no indication that ERDF will operate tomorrow, 10/25. The attached plot from last week shows a couple of interesting features. 1) The Wednesday 10/15 1-3Hz noise doesn't show the typical truck-induced bimodal day shift/swing shift appearance. The Wednesday noise instead is just a single hump that rises above the other weekday levels. The Hanford haulers were mostly shut down on 10/15 due to high winds. Wednesday noise appears to come from winds at LHO. The 0.3-1Hz trace, which isn't very sensitive to trucks, tends to corroborate this. 2) On Saturday 10/18, the trucks did not carry containers back and forth to ERDF, but crews at the pit emptied containers and ran their bulldozers and compactors until roughly noon. The plot provides the somewhat rare opportunity to see pit noise in the absence of transportation noise.
Patrick and Shivaraj We updated the PCAL library to inlcude calibration of temperature and OFS gain channels and rename OPTICALFOLLOWERSERVOOSCILLATOR channel to OPTICALFOLLOWERSERVOOSCILLATION. We committed the changes and restarted EPICS database to update these changes. After that we burtrestored h1ecatx1 and h1ecaty1 to an earlier time this morning. Everthing seemed went fine.
re 14594, the WHAM2 ISI config change attempt yesterday. Attached are performance spectra from this moring at 3am, and at 8am this morning after I switched the ISI back to the same configuration before we started--that is with the X Y Z blends switched from 01-28 to 100mHz.
I'd say the differences are minor and no great improvement but the ground noise is also slightly higher at 8am. Plus it would be a lot easier with overlay plots.
I'm pretty sure we've decide to actually run with all the HAM ISI dofs to be the 01_28 blends. THe X Y Z dofs being in 100mz is a left over from when Sebastien was working on Sensor Correction.
HEPI | Impact | ISI | Impact | aLOG | |
HAM1 |
Wrong | Magnitudes are correct so displacements are right but the rotational dofs have the wrong sign wrt RHR. | NA | iLIGO Platform | |
HAM2 | Wrong | Ditto | Wrong | Rotational dof signs are wrong for RHR | |
HAM3 | Wrong | Ditto | Wrong | Ditto | |
HAM4 | Correct | Signs & Amplitudes are right for examining platform movement | Correct | Signs & Amplitudes are right for examining platform movement | 14383 |
HAM5 | Correct | Ditto | Correct | Signs & Amplitudes are right for examining platform movement | 14480 |
HAM6 | Correct | Ditto | Correct | Signs & Amplitudes are right for examining platform movement | 14516 |
ITMY | Wrong | X & Y dof mags wrong. ~x2 | Correct | Signs & Amplitudes are right for examining platform movement | 14578 |
BS | Wrong | Rotation dof mags are incorrect, RZ x4, RX & RY x2 | Wrong | X Y RX & RY signs are wrong putting positional study in jeopardy | |
ITMX | Correct | Signs & Amplitudes are right for examining platform movement | Correct | Signs & Amplitudes are right for examining platform movement | 14535 |
ETMX
ETMY |
Correct
Correct |
Ditto
Ditto |
Wrong
Wrong |
X Y RX & RY signs are wrong putting positional study in jeopardy
Minor errors in off diag elements, basic study use okay but fixing will require completesweep through system. |
Above is an update to my 14373 SEI Matrices Status. The Blue Entries are new status since the 8 Oct report. Greens are all good and the reds are Platforms still requiring attention.
The ISS Diffracted Power was out of range this morning. It is been set to the proper values.
We have limited access to the LVEA today. Therefore, try to avoid going in by any reason.
Hugo & Hugh
With new ISI TFs from Sept 16 when the ISI was Under Vacuum with the HEPI Position Loops closed, I made new Level 3 control scripts for HAM2. I took the ISI down with Guardian pausing the manager and putting the ISI in READY.
In this state, the local 2 cartesian (& back) matrices were corrected--the X Y RX & RY signs were incorrect-- using the Populate_Matrices_HAM_ISI.m function. As expected, the cartesian sign of these dofs switched and the Target Position was correspondingly changed (by hand.) I then loaded the new filter file. Everything as expected so far. When the ISI Guardian was set back to High Isolate, it performed as expected until it was changing the horizontal dof gains from 0.01 to 1.0. At this point it trips in a fraction of a second on the GS13. In subsequent attempts it almost always repeats this behaviour although sometimes the trip was the actuator rather than the GS13, see the attached for example and zoom in.
OK so I screwed up the controllers. We tried level 2 controllers. Wait Guardian isn't set up to do level 2 so I try the old ISI Command scripts: yes it works on level 2 & level 1...
So Okay, I must have screwed up the controllers. I reverted back to the achive file that was made Oct2 10:16: still Guardian trips the level3 attempt at the same spot... revert back to Oct2 10:13 (what's up with that?.) Same result.
We try some manual turn on, that is 1 dof at a time, hey we can isolated this way. Then we take it down again and try with Guardian, no, it still turns on the Vertical Dofs fine but trips when switching the Horizontal dofs from0.01 to 1.0 gain.
Then! Then we try level 3 with the old ISI command scripts--it works! We turn Guardian back on, take the ISI down to ready and then attempt to get back to Isolation, nope, sorry, still trips. Back up to level 3 Isolation with the old ISI command scripts. This is where it sits...Strange.
Strange in that the ISI command script turns RX & RY dofs on first while Guardian does Z RX & RY first. Then when the command scripts have to do Z X Y & RZ, Guardian only has to do X Y & RZ. Plus, guardian does everything sequentially as in it waits for boosts to be on before he next step whereas the old command scripts are often doing more things at once such as engaging the boost while starting the Alignment bias servoing.
So we have to conclude that something must be up with Guardian although it seems unlikely. I restarted all Guardians 16 October and all went back to high isolate with no problem.
We have a before these changes performance plot. We'll take another look after things settle down and compare.
Summary
Matrices and Controllers reverted but ISI will not Isolate with Guardian. It does Isolate using the old ISI Command Scripts.
re "I restarted all Guardians 16 October and all went back to high isolate with no problem."
I actually didn't bring the isolation down. The Guardian was restarted without impacting the platform, so it could be that there is a long standing issue with Guardian. Based on the Watchdog trip indicator, Oct 2 was the last time the HAM2 ISI tripped but of course that doesn't mean someone might have run the ISI down with the Guardian. There are some ODC BIt changes around 16 Oct...
It may not be a fair comparison: The first attached plot is the ground motion and HAM2 GS13 spectra at 3 this morning with 100mhz blends on the X Y Z dofs, 01_28 blends elsewhere. The second plot is from 1430pdt where we have 01_28 filters on all dofs. Of course the input ground motion is different,there is more motion on the ISI X Y Z dofs in the 1 to 10hz band. The other dofs are pretty similar. We'll put these blends back and check when the commissioners can deal with the change.