As requested by Jim, I loaded the filters in BS HEPI and then ran the Switch_To_Test.py for setting the BS ISI to the test configuration at around 9:05 in UTC or 2:05 in PDT. Note that the oplev damping loops are running.
Peter F, Rana, Kiwamu,
As a preparation for the full lock study, we experimentally tried adjusting good coefficients for decoupling MICH from the other two in the LSC output matrix. Here are the results.
Precision for the PR2 and SR2 coefficients seem to be roughly a few % and 10 % respectively.
(Adjusting coefficients using the realtime lockin oscillators and demodulators)
We used the LSC realtime lockin oscillators and demodulators to find out the good output matrix for MICH. We drove BS at 131 Hz with an amplitude of 1000 cnts. At first, in order to find out the coefficient for removing PRCL out of MICH, we locked the sideband PRMI. We shook the M2 stage of both BS and PR2 in the longitudinal while changing the coefficient for PR2 manually until in-phase signal vanished at REFLAIR_RF45. We, of course, assumed REFLAIR_RF45 to be well tuned in its demod phase. Before the measurement we copied BS's inversion filter over to PR2 such that their frequency responses as seen from the LSC frontend are identical. In PR2, all other filters should be off and gain should be just unity.
Moving onto DRMI, we then shook PR2, BS and SR2. The coefficients for PR2 and BS were set to 1.0 and 0.02 so that we shake MICH more purely. Experimentally adjusting the coefficient for SR2, we found it to -0.014. The sign is negative because the effect of BS onto PRCL and SRCL are different and therefore it makes sense. We don't know why the PRC and SRC coefficients are different by 40% or so.
(Some tricks in SR2 M2 stage)
So for the setup in SR2, since one of the coil actuators had been dead, we used only LL and UR coils such that they still actuate without a large cross-couplings to the angular degrees of freedom at a cost of losing the half of actuation efficiency. This factor of two was then compensated in the Euler matrix by increasing them by a factor of two. We also copied the same BS inversion filter to SR2.
Rana, Kiwamu, Lisa, Alexa, Sheila, Evan
We took some time to measure the 1f and 3f DRMI sensing matrices.
To do this, we used the digital lock-in oscillators on the LSC screen to feed back onto some of the DRMI optics (PR2, SRM, BS, and a combination of 1×BS + 0.02×PR2 that we refer to below as BS+PR2).
The procedure was as follows:
cdsutils servo -r LSC-LOCKIN_1_DEMOD_5_Q_OUTPUT -g -10 -s 0 -t 100 LSC-LOCKIN_1_DEMOD_5_PHASE.z avg -s 8 to average them.Results for the 1f sensing matrix are as follows. The drive amplitudes have been divided out (and the entire matrix normalized).
| 1f | PR2 | SRM | BS | BS+PR2 |
| 9I | 2604(2) | 0.136(4) | −51.60(9) | −3.56(2) |
| 9Q | 107.5(9) | −0.0364(3) | −2.254(15) | −0.035(2) |
| 45I | 2015(4) | −3.70(4) | −39.69(6) | −5.51(5) |
| 45Q | −693(7) | −0.469(6) | 29.58(8) | 18.70(5) |
The matrix elements for SRM (in red) are probably bogus, because we were saturating the SRM actuator while driving.
Then we repeated this for the 3f signals, with REFLAIR27I→DEMOD 11, REFLAIR27Q→DEMOD 12, REFLAIR135I→DEMOD 13, and REFLAIR135Q→DEMOD 14. The drive amplitudes were 15, 9333, and 1999 counts for PR2, SRM, and BS. The results are as follows. Again the drive amplitudes have been divided out (and the entire matrix normalized).
| 3f | PR2 | SRM | BS |
| 27I | 9400(40) | −2.560(11) | −173.5(3) |
| 27Q | −154(8) | 0.618(9) | 14.72(5) |
| 135I | 2940(50) | −7.96(7) | −58.3(5) |
| 135Q | −22 550(90) | −0.17(7) | 543(2) |
DRMI lost lock before we were able to get the BS+PR2 measurement for 3f.
It seems as though an oscialltion in the BS (visible on the stage 2 GS13) caused a DRMI lock loss. The oscillation seems to be in the ISI channels, but I don't see it in the control signals or on the OpLevs.
HAM3 also just tripped, we don't think there is anyone in the LVEA right now
Chris, Kiwamu, Suresh, Sheila
We have had several more of these odd HAM3 trips tonight. We chased it down to a problem in the IMC_LOCK guardian, where a huge impulse is sent to MC2 in the down state. We are strongly suspicious of a the lines where the gain of MC2_M2_LOCK_L and M1_LOCK_L are set to zero, and the integrator in M1 and boost in M2 are switched off. Both of these filters are switched with zero history, immediately. We changed the boost in M2 to have the output ramped off over 5 seconds. We also changed the code to nicely clear the integrator in M1:
ezca.switch('SUS-MC2_M1_LOCK_L', 'INPUT', 'OFF')
ezca.get_LIGOFilter('SUS-MC2_M1_LOCK_L').ramp_gain(0, 5)
ezca['SUS-MC2_M1_LOCK_L_RSET'] = 2
ezca.switch('SUS-MC2_M1_LOCK_L', 'FM1', 'OFF')
ezca['SUS-MC2_M1_LOCK_L_GAIN'] = -1
We haven't loaded this, since by the time we figured this out we had already moved onto locking DRMI.
One thing that is different tonight is that we started trying to use the ISC_LOCK guardian, which is managing the IMC_LOCK guardian. It's not clear why this would have caused these kicks to the IMC. There is some kind of bug in the way that the ISC_LOCK is managing its subordinates, so sometimes they hang up, and do not leave a state which has finished even though there is a path to the requested state.
Fixing this problem in the ISC guardian may help to prevent the HAM ISI trips that we have occasionally had on restarting beckhoff code.
I've set up a script to turn on the RH on ITMY around 2:30AM for 3 hours. To kill it if desired, either log in to operator1 and kill the tmux process, or just turn off the RH after it turns on.
J. Kissel
I attach pictures of the switchable configuration boards of all optical levers in the corner station. My fingers are supposedly indicative of the rack / chassis into which they're plugged, but I add a legend here anyways with the real rack numbers, and I include the exact switchable filter status. Note, I only report one of the channels' switch status, because (thank god) in every instance, all four channels are configured in the same way.
Rack Board Optical Lever Config Board |------------------ Switch Status ----------------| Centered? Raw Segment Counts
Number Installed? Board Switch
CH Name CH Name Function HI LO ON/OFF
SUS H1-R1 1 HAM2 Table? No N/A (HAM2) No 1:-339 3:-2243
(Pg 1) HAM3 Table? 4:-399 2:-3548
2 HAM2 Table? No N/A (HAM3) Maybe 1:-1260 3:-1167
HAM3 Table? 4:-1921 4:-1594
SUS H1-R2 1 none? No N/A N/A
(Pg 2)
2 PR3 Yes B0 1 +24 [dB] X OFF PR3 Yes-ish 1:-5600 3:-7400
(pgs 3-5) B1 2 +12 [dB] X ON 4:-17800 2:-17500
B2 3 +6 [dB] X ON
B3 4 +3 [dB] X OFF
B4 5 (1:10) X ON
B5 6 (1:10) X OFF
B6 7 (1:10) X OFF
B7 8 Latch X OFF
Total PR3 TF = (1:10), G = 18 [dB]
SUS H1-R4 1 HAM4 Table? No N/A (HAM4) No ADC Noise
(pgs 6-7) HAM5 Table?
2 HAM4 Table? No N/A (HAM5) No ADC Noise
HAM5 Table?
SUS H1-R3 1 none? No N/A N/A
(pg 8)
2 SR3 Yes B0 1 +24 [dB] X OFF SR3 Yes-ish 1:-9100 2:-9200
(pgs 9-10) B1 2 +12 [dB] X OFF 4:-7040 3:-9200
B2 3 +6 [dB] X OFF
B3 4 +3 [dB] X OFF
B4 5 (1:10) X ON
B5 6 (1:10) X OFF
B6 7 (1:10) X OFF
B7 8 Latch X OFF
Total SR3 TF = (1:10), G = 0 [dB]
SUS H1-R6 1 BS Yes B0 1 +24 [dB] X OFF BS Yes-ish 1:-6800 2:-6300
(pgs 11) (pg 12) B1 2 +12 [dB] X ON 4:-7250 3:-7350
B2 3 +6 [dB] X OFF
B3 4 +3 [dB] X ON
B4 5 (1:10) X ON
B5 6 (1:10) X OFF
B6 7 (1:10) X OFF
B7 8 Latch X OFF
Total BS TF = (1:10), G = +15 [dB]
2 ITMY Yes B0 1 +24 [dB] X OFF ITMY Yes-ish 1:-2470 2:-2380
(pgs 15) (pg 16-20) (NOT AS SHOWN) B1 2 +12 [dB] X OFF 4:-2250 3:-2900
B2 3 +6 [dB] X OFF
B3 4 +3 [dB] X OFF
B4 5 (1:10) X OFF
B5 6 (1:10) X OFF
B6 7 (1:10) X OFF
B7 8 Latch X OFF
Total ITMY TF = (flat), G = 0 [dB]
SUS H1-R5 1 ITMX Yes B0 1 +24 [dB] X OFF ITMX Yes-ish 1:-1000 2:-700
(pgs 13) (pg 14) B1 2 +12 [dB] X OFF 4:-1000 3:-700
B2 3 +6 [dB] X OFF
B3 4 +3 [dB] X OFF
B4 5 (1:10) X OFF
B5 6 (1:10) X OFF
B6 7 (1:10) X OFF
B7 8 Latch X OFF
Total ITMX TF = (1,1:10,10), G = 0 [dB]
Note that I've left ITMY with all of its switches OFF because we want to assess whether the internal gain of the QPD head (see D1100290) is set to the "high" (100 [kOhm] transimpedance) or "low" (10 [kOhm] transimpedance) -- and I got booted out of the LVEA for locking the IFO. Seems like 2000 counts is a little to low, we really want something like 10000 [cts] per quadrant. We will adjust tomorrow.
I've also confirmed that all over-all transfer functions for each optic has digitally compensated for correctly.
Continuing a trend of stealing from LLO without shame, I looked at the L1 ISI foton file (via LLO's DAQ svn) to see what filters they were using for St1 sensor correction. I've attached some plots comparing some of the filters. It would be helpful if someone at LLO could summarize (again) what is being used in a complete and accurate way, because some of the filters have very similar names. FF is a common name for different filters. In the first plot, the green (I think) line is the sensor correction a version of the filter Rich Mittleman designed, that we have been using at ETMX with the BRS. The other lines, with an rdr are Ryans filters that I found in the LLO ITMY foton file, which I named based on what direction I found them. Second plot is the phases of same filters, with the same color scheme and order.
zpk's for the different filters:
rdr X,Y & Z Senscor
zpk([0.005868986282643151+i*0.005869222531082809;0.005868986282643151-i*0.005869222531082809;0;0;0;
0],[0.02121320343296193+i*0.0212131204375177;0.02121320343296193-i*0.0212131204375177;0;0;
5.326821252114442e-10;0.03999999945769176;0.0999999989897627;1.299999570000159;
6.999932741807108],6217465930.447066,"n")
rdr Y Senscor
zpk([0.006840399999901201+i*0.01879390059988029;0.006840399999901201-i*0.01879390059988029;
0.06062180000003715+i*0.03500000003808696;0.06062180000003715-i*0.03500000003808696;0],
[0.01368079999995818+i*0.03758769978878637;0.01368079999995818-i*0.03758769978878637;
0.03500000000007082+i*0.0606217999686012;0.03500000000007082-i*0.0606217999686012;
0.01000000066387688;0.03999999933609382],99.46935875676593,"n")
Mittleman Senscor
zpk([0.05785089999994956+i*0.06894400006181636;0.05785089999994956-i*0.06894400006181636;0;0;
1.054942208843087e-09;0.009999998945009785],
[0.003420200000103372+i*0.00939692986876447;0.003420200000103372-i*0.00939692986876447;
0.01267849999986304+i*0.02718920009938821;0.01267849999986304-i*0.02718920009938821;
0.04499510000006567+i*0.05362309993610679;0.04499510000006567-i*0.05362309993610679;
0.005000000000048375],0.006161864174656091,"n")
rdr Z Senscor
zpk([0;0;0],[0.003535529999897033+i*0.003535531890194072;0.003535529999897033-i*0.003535531890194072;
0.000868241000053361+i*0.004924041464289278;0.000868241000053361-i*0.004924041464289278;
50.0000000000001],252959718.8553666,"n")
The "Mittleman" and rdr Z" Filters (pink and green?) are both for use only with Z or a tilt correct tranlsation signal using Krishna's tilt meter.
The red, "rdr X,Y & Z" is Ryans 0.5Hz notch filter deisgned to be used in parallel with low frequency stage 1 feedback.
The blue is a substitute for the FIR filter chain(the FIR filter probably is better), obvously has a different sign from the others.
08:00 resarted Alarm Handlers; CDS overview: noticed some conflicting counts on fw1 and some timing flags at the mid station; Vacuum: noticed IP-01 flashing red. Sent out emails to the corresponding folks for a check-out.
08:30 ISS loop coming in and out of saturation. Diffracted power all over the place. I bumped the refsignal up a click and the diff power stabilized. I tried a series of clicks following and got the diff power back up to ~7.6%
09:09 Betsy and Travis out to West bay to work on 3IFO Quad.
09:10 Andres out to W Bay
09:20 Peter F out to LVEA
09:55 J Bartlett, also Andres out of LVEA
10:25 Ryan reported a possible network outage lasting ~10 minutes maybe upstream of PM&L
? Alastair putting power meter on TCS X arm table (WP 4925)
11:12 Robert working in beamtube between mid Y and end Y (shaking tests)
11:21 Dan to end X to look for equipment
11:56 Shivaraj to end X and end Y to mount magnetometers on tripods
12:38 ISS first loop went erratic again until ~
13:19 Karen at end Y
13:51 Karen out of end Y
14:30 Dan H out to HAM6
15:20 Rana brought to my attention folks entering VEA areas. Not everyone is reporting their desire to enter to the operator. Cleaning personell are calling after they've arrived at end stations.
The ISS loop was sporadic all afternoon
As a first step towards a fast shutter at the AS port, I swapped the QPD transimpedance amplifier box for ASC-AS_C for one with the SUM output spigots for the shutter input. From rack R5 slot 20:
Out: S1102836
In: S1301506
I also changed the P3 jumper on the OMC PZT LV driver board to the correct position (across pins 2 & 3). Now, a 0V input to the 'shutter logic' input on the front of the OMC PZT driver box is recognized as a 'fault' condition (i.e., with 0V input, the PZT shutter function is triggered). This draws the PZT1 drive down to 0V (from 10V). It also pretty well disables the dither output to PZT1, so no OMC locking until we connect the PZT shutter input to a 5V supply (like, for example, the shutter logic controller on top of ISCT6).
It looks like the ASC-AS_C sum is about the same as before the swap, but we probably need to redo the dark offsets.
J. Kissel, R. Abbott Was checking the status of the ETMX optical lever against Doug's status report (see LHO aLOG 14711) before I took some measurements using it, and found that the ETMX optical lever's compensation filters (i.e. H1:SUS-ETMX_L3_OPLEV_SEG?) have to filters, "gain2" and "gain1.4" (which are gains of 0.5 and 0.707, respectively) that are turned ON. I couldn't find any aLOGs about these filters, and it doesn't really make sense to me to compensate for any analog gain, given that we normalize by the SUM. Anyone know anything about this? %%%%%%%%%% EDIT %%%%%%%%%%% Doug's status chart reports values exactly backwards and bit-shifted up 1. "HIGH" bits are LO, and "B1" = B0. The confusion arised from Evan's convention for high vs. low, and that the switches are numbered starting at 0 on the board, and 1 right next to the switch. Therefore, the ETMX is compensated for correctly. Will post corrected assessment shortly. %%%%%%%%%% Historical Log, for future reference %%%%%%%%%%%%% Piecing together the puzzle, from Doug's aLOG (LHO aLOG 14711), and the various circuit schematics (Configuration Daughter Board D1001631, Chassis Assembly, ), Evan's recent PR3 tune up (LHO aLOG 14631), my aLOG from many moons ago (LHO aLOG 3619), and a phone call to R. Abbott, I can determine that having each channel in the following configuration (as indicated by Doug's aLOG, where B0-B7 is the configuration for the first channel, and the remaining 3 channels, B8-B15, B16-B23, and B24-31 are in the same configuration), Switch Function Setting Status Name HI LO B0 G = 24 [dB] X ON B1 G = 12 [dB] X ON B2 G = 6 [dB] X OFF B3 G = 3 [dB] X OFF B4 1:10 Whitening X ON B5 1:10 Whitening X OFF B6 1:10 Whitening X ON B7 Latch X ON Note -- and this seems to be a common misconception -- when the switch is LO (or "0", or away from the chassis as installed) the switch is ON / ENGAGED. Therefore, the total transfer function: G = 36 [dB], two stages of (1:10) whitening. (filter zero and pole notation is (z:p) -- note that the poles and zeros is INCORRECT [i.e. inverted] in my old aLOG) Thus, the current compensation scheme, with FM1 (10:1) with FM4 and 5 (the "gain" filters mentioned above), it seems as though the switches have been interpreted in exactly the opposite fashion as the schematic intends. Even worse -- the LATCH is ON, which means it's actually telling the board to engage whatever the settings were *before* the latch were engaged, and the current status of the bits B0 through B7 DON'T MATTER. #facepalm Will get with Rana / Evan / Doug / Jason about this.
PeterF, Rana
We checked out the SR3 optical lever signals and determined that we needed to turn on one stage of whitening (1:10) and no whitening gain stages to get us cleanly above ADC noise everywhere.
Measured power output of X-arm laser to be 56.3W directly after laser. Thermopile in laser has a lower V/W sensitivity (0.116V/W) than others measured. I've updated the calibration to reflect this so that the channel H1:TCS-ITMX_CO2_LSRPWR_HD_PD_OUTPUT now gives correct laser output power.
I have also redone the calibration of the rotation stage using a second power meter on the table at the output to the CP. The rotation stage position for a minimum output was correctly set at (37 degrees) giving a zero reading on a 100W and 10W power meter heads (we can go to a more sensitive power meter also, but here I'm measuring the full range of the rotation stage). Maximum is seen at 82 degrees giving 5.44W.
Attached is a plot of angle to power for both power meters.
I've checked the calibration of the rotation stage, which was almost correct, and have slightly altered the maximum power output setting for this. Minimum power and minumum power angle were both correct.
It looks like we are seeing the rotation stage not always going to exactly the requested angle. This results in non-zero power output when minumum power is requested. This is not an issue with the power meter - it is real. So when there is a non-zero power reading on the power meter it means there is power going to the CP. Repeated presses of the 'Go to minimum power' button result in the rotation stage eventually going to the correct angle. If it still won't go to zero, then using the 'Home' button can reset the rotation stage position.
Observing the many trips HAM2 has experienced using the guardian recently, I've seen that after the first group of dofs are isolating, the remaining dof error points often grow large and when these dofs gains are ramped up, the ISI trips with a rung up GS13. I saw that the guardian was loading the reference locations between the dof groups engagement. Whereas the old seismic command scripts would only load the reference locations after all the dofs are isolating at the free hanging position. Remember, the command script was not having any trouble turning the isolation back on.
Jamie modified the Guardian to do the loading of the reference location at the end of the isolation process like the command scripts. When I got my window to test this, I actually forgot at first to restart the guardian. And, the first couple times guardian was able to bring it up! When I realized it wasn't doing it the new way, I remembered the restart, but I kept trying the guardian. On the third or fourth attempt with the guardian it tripped. I then restarted the guardian and then the guardian was successful at full isolation at least 6 times before the commissioners returned to the control room.
So again, my theory is, when the isolation loops are servoing to a place away from the free hanging position, the uncontrolled dofs error points can grow large. This is dependent on many thing and lots of luck good & bad. When the loop is engaged at low gain, the error point may be swinging about zero and if the loop gain is changed to 1.0 (from 0.01) at an unlucky time, bam goes the GS13.
Currently, HAM2 ISI is successfully back under Guardian conrol.
Here are trends of the HAM2 ISO INMONs, OUTPUTS and the location servo Residuals. I expected something to jump out in the OUTPUT or the residuals but nothing really at this resolution. Well at least the INMONs correlate to the trip out. You can see three successful isolations on these X & Y channels at the beginning of the plots. You can see the OUTPUT quickly grow large after the gain has ramped. On the fourth attempt, which is the trip, the INMON spikes much larger than before but I think it does so after the trip turn off...Yeah looked at full data and the large spike is after the OUTPUT drops to zero.
So well...I think my theory is sound but I don't think I've convinced anyone yet with data.
Keith T, Jim, Daniel, Dave
As Keith mentioned in yesterday's CDS meeting, the RFM IPC error rate at the SUS model is related to the CPU usage of the LSC sender. To show this, I've attached a minute trend plot of the past 7 months of IPC error rates at SUS-EX, along with the CPU-MAX of the LSC model. As can be seen, when the LSC CPU-MAX is regularly around 40uS, the receive error rate goes up from 2 to 10 errors per 16384 received packets.
At 10 errors per 16384 packets, the error rate is 0.06%. Can we determine if this is significant?
questions/ideas which have been mentioned when thinking about possible solutions:
reduce LSC processing (filter modules unnessary or unnessarily complicated) split LSC into two cores (again) delay the RFM receiver by one whole cycle (adds 60uS latency to control signal) replace LSC computer with faster hardware (being investigated by Rolf, Keith)Daniel, Dave
We have stopped the following CSD guardian nodes from running. They can be restarted at a future date if needed.
H1ECATC1PLC2 H1ECATX1PLC2 H1ECATY1PLC2 H1ISCEX H1ISCEY H1LSC H1LSCAUXIf we choose to restart these, they should be generically renamed (dropping the H1 in the name)
also stopped the following nodes:
LSC LSC_PRMI-VAR_FINESSE
Kiwamu did some work to figure out what output matrix values are needed to drive mostly MICH; it is 1×BS + 0.02×PR2 − 0.014×SRM. Rana then measured the sensing matrix with 333 counts on MICH, 17 counts on PRM, and 18999 counts on SRM (and without saturation). WFS were engaged, and the loops were notched at the drive frequency (131.7 Hz).
Here is the 1f sensing matrix, with the drives appropriately divided out.
And likewise for 3f.
Also last night, we took similar measurements of the PR2/SR2 portions of the DRMI sensing matrices while we tried bringing in the arms. Sheila started at ≈ 7.5 nm and over a few minutes brought the arms to slightly under 4 nm, at which point we lost lock (the conversion from displacement to detuning is 7 nm / Hzgreen). So the attached plots should be read from left to right.
The big jump at 6.5 nm is because Kiwamu had to tune up the DRMI alignment to prevent lock loss. So the jump in the sening matrix elements isn't surprising. Beyond that, the values appear more or less constant, to within uncertainty.