Kyle noticed that the web MEDM snapshots had stopped when script0 has issues yesterday at 6pm PDT. I have restarted them after the reboot of script0.
Installation Meeting Minutes 8:15-8:40 Fred, Richard, Aaron, Gerardo, Jeff B, Vern, Bubba, Travis, Hugh, Mitchell, Mike, Arnaud, Thomas, Daniel, David, Corey, etc. • Went over Work Permits • Work on HAM5 ISI (table balancing) – Hugh • BK hammering test on SR3/SRM suspensions (HAM5) – Arnaud • TCS work continues - David • Sled getting ready to be moved to HAM6 – Dan/Corey • Quad upper structure work in LVEA (West Bay) – Travis/Betsy • ESD driver work at End Y – Richard/Filiberto • Staging building activities ongoing – Mitchell/Scott LVEA is Laser SAFE
Moday and Tuesday, no restarts reported. Maintenance was deferred until the vent.
script0 was not running correctly, we ended up rebooting it via the front panel reset button.
Sheila, Lisa
Previous alignment positions were not saved correctly, apparently, so when the Guardian was restarted today we lost our references.
Daniel had recovered a good green alignment for both arms, but the IR beams where not very well aligned. We tried to look at the PRMI anyway but we realized that it was very misaligned and far from locking. Since a good alignment for IR had not been done in a while, we decided to realign everything from scratch.
Unfortunately we have been tripping SUS/ISIs several times today by engaging ALS DIFF during this alingment process, and as Arnaud reported, we were misaligning the X arm again with every kick..so the whole process took a while.
We leave the X arm with both green and IR aligned; the Y arm is ok fro green, but not for IR.
More alignment tomorrow..
On a positive side, we used dither + green WFSs for the X arm several times today, and we could consistently achieve a good alingment for green.
I accidentaly sent a transient to the suspension. It makes sense that we loose lock when I do something like that, and we give the suspension a kick, but it doesn't seem like this should trip the ISI.
There was an error, most likely from a divide by zero somewhere in the corner beckhoff. I reset all (including persisent variables) a few times, and then did a burt restore to 4 pm.
This morning, B&K hammer measurements were carried out on SR2 suspension cage located in HAM4. Attached plots are comparing SR2 force to acceleration transfer functions in two degrees of freedom X and Y (respectively corresponding to Longitudinal and Transverse suspension's euler basis -not global coordinates-), with similar measurements carried out on MC1 MC3 PRM.
The main/lowest structural resonance below 150Hz (97Hz in both directions) is well damped, and the Q is comparable with the one from other suspension cages.
[Method]
HEPI and ISI were unlocked during the measurement. A tri-axis accelerometer was mounted on the top corner of the structure (as shown in the picture). Data acquisition was made with the Pulse software on the windows laptop, and the template used was copied from T1000697, using a hammer trigger threshold of 10N.
Data collected lives under the svn in
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/SR2/BandK
[Sheila Lisa Jeff Arnaud]
Sheila and Lisa were having alignment issues in the X arm this afternoon, in the sense that some fine alignment was always needed after recovering from a lock loss. A good example is shown in the attached screenshot. In that case, diff lost lock, and kicked both etmx and etmy suspensions (top right plot showing the diff error signal in purple). At that point the green transmitted power in the X arm (bottom left plot) dropped to 0, and came back to 0.5 after the test mass settled down (instead of expected 1). This was due to the alignment : comparing the pitch of the test mass with the oplev before and after the event, we see that etmx pitched down by ~1.5urad (left middle plot in green). Neither ISIs HEPIs SUS were tripped during those times, the Ry DC values (corresponding to sus pitch) from ST1 ST2 ISI and HEPI were identical, etmx yaw was identical, itmx pitch and yaw were similar.
Sheila then used itmx (bottom right plot in yellow) to realign the cavity, pitching it by ~1.5urad which was just enough to bring the transmitted power back to 1 (bottom left plot in red).
It would be interesting to know if this is what happens all the time.
Wrong login, this was me.
Can you check if the ETMX keeps pitching down from lock to lock, or if it just snaps back? We are actuating to the top mass, so it is conceivable that a lock loss changes the alignmement through the relief of the length actuation. The problem is that the new alignment scheme feeds back to the PZT which will not snap back, but would follow an alignment drift over time when locked.
2 hour graph of X-arm locking with WFS off. Power drops to 0.5 in about an hour due to length to angle coupling. The tidal motion is about 60µm/h. The effect is clearly seen in ETMX YAW and to a lesser degree in PIT. ITMX yaw is flat. What's going on with ITMX PIT? No external drive was sent to ITMX.
- Keita to EY to work on WFS - Jeff and Andres to HAM5 - Betsy in the LVEA for INS work - Water delivery - Ln2 delivery to CP2 - Restart guardian nodes, the memory is getting full. Jamie is working on a long term solution to this - Tour through the LVEA at 4:30 PM PT
stage 1 actuators, this happened when diff lost lock.
The same thing happened again, this time the plotting script worked.
ETMY sus also tripped just now
Since Sebastian and Lisa have been trying to look into this, Rich and I had a look at some trends from around the time of these trips.
The ALS DIFF guardian down state exits too soon, moving on to the prepare state too soon (alternatively, it fails to clear the history of DARM, which is par tof the down state but isn't working.) This causes a transient to be sent to the suspension. Of course we need to fix the guardian, but also need to decide if these kind of transients should be tripping the ISIs.
The screen shots below show the input to the suspension from the LSC model, the outputs to the test mass and to the top mass, and the time of the WD trip for two of these trips.
Thank you for the follow up Sheila.
I'm still analyzing those trips to be sure that we have a full understanding of what's happening before taking any decision. I should have a full report before the end of the week.
As Dave reported earlier (alog 12078) we again experienced memory issues on the h1guardian0 machine over the weekend, this time associated with the BSC ISI guardian nodes. The symptoms are the same as what we saw previously with the so-called "CSD" nodes: node worker process memory usage and thread count shoots through the roof. The problem seems to have in fact started Thursday evening, roughly around the time I restarted all the ISI nodes to incorporate a seemingly inocuous improvement to the handling of the ISI "MASTERSWITCH" (alog 12036). This turns out to be the key to the whole problem.
Later, on Friday, Arnaud reported that some of the ISI nodes went into error because of channel access errors associated with the masterswitch (alog 12058). Associating this with the other issues was the eureka! moment for me. Here's a summary of the issue:
The problem has to do with so-called "IFO rooted" channels. Guardian modules can specify a channel access prefix, which is prepended to all further channel access calls (e.g. the prefix 'ISI-ETMX_ST1' will cause calls to "ezca['FOO']" to be converted to the channel 'H1:ISI-ETMX_ST1_FOO'). This has a couple of benefits, most notably that we can use the exact same code for systems that differ only in the prefix. However, sometimes it is necessary for a prefix-specified module to monitor the state of a channel that is not in it's domain (e.g. ISI_ETMX needs to look at the state of the ISI masterswitch at 'H1:ISI-ETMX_MASTERSWITCH'). This is done by specifying the full channel starting with a colon (ezca[':ISI-ETMX_MASTERSWITCH']), which tells the ezca module to ignore the channel prefix for this channel. We've been noticing problems with these IFO-rooted channels for a while now, but it now appears that those problems and the memory/thread usage problems are in fact related. Each IFO-rooted channel call is somehow creating a new PV object that is not being managed efficiently.
Last Thursday I modified the ISI code to start monitoring the ISI masterswitch, which is an IFO-rooted channel for those nodes, after which the ISI nodes started having memory/thread issues. Similarly, the CSD nodes connect to only IFO-rooted channels (and thousands of them) thus all the problems we've been seeing with those nodes. (Note: the CSD problem can be easily solved by properly stripping prefixes from the channels it accesses).
The solution for the ISI nodes was a targetted change to the function that checks the current value of the masterswitch so that the switch state is not checked for the ISI stages. This is a temporary change until we can get the core ezca handling of IFO-rooted channels fixed.
To illustrate the change, here is an ISI node running with IFO-rooted masterswitch checking. After just a couple of minutes, the memory usage (SZ, RSS) and thread count (C) are both creeping up:
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD controls 19314 1488 1 91613 27120 8 15:47 ? 00:00:01 guardian ISI_ITMY_ST1 controls 19342 19314 13 112312 34300 8 15:47 ? 00:00:13 guardian ISI_ITMY_ST1 (worker)
In contrast, here is the same node after the masterswitch check was removed, and after more time has ellapsed than the example above:
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD controls 20932 1488 0 91621 27004 7 15:49 ? 00:00:05 guardian ISI_ITMY_ST1 controls 20967 20932 9 109968 24760 0 15:49 ? 00:00:48 guardian ISI_ITMY_ST1 (worker)
Node the memory usage and thread count are lower and stable after a longer time. This is pretty conclusive to me that these IFO-rooted channels are indeed the cause of the problem.
I have committed the patch (userapps svn r8075) and restarted all nodes for BSC ISI stages (ISI_{BS,{I,E}TM{X,Y}}_ST{1,2}). This should eliminate both the resource problems with the ISI nodes, and the EzcaErrors associated with the masterswitch channels.
Now it's off to figure out how to fix the problem upstream in the Ezca object...
Both were T240 trips. There isn't a clear reason, the tour hasn't started yet, the door work is finished.
Something was tilting the platform
J. Kissel The Messages: (1) The effective bias voltage charge on the test mass is -28 [V_pk], which quite small compared to the BIAS range of 400 [V_pk] (and in conflict with Keita's assessment from the optical levers [see LHO aLOG 11905]). (3) The force actuation strength is asymmetric with respect to bias voltage, confirming Keita's result (see LHO aLOG 11872) (2) The force coefficient at 11 [Hz], measured as a function of bias voltage, is roughly 0.5e-10 [N/V^2], a factor of 8 below the expected 4.2e-10 [N/V^2] (expected from pg 7 of G0900956) DETAILS ------------- Brett: please proof read: ------------- Note, no linearization algorithm was used during this experiment. (1) Using the time-honored MIT method of measuring test-mass charge (e.g. from Brett, John, and Lisa), I drove a single frequency line into the control voltage H1 SUS ETMX ESD drive, looking at the X arm cavity length (as measured by green), while stepping the bias voltage through its entire range. As one steps down the bias voltage, we expect the linear response to drop to zero. Further, one can take the slope of the response as the bias decreases from both the positive and negative side and predict an effective bias voltage created by residual charge on the test mass. I attach the results. Using a linear regression, weighted by the coherence as follows, unc = sqrt( (1 - coh) / 2*nAvgs*coh ) weight = 1 / unc^2; I calculated the intersection of the two slopes to be -27.99 [V_pk]. I took an excessive amount of data points, because I didn't know what I was doing when I started and wasn't sure how the results would turn out. The template is here for excitation: /ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGL3/Data/2014-05-15_H1SUSETMX_L3_ChargeTest.xml For the record, I tried using the script recommended by Brett (LHO aLOG 11914), but after updating all the necessary channel names and input variables. it failed deep with in its subfunctions because of some xmlconv library that's now gone missing since 2009. I didn't have the time nor expertise to debug, so I just changed the bias by hand and measured the ten averages with the GUI DTT session, capturing references as I went, and exported at the end for processing. (2) No further details here, it's obvious that the drive strength is weaker with positive bias voltage than with negative. I have no good explanation for this. (3) From these displacement response results, one can also obtain the force coefficient for each bias step. The calculation is as follows: disp_mpk = 1e-12 * sqrt(2) * sqrt(binWidth) * disp_pmrtHz; force_Npk = disp_mpk * 1./compliance_11Hz.mpN; V_CTRL.Vpk = nActs * sqrt(2) * sqrt(binWidth) * esdGain * excChannel.amplitude.V_DAC.VrtHz; V_BIAS.Vpk = bias.V_ESD; forceCoefficient = abs(force_Npk ./ (2*V_CTRL.Vpk * V_BIAS.Vpk)); where disp_pmrtHz = displacement response of the arm cavity length at 11 [Hz] during each bias voltage setting, in [um] (which I subsequently turned into [pm] for plotting clarity) as measured from the calibrated ALS control signal, H1:ALS-X_REFL_CTRL_OUT_DQ sqrt(2)*sqrt(binWidth) = sqrt(2)*sqrt(0.09375 [Hz]), calculation needed to change from noise units ?_{rms}/rtHz to amplitude units of ?_pk 1e-12 = 1e-12 [m/pm] compliance_11Hz.mpN = 5.3e-06 [m/N], TST to TST, L to L transfer function at 11 [Hz], obtained from the QUAD model. excChannel.amplitude.V_DAC.VrtHz = 12.45 [V/rtHz], measured output request voltage from the DAC (calibrated from counts out of the digital last output to ESD, i.e. H1:SUS-ETMX_L3_MASTER_OUT_LL_DQ using the 180bit DAC gain, 20/2^18 [V/ct]) esdGain = 40 [V/V], gain of the ESD driver sqrt(2)*sqrt(binWidth) = sqrt(2)*sqrt(0.09375 [Hz]), calculation needed to change from noise units ?_{rms}/rtHz to amplitude units of ?_pk nActs = 4, since the calibrated output requested voltage is only from one channel. and I've computed the force coefficient, a, using only the linear term, since I used the amplitude of the linear term alone F = a (V_CTRL sin(wt) - V_BIAS)^2 = a V_CTRL^2 sin^2(wt) - 2 a V_CTRL V_BIAS sin(wt) + a V_BIAS^2 [ sin^2(wt) = 1/2 - 1/2 cos(2wt) + O(4w) ] = a(V_BIAS^2 + 1/2 V_CTRL^2) - 2 a V_CTRL V_BIAS sin(wt) + - 1/2 a V_CTRL^2 cos(2wt) DC Term Linear Term Bi-linear Term =>> Linear Term, a = F (2 V_CTRL V_BIAS) Confusingly, this resulting coefficient, a = 0.5e-10 [N/V^2] is a factor of 8 weaker than expected, which is *different* than the factor of 4 weaker that was needed to explain the linear transfer functions taken last week (see the second attachment of LHO aLOG 11676). The script that processes the measurements is here: /ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGL3/Scripts/analyzeesdcharge_20140515.m
Jeff, This is getting pretty mysterious. Suggest that you try to establish whether you actually can see the capacitance change with relative motion of the test mass with respect to the recoil mass in each ESD quadrant. The bridge circuit would be a good way of doing this. I will be at LHO starting Monday and could help out with the measurement. One explanation is that you do not really have a solid connection to either the bias or the control electrodes - an open connection with either only capacitive coupling or a high resistance. RW
Jeff, 1 piece of good news - this amount of voltage implies that the isolation performance is probably not compromised by the charge. Long ago I did a rough estimate for how much charge could be on the optic in a blob near the tip of the earthquake stop. https://dcc.ligo.org/T080214 Although the field gradient is the important thing, and this is a function of both of the amount of charge and the spatial distribution there-of, one can say the if the blob is about the size of the stop, then the voltage of it is about 140 V before it compromises the isolation performance. -Brian
J. Kissel, B. Shapiro, Brett has caught a mistake in my calculations above, in that I *should not* multiply the control voltage, V_CTRL.Vpk, by a factor of four (i.e. nActs). My thoughts on including the factor of four were that I was measuring the requested DAC output of *one* quadrant, and since there're 4 quadrants, I add the force created by each of them. However, given how the force coefficient was defined for all four quadrants, it's better to think of each of the four quadrants creating a ring of control voltage, all held at the same requested value. It's then the potentional difference between this control ring and the bias ring that generates the forces on the test mass. Hence the caluclation for the force coefficient should be disp_mpk = 1e-12 * sqrt(2) * sqrt(binWidth) * disp_pmrtHz; force_Npk = disp_mpk * 1./compliance_11Hz.mpN; V_CTRL.Vpk = sqrt(2) * sqrt(binWidth) * esdGain * excChannel.amplitude.V_DAC.VrtHz; V_BIAS.Vpk = bias.V_ESD; forceCoefficient = abs(force_Npk ./ (2*V_CTRL.Vpk * V_BIAS.Vpk)); (Thanks for the clarification Brett!) In addition, while debugging, I caught another error: The uncertainty from each data point as calculated above computes the *relative* uncertainty. As displayed on the graph, since the plot shows absolute amplitude of the displacement, the uncertainty should be displayed in absolute units, as in unc = disp_pmrtHz.full .* sqrt( (1 - coh) / 2*nAvgs*coh ) weight = 1 / unc^2; Finally, even more dumb of a mistake, a copy and paste error meant I was using the asd vector as my coherence vector. All of the above errors, bugs, and mistakes have been corrected, and I attach a revised plot. The new force coefficient is roughly 2.2e-10 [N/V^2].
