Just because we are a tad paranoid now, I took V and P TFs of the ETMx main chain and a P TF of the reaction chain around noon today. All is well with this suspension still. BSC9 pressure is ~2x10-6 Torr.
Jim and Dave:
WP5003. We replace three ADC cards in the h1seiex IO Chassis which were PMC cards in PCIe carriers. Three regular PCIe ADC cards were removed from the DTS x1seiham for this swap out. This swap is needed for the RCG2.9 upgrade planned for tomorrow.
There are four ADC cards in this system, all but the first ADC were replaced.
I performed some DAQ trending of raw ADC channels in the swapped cards prior and post swap to verify the data looks continguous.
The procedure was: remove h1seiex from Dolphin network, stop all models, power down h1seiex, powerdown IO Chassis, replace ADC cards, power up IO Chassis, power up h1seiex, let all models autostart.
The NDS server on h1nds0 was shut down for a few minutes to allow building of RCG-2.9 daqd executables for the frame writer, nds, and broadcaster. This should not have affected anyone since the default NDS server is h1nds1.
Tweaked the AOM defracted power down to ~ 7.4% from ~8.5%.
SEI Jim W.: Installing new controllers for HAM3 to address .6 Hz resonance Hugh: Would like to take HEPI down tomorrow to install and calibrate new pressure sensors 3IFO 3 visitors from LLO: Jeremy, Brian and Danny
The cdswiki computer was unresponsive this morning, so I rebooted it. Web based access to MEDM screens and work permits should be restored, among other services.
We have exorSIIIIZED the daemons; this house is CLEAR. This Dr. Arroway! I'm okay to go! I'm okay to go! I'm okay to... go... Where we're going... we don't NEED roads. Attached are H1 SUS ETMX, usual, quick transfer function assessment of the two more sensitive degrees of freedom to rubbing anomalies, with the EX vacuum chamber now down to 6e-6 [Torr]. They, and the references show that the SUS remains free of rubbing. Nice work Betsy / Travis / Brett! But, for the record, we should check all DOFs and all three SUS chains in the chamber tomorrow.
J. Kissel Probably a little bit late given Seb's recent discoveries about the HAM3 ISI's 0.6 [Hz] sharp feature blend filter sensitivity (see LGO aLOG 16001), but just to prove that it's NOT the HSTSs on the table, I've taken damped transfer functions of both MC2 and PR2. Damped is their nominal state, though one could argue that damped + global control is, but the dynamics will only become *more* suppressed and *less* Q-ey with global control -- assuming it was designed with suitable phase and gain margins. We'll presume yes. Note that these are still the old, poorly, individually tuned, damping filters, which is why the two SUS behave differently. We plan on installing the LLO design soon. This again, will only make the SUS *less* Q-ey. .xml files live in: /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC2/SAGM1/Data/2015-01-12_0607_H1SUSMC2_M1_WhiteNoise_*.xml /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/PR2/SAGM1/Data/2015-01-12_0542_H1SUSPR2_M1_WhiteNoise_*.xml
J. Kissel, with A. Staley (remotely again) After fighting the IMC for quite a bit (see LHO aLOG 16005), Hugo wanted to set up a simple MICH lock in order to test his recent updates to the ISI guardians which allow for switching the ISI BS's stage 2 GS13s from high gain to low gain and back, without breaking the MICH lock (See results in LHO aLOG 16007) but I wanted to put down what needed done before I was able to restore a signal at the AS port, as it was left in an unknown & undocumented state from last night's Koji / Dan commissioning activities. - Request ISC_DRMI guardian to DOWN. It had been left in "LOCK_DRMI_1F," which among other things, was dividing the input power by two three-five seconds after the IMC locked. - Request ISC_DOF guardian to DOWN. This restores all globally controlled suspensions that should be to MANAGED (PRM, PR2, BS, SR2, SRM) - Clear all DRMI WFS offsets from the M1 LOCK P and Y banks of the BS HLTS and HSTSs (which had seemingly random in size and DOF, some rather large, steering the beam who knows where) - Change the ramp time to something large (like 10 [sec]) - Remember the gain value - Set gain to zero - Clear filter history - Restore gain - Request MICH_DARM_LOCKED of the LSC_CONFIGS guardian. - Clear history DC centering on for the IFO WFS - Open ASC overview - Open DC CENTERING (smallish, blue button, in the middle of the screen) - Pull up H1:LSC-ASAIR_A_LF_OUT_256_DQ on a live dataviewer trace, and/or pull up AS port camera (sitemape > CDS > Digital Video Cameras > H1 AS Air > ! Video Control > Launch Fixed Video) - Wait patiently while relevant optics swing into position (may take 15-30 seconds) - Once locked (which happens reliably automatically), you need to tune the BS alignment -- but no more than 0.1 to 0.2 [urad] in either pitch or yaw. Look for 01 modes to disappear, and leave the camera dark, and the AS AIR signal (H1:LSC-ASAIR_A_LF_OUT_256_DQ) to be in the -20 to -15 region. I've left the functional parts of the IFO in this state -- with Simple Michelson locked on a dark fringe.
The large offset in the M1 P,Y LOCK is due to the fact that guardian has not been fully updated since the ASC loops have been modified; the guardian doesn't know to clear the history of these filters which now have an integrator. This has been on our to do list.
The Sensor Correction screens of the HAM-ISIs and HEPIs were updated uppon Jeff's request.The BSC-ISIs were taken care of earlier this week.
All the SEI platforms on site have their Sensor Correction screens updated now.
Screenshots of the new screens are attached. Links to the old screens are still available on the bottom right of the new ones.
Work was performed under WP #4998 which will remain open for a couple days to allow last minute requests to be put-in. The new screens will then be committed under the seismic SVN
(Entry by Kyle)
Shut down X-end QDP80 (fixed pinout problem with Y-end and X-end scroll pump relay box connectors which had prevented me from using these)
1320 -1345 hrs. local -> In and out of Y-end LVEA
J. Kissel Something went wrong with the IMC after my safe.snap captures. It seems like the MC WFS DC signal indicates that they've lost there spots, so the ASCIMC servos are continually steering the IMC slowly off resonance. I attach two trends. The first is the past 6 hours, showing that when I started taking down suspensions, the MC WFSs lost there spots. The second is the last hour, showing how the IMC is in a bad locking loop, where it acquires, and then slowly tanks. I've burtrestore'd all relevant epics IOCs to 02:10p PDT (before I came in) I could think of, i.e. burtrestore h1susmc1epics 14:10 burtrestore h1susmc2epics 14:10 burtrestore h1susmc3epics 14:10 burtrestore h1ascepics 14:10 burtrestore h1ascimcepics 14:10 burtrestore h1sushttsepics 14:10 But things are still stuck in the loop. I'm going to continue to try to debug, but I solicit any remote help, if anyone's out there reading... Will post if I find a/the solution.
J. Kissel, w/ a remote A. Staley & E. Hall
Though we're still not sure why the spots have gotten mis-centered on the IMC WFSs, I was able to offload the previously functional IMC WFS requested values to the OPTICALIGN alignment offsets in MC1, MC2, and MC3, and this extended the cycle by a few more minutes. This still did not get light on MC WFS B so the ASCIMC loops still eventually pull the alignment of into la-la land. Thinking of what happened this past friday, Alexa suggests just to leave the IMC WFS OFF, and I have done so. The IMC has been locked rock solid since.
A couple of debugging notes, but still no answer:
Also -- Alexa informed me that the power 50% power drops were because the DRMI guardian had not been set to the DOWN state, so once the IMC locked, it would downgrade the input power from 10 to 5 [W]. This was a red herring, which I thought was more MC WFS mysteriousness that I couldn't figure out at first.
Here's a comparison between previously functional alignment offsets (with MC WFSs engaged, so modified from the static H1:IMC-${OPTIC}_${DOF}_OFFSET), and those that have had the previously functional MC WFS DC values (as read by the output M1 LOCK banks) offloaded to them:
H1:IMC-${OPTIC}_${DOF}_OFFSET H1:SUS-${OPTIC}_M1_LOCK_${DOF}_OUTMON H1:SUS-${OPTIC}_M1_OPTICALIGN_${DOF}_OFFSET
P Y P Y P Y
MC1 this morning +0.7 +0.2 -18.1 +16.8 +1024.2 +2100.0
MC1 with WFS offload +0.7 +0.2 +0.7 +0.2 +1006.1 -2083.2
MC2 this morning -21.5 +27.6 +18.4 +1.8 515.8 -556.7
MC2 with WFS offload -21.5 +27.6 -21.5 +27.6 534.2 -554.9
MC3 this morning +5.8 +18.0 -13.0 +1.5 -527.3 -2100.0
MC3 with WFS offload +5.8 +18.0 +5.8 +18.0 -540.3 -2198.5
I did NOT save these new alignment values to the SUS' ALIGNED file, it was just a stab in the dark.
I've taken a whole bunch more trends, which indicate that the alignment offsets were identical to before I got started, until I changed them as indicated above. However -- zooming in on when, and in what order I turned ON and OFF the suspensions vs. relocking the IMC, I find there may be a pattern / problem in the order and timing in which I took down the three MCs. After staring at all five plots at once (I know, it's hard to do without a lot of screen real estate, but maybe open in 5 different tabs and flip between), I recall that I turned OFF MC1 first, *without* pausing the IMC guardian or requesting it to be DOWN. Before I moved on to the other suspensions, I restored MC1 to confirm that the IMC came back. As such, in the interim, I think the IMC WFS began to steer the IMC in a bad direction with the misaligned (i.e. OFF) MC1, pulling the integrators to mis-center the spots -- especially on MC WFS B. Further, I didn't have nearly as many IMC screens open as I did after trying to diagnose the problem, so I didn't see that the MC WFS were being pulled off course.
I thought, that if this is true, it should be as simple as restoring the Sunday "morning" offsets, and clearing the history on the MC WFS integrators, and the spots should immediately become re-well centered on the MC WFS again. BUT, that didn't work either. With the "morning" offsets on the MCs, and history cleared on the ASC IMC loops, the spots reappear on the MEDM cross-hairs, but just barely. Which means the TRANS power began to tank again with full gain MC WFS on.
Now I think, we should use the picomotors to recenter the MC WFS. But, I have zero experience with the pico-motor game, and I have learned to harbor great fear of them, especially after Suresh informs me that they're somehow wired such that a YAW request moves the beam in PITCH. Do we not have a DC centering servo on the MC WFS?
For now, I leave the MC WFS OFF (via the gain slider in the bottom left corner), and I've cleared the history of the ASC IMC DOF loop filter banks.
J. Kissel, with Suresh (remotely) this time For starters -- thanks for all your help remote commissioners! So I think Suresh might have nailed down the problem: - Last week, Thursday afternoon, Jan 8th, Suresh commissions the MC WFS DC Centering servos, a.k.a. DOF 4 servos, a.k.a. the MC1 / MC3 differential pitch and common yaw servos (see LHO aLOG 15865). First with a gain of -0.1, and slightly later with a gain of -1.0. In doing so, he changes the alignment sliders of the IMC suspensions, but does not *save* the new alignments to the aligned.snap text files. - The next morning, when the IMC guardian was changed from LOCKED to DOWN, the old, now invalid, alignment values returned (see LHO aLOG 15968). When the new centering servos started pulling the spots off the MC WFS from the bad alignment, the DOF 4 gains were changed to zero. "Turn it off! Turn it off!" - When the alignment values were restored, the DOF4 centering loops were left off, because they didn't instantly work as designed, and there were bigger fish to fry. We now know it's because the MC WFS are so miscentered that even with a good IMC alignment for starters, then pull things off the quadrants. - These new alignment values remain stored, and have new even been captured in a safe.snap. - That the IMC can lock by itself, and stay locked robustly, without guardian or WFS implies that this indeed a good alignment on the rest of the REFL path (i.e. on the trigger PD and the IMC REFL Length diode). - So, we should re-center the spots on MC WFS using the pico-motors by hand, as I mention above in a previous comment. I didn't yet enact on the solution, because I wanted to do a few other things tonight (and because of my previously mentioned superstitious fear of picomotors), but I write it down for those to attack whenever they can.
Evan and I recentered the IMC wfs, they are working now.
I found the 9 sec timing error in the test mass charge measurement mentioned in LHO log 15983. See UR_timing_err.pdf. The problem was in the matlab analysis script, not in the python measuring script. The measurement script saved a seperate file for the excitations for each quadrant and bias level. See for example, UR_Exc_raw_data.pdf. The excitations in these files are preceded by 9 or 10 seconds and trailed by 3 or 4 seconds where the excitation is nominally zero. The analysis script then finds the exact point where the excitation starts by searching for the first point where the excitation data is non-zero. The problem is that the very beginning of the excitation is not always exactly zero because of some tiny residual from a previous excitation. The analysis code was aware of this problem, and therefore cut out the first second. However, 1 second was not always quite enough. Sometimes a non-zero data point or two would survive the 1 sec cut. So I extended this cut to 2 seconds. So far this removes the problem from all the analysis. See UR_timing_err_fixed.pdf.
The new trend results are in ETMY_Charge_Trend_8Jan2015.pdf. The timing error fix inf;luenced the pitch results by +- 5 volts and the yaw results by +- 15 V.
All charge scripts are on the SVN in
.../SusSVN/sus/trunk/QUAD/Common/Scripts/
The charge measuring script is
ESD_UL_LL_UR_LR_charge_07-H1.py
This scripted can be run on repeat many times using the script
ChargeTest_H1.py
It is set to repeat the measurement 10 times, waiting 5 minutes between runs (currently each run is about 10 minutes long).
The Matlab analysis script that processes the results is
ESD_UL_LL_UR_LR_analysis07_H1.m
There is a script for plotting trends over multiple measurements, but it needs some work to generalize it for different suspensions and different data sets.
Long_Trend_H1_ETMY.m
In log 15983 the date stamp of the trend figure should read Jan 8, not Jan 9.
In order to calculate the sensor correction gain, I put HAM2 in this configuration for the night:
All DOFs:
- 750mHz blend
- Sensor correction OFF
I'll put it back to its nominal configuration Monday morning
The matching gains for X, Y and Z are
0.9750
0.8431
0.8167
Following the work done last week (see this alog), I've done some more tests today.
First the unsuccessful results:
- I've switched from a lvl3 conroller configuration to a lvl2 with no success. The issue doesn't come from the control loops.
- I turned OFF all the damping loops of suspensions (MC2 and PR2). This didn't affect at all the amplitude of the peaks: the suspensions behavior seems unrelated with our issue.
Now the good part:
The problem seems directly correlated with the blends, and especially with the blends on RX and RY. I'm making this affirmation because of 2 things:
1. The peak disappears when I switch the blends on RX and RY. It stays up otherwise. In the plot atrached, I've switched from a 01_28 blend (solid lines) to a 100mHz blend (dash lines) on RX & RY and the peaks are totally gone. This is true if I switch tfrom 01_28 to another set of blends as well, I took 100mHz as an example.
2. I've installed a set of blend filters on RX and RY called 01_28t. They are very similar to the 01_28 blends, I just moved poles and zeros around to see if it makes any difference. It does: the frequency of the peaks shifted from 0.617Hz to 0.664Hz (see plots attached).
Conclusion: We are using this 01_28 set of blends on all the units without any problem: only HAM3 seems to present this bizarre behavior... But even if I still don't understand what's happening, we have a solution. The 100mHz blends are not ideal for these DOFs, and I'll design a better set of blends ASAP.
Conclusion (bis): Current configuration on HAM3
X,Y,Z,RZ: 01_28 blend
RX,RY: 100mHz blend
Sensor correction ON on X,Y,Z.
Very nice, can you take some closed loop TFs so we can understand what is going on?
J. Kissel, K. Venkateswara, K. Izumi, J. Warner While Kiwamu was trying to lock the Michelson, several things went wrong, but it uncovered a flaw in the new gain switching of the GS13s that has been implemented in the Guardian (see LHO aLOG 15537. Here's what happened. (1) Kiwamu incorrectly brought up the BSC ISI in "fully isolated," which turns on stage 2 isolation loops, and switches the GS13s to high-gain mode. (2) As expected, while trying to acquire MICH lock, impulses sent to the SUS BS kick the cage as well, which is attached to the BS ISI ST2, and trips the ST2 on the GS13s watchdog. (3) We then reset the watchdog, and switched to "isolated damped," and this triggered the new guardian feature to *start* switching the GS13s back to low gain, but with MICH still trying to lock, impulses would still trip the watchdog before guardian had the chance to switch *all* of the GS13s gains. (4) This, trigger-happy watchdog resetting, and guardian half transitioning, caused a nasty loop of guardian sloshing the GS13 gains back and forth between high and low, which, with MICH impulses, continued to trip the watchdog. I attach a plot of one of the WD trip, which clearly shows that the V3 GS13 had failed to have its gain switched to low. We should (a) look the new code to make sure there isn't a bad loophole regarding the GS13 gain switching when transitioning from "fully isolated" to "isolated damped" (b) look for ways to increase the switching speed, or add a pause / check that the switch has occured on all GS13s before proceeding with the transition (c) remember that these are physical, several thousand pound systems -- if you have to reset watchdogs repeatedly something is wrong and you don't know why, don't just blindly continue to mash the reset button, figure out what's wrong, or do what Kiwamu did and ask an expert! #justwait
The GAIN and DWT filters' switching mode is set to zero-crossing, with a time-out of 2s (see attachement). Even though Guardian engages the filters properly, they don’t actually switch until a certain time, causing MICH to start acquiring lock before the ISI is ready for it.
This could be solved by selecting the immediately switch mode for the GS13 GAIN and DWT filters. But, after discussing it with Jeff yesterday it turned out that he recalled switching gains with the ISO on, which would be way less stable without zero-crossing.
I modified the SEI guardian to add a 3s wait at the end of the gain switch sequence to give the filters the time they need to switch with the current zero-crossing configuration, before allowing MICH to start acquiring lock.
This fix should be tried next time we start acquiring lock on MICH.
Jeff, Hugo,
The SEI guardian patch I made was tested today. Jeff locked MICH while SEI_BS was in the ISOLATED_DAMPED state (GS13 in low-gain). Once MICH was locked, we switched SEI_BS to FULLY_ISOLATED (GS13s in High Gian, and ST2 Iolation loops turned on). The ISI did not trip, and MICH remained locked.
In order to make sure that this was ra reliable fix, we went ahead and switched the state of the SEI_BS node back and forth between ISOLATED_DAMPED and FULLY_ISOLATED a couple times. Once again, the ISI did not trip, and MICH remained locked.
Time series of the state of the SEI_BS Guardian node, versus the MICH error signal, are attached
The updated code was commited under the SVN:
/opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/util.py -r9543
Note: Jamie gave me good feedback on how to improve this new code. The goal here was to make sure it works. I will optimize it once I am back at Stanford.
