Sudarshan, Dan
The ISS second loop didnot close using the automated python script. We also tried to close the loop manually with the steps described in alog 14291 but without any success. The loop rails each time we try to close it. From some prelimianry investigation, it looks like there is some problem with the electronics readout of the PDs. Attached is a plot showing all 8 PD signals of the ISS photodiode array.
I have started one of two chillers at each mid station for periodic running and maintenance checks. They will run overnight and then I will switch to the 2nd chiller tomorrow.
Hugo, Arnaud, Dave:
Hugo and Arnaud committed the latest versions of isi/common/models/[isi2stagemaster.mdl/isihammaster.mdl] to the repostitory. I upgraded H1, svn revision numbers given below
file | old | new |
isi2stagemaster | r8417 | r9545 |
isihammaster | r8122 | r9545 |
I tested the new common models by compiling an ISI HAM and ISI BSC model. I then compiled all the models against RCG2.9 with no failures.
The H1.ipc file was regenerated performing two run throughs of "make -i World". The new H1.ipc is the same as the old with the exception of three IPC channnels which have become obsolete:
I have replaced the original IPC file for the remainder of today in case any models need to be restarted.
Output power is ~ 32.7 W (Should be ~ 30 W) Watchdog is active No warning in SYSSTAT other than "VB program online" PMC Locked for 7 days Reflected power is 8.77% of transmitted power (Should be 10% or less) FSS Reference cavity has been locked for 5 hours Trans PD threshold is .4 V (Should be at least .9 V) ISS Diffracted power is ~ 7.4% Last saturation event was 3 days and 18 hours ago
With BrianS's help, opened up two containers, inventoried cables etc and installed the shorting plugs on the GS-13 cables.
So all the BSC ISIs (in Storage) and three HAM ISIs are done. Two HAMs remain.
Saturday, no restart reported
model restarts logged for Sun 11/Jan/2015
2015_01_11 22:26 h1fw1
one unexpected restart.
Just because we are a tad paranoid now, I took V and P TFs of the ETMx main chain and a P TF of the reaction chain around noon today. All is well with this suspension still. BSC9 pressure is ~2x10-6 Torr.
Jim and Dave:
WP5003. We replace three ADC cards in the h1seiex IO Chassis which were PMC cards in PCIe carriers. Three regular PCIe ADC cards were removed from the DTS x1seiham for this swap out. This swap is needed for the RCG2.9 upgrade planned for tomorrow.
There are four ADC cards in this system, all but the first ADC were replaced.
I performed some DAQ trending of raw ADC channels in the swapped cards prior and post swap to verify the data looks continguous.
The procedure was: remove h1seiex from Dolphin network, stop all models, power down h1seiex, powerdown IO Chassis, replace ADC cards, power up IO Chassis, power up h1seiex, let all models autostart.
The NDS server on h1nds0 was shut down for a few minutes to allow building of RCG-2.9 daqd executables for the frame writer, nds, and broadcaster. This should not have affected anyone since the default NDS server is h1nds1.
Tweaked the AOM defracted power down to ~ 7.4% from ~8.5%.
SEI Jim W.: Installing new controllers for HAM3 to address .6 Hz resonance Hugh: Would like to take HEPI down tomorrow to install and calibrate new pressure sensors 3IFO 3 visitors from LLO: Jeremy, Brian and Danny
The cdswiki computer was unresponsive this morning, so I rebooted it. Web based access to MEDM screens and work permits should be restored, among other services.
We have exorSIIIIZED the daemons; this house is CLEAR. This Dr. Arroway! I'm okay to go! I'm okay to go! I'm okay to... go... Where we're going... we don't NEED roads. Attached are H1 SUS ETMX, usual, quick transfer function assessment of the two more sensitive degrees of freedom to rubbing anomalies, with the EX vacuum chamber now down to 6e-6 [Torr]. They, and the references show that the SUS remains free of rubbing. Nice work Betsy / Travis / Brett! But, for the record, we should check all DOFs and all three SUS chains in the chamber tomorrow.
J. Kissel Probably a little bit late given Seb's recent discoveries about the HAM3 ISI's 0.6 [Hz] sharp feature blend filter sensitivity (see LGO aLOG 16001), but just to prove that it's NOT the HSTSs on the table, I've taken damped transfer functions of both MC2 and PR2. Damped is their nominal state, though one could argue that damped + global control is, but the dynamics will only become *more* suppressed and *less* Q-ey with global control -- assuming it was designed with suitable phase and gain margins. We'll presume yes. Note that these are still the old, poorly, individually tuned, damping filters, which is why the two SUS behave differently. We plan on installing the LLO design soon. This again, will only make the SUS *less* Q-ey. .xml files live in: /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC2/SAGM1/Data/2015-01-12_0607_H1SUSMC2_M1_WhiteNoise_*.xml /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/PR2/SAGM1/Data/2015-01-12_0542_H1SUSPR2_M1_WhiteNoise_*.xml
J. Kissel, with A. Staley (remotely again) After fighting the IMC for quite a bit (see LHO aLOG 16005), Hugo wanted to set up a simple MICH lock in order to test his recent updates to the ISI guardians which allow for switching the ISI BS's stage 2 GS13s from high gain to low gain and back, without breaking the MICH lock (See results in LHO aLOG 16007) but I wanted to put down what needed done before I was able to restore a signal at the AS port, as it was left in an unknown & undocumented state from last night's Koji / Dan commissioning activities. - Request ISC_DRMI guardian to DOWN. It had been left in "LOCK_DRMI_1F," which among other things, was dividing the input power by two three-five seconds after the IMC locked. - Request ISC_DOF guardian to DOWN. This restores all globally controlled suspensions that should be to MANAGED (PRM, PR2, BS, SR2, SRM) - Clear all DRMI WFS offsets from the M1 LOCK P and Y banks of the BS HLTS and HSTSs (which had seemingly random in size and DOF, some rather large, steering the beam who knows where) - Change the ramp time to something large (like 10 [sec]) - Remember the gain value - Set gain to zero - Clear filter history - Restore gain - Request MICH_DARM_LOCKED of the LSC_CONFIGS guardian. - Clear history DC centering on for the IFO WFS - Open ASC overview - Open DC CENTERING (smallish, blue button, in the middle of the screen) - Pull up H1:LSC-ASAIR_A_LF_OUT_256_DQ on a live dataviewer trace, and/or pull up AS port camera (sitemape > CDS > Digital Video Cameras > H1 AS Air > ! Video Control > Launch Fixed Video) - Wait patiently while relevant optics swing into position (may take 15-30 seconds) - Once locked (which happens reliably automatically), you need to tune the BS alignment -- but no more than 0.1 to 0.2 [urad] in either pitch or yaw. Look for 01 modes to disappear, and leave the camera dark, and the AS AIR signal (H1:LSC-ASAIR_A_LF_OUT_256_DQ) to be in the -20 to -15 region. I've left the functional parts of the IFO in this state -- with Simple Michelson locked on a dark fringe.
The large offset in the M1 P,Y LOCK is due to the fact that guardian has not been fully updated since the ASC loops have been modified; the guardian doesn't know to clear the history of these filters which now have an integrator. This has been on our to do list.
J. Kissel Something went wrong with the IMC after my safe.snap captures. It seems like the MC WFS DC signal indicates that they've lost there spots, so the ASCIMC servos are continually steering the IMC slowly off resonance. I attach two trends. The first is the past 6 hours, showing that when I started taking down suspensions, the MC WFSs lost there spots. The second is the last hour, showing how the IMC is in a bad locking loop, where it acquires, and then slowly tanks. I've burtrestore'd all relevant epics IOCs to 02:10p PDT (before I came in) I could think of, i.e. burtrestore h1susmc1epics 14:10 burtrestore h1susmc2epics 14:10 burtrestore h1susmc3epics 14:10 burtrestore h1ascepics 14:10 burtrestore h1ascimcepics 14:10 burtrestore h1sushttsepics 14:10 But things are still stuck in the loop. I'm going to continue to try to debug, but I solicit any remote help, if anyone's out there reading... Will post if I find a/the solution.
J. Kissel, w/ a remote A. Staley & E. Hall Though we're still not sure why the spots have gotten mis-centered on the IMC WFSs, I was able to offload the previously functional IMC WFS requested values to the OPTICALIGN alignment offsets in MC1, MC2, and MC3, and this extended the cycle by a few more minutes. This still did not get light on MC WFS B so the ASCIMC loops still eventually pull the alignment of into la-la land. Thinking of what happened this past friday, Alexa suggests just to leave the IMC WFS OFF, and I have done so. The IMC has been locked rock solid since. A couple of debugging notes, but still no answer: Also -- Alexa informed me that the power 50% power drops were because the DRMI guardian had not been set to the DOWN state, so once the IMC locked, it would downgrade the input power from 10 to 5 [W]. This was a red herring, which I thought was more MC WFS mysteriousness that I couldn't figure out at first. Here's a comparison between previously functional alignment offsets (with MC WFSs engaged, so modified from the static H1:IMC-${OPTIC}_${DOF}_OFFSET), and those that have had the previously functional MC WFS DC values (as read by the output M1 LOCK banks) offloaded to them: H1:IMC-${OPTIC}_${DOF}_OFFSET H1:SUS-${OPTIC}_M1_LOCK_${DOF}_OUTMON H1:SUS-${OPTIC}_M1_OPTICALIGN_${DOF}_OFFSET P Y P Y P Y MC1 this morning +0.7 +0.2 -18.1 +16.8 +1024.2 +2100.0 MC1 with WFS offload +0.7 +0.2 +0.7 +0.2 +1006.1 -2083.2 MC2 this morning -21.5 +27.6 +18.4 +1.8 515.8 -556.7 MC2 with WFS offload -21.5 +27.6 -21.5 +27.6 534.2 -554.9 MC3 this morning +5.8 +18.0 -13.0 +1.5 -527.3 -2100.0 MC3 with WFS offload +5.8 +18.0 +5.8 +18.0 -540.3 -2198.5 I did NOT save these new alignment values to the SUS' ALIGNED file, it was just a stab in the dark. I've taken a whole bunch more trends, which indicate that the alignment offsets were identical to before I got started, until I changed them as indicated above. However -- zooming in on when, and in what order I turned ON and OFF the suspensions vs. relocking the IMC, I find there may be a pattern / problem in the order and timing in which I took down the three MCs. After staring at all five plots at once (I know, it's hard to do without a lot of screen real estate, but maybe open in 5 different tabs and flip between), I recall that I turned OFF MC1 first, *without* pausing the IMC guardian or requesting it to be DOWN. Before I moved on to the other suspensions, I restored MC1 to confirm that the IMC came back. As such, in the interim, I think the IMC WFS began to steer the IMC in a bad direction with the misaligned (i.e. OFF) MC1, pulling the integrators to mis-center the spots -- especially on MC WFS B. Further, I didn't have nearly as many IMC screens open as I did after trying to diagnose the problem, so I didn't see that the MC WFS were being pulled off course. I thought, that if this is true, it should be as simple as restoring the Sunday "morning" offsets, and clearing the history on the MC WFS integrators, and the spots should immediately become re-well centered on the MC WFS again. BUT, that didn't work either. With the "morning" offsets on the MCs, and history cleared on the ASC IMC loops, the spots reappear on the MEDM cross-hairs, but just barely. Which means the TRANS power began to tank again with full gain MC WFS on. Now I think, we should use the picomotors to recenter the MC WFS. But, I have zero experience with the pico-motor game, and I have learned to harbor great fear of them, especially after Suresh informs me that they're somehow wired such that a YAW request moves the beam in PITCH. Do we not have a DC centering servo on the MC WFS? For now, I leave the MC WFS OFF (via the gain slider in the bottom left corner), and I've cleared the history of the ASC IMC DOF loop filter banks.
J. Kissel, with Suresh (remotely) this time For starters -- thanks for all your help remote commissioners! So I think Suresh might have nailed down the problem: - Last week, Thursday afternoon, Jan 8th, Suresh commissions the MC WFS DC Centering servos, a.k.a. DOF 4 servos, a.k.a. the MC1 / MC3 differential pitch and common yaw servos (see LHO aLOG 15865). First with a gain of -0.1, and slightly later with a gain of -1.0. In doing so, he changes the alignment sliders of the IMC suspensions, but does not *save* the new alignments to the aligned.snap text files. - The next morning, when the IMC guardian was changed from LOCKED to DOWN, the old, now invalid, alignment values returned (see LHO aLOG 15968). When the new centering servos started pulling the spots off the MC WFS from the bad alignment, the DOF 4 gains were changed to zero. "Turn it off! Turn it off!" - When the alignment values were restored, the DOF4 centering loops were left off, because they didn't instantly work as designed, and there were bigger fish to fry. We now know it's because the MC WFS are so miscentered that even with a good IMC alignment for starters, then pull things off the quadrants. - These new alignment values remain stored, and have new even been captured in a safe.snap. - That the IMC can lock by itself, and stay locked robustly, without guardian or WFS implies that this indeed a good alignment on the rest of the REFL path (i.e. on the trigger PD and the IMC REFL Length diode). - So, we should re-center the spots on MC WFS using the pico-motors by hand, as I mention above in a previous comment. I didn't yet enact on the solution, because I wanted to do a few other things tonight (and because of my previously mentioned superstitious fear of picomotors), but I write it down for those to attack whenever they can.
Evan and I recentered the IMC wfs, they are working now.
In order to calculate the sensor correction gain, I put HAM2 in this configuration for the night:
All DOFs:
- 750mHz blend
- Sensor correction OFF
I'll put it back to its nominal configuration Monday morning
The matching gains for X, Y and Z are
0.9750
0.8431
0.8167
Following the work done last week (see this alog), I've done some more tests today.
First the unsuccessful results:
- I've switched from a lvl3 conroller configuration to a lvl2 with no success. The issue doesn't come from the control loops.
- I turned OFF all the damping loops of suspensions (MC2 and PR2). This didn't affect at all the amplitude of the peaks: the suspensions behavior seems unrelated with our issue.
Now the good part:
The problem seems directly correlated with the blends, and especially with the blends on RX and RY. I'm making this affirmation because of 2 things:
1. The peak disappears when I switch the blends on RX and RY. It stays up otherwise. In the plot atrached, I've switched from a 01_28 blend (solid lines) to a 100mHz blend (dash lines) on RX & RY and the peaks are totally gone. This is true if I switch tfrom 01_28 to another set of blends as well, I took 100mHz as an example.
2. I've installed a set of blend filters on RX and RY called 01_28t. They are very similar to the 01_28 blends, I just moved poles and zeros around to see if it makes any difference. It does: the frequency of the peaks shifted from 0.617Hz to 0.664Hz (see plots attached).
Conclusion: We are using this 01_28 set of blends on all the units without any problem: only HAM3 seems to present this bizarre behavior... But even if I still don't understand what's happening, we have a solution. The 100mHz blends are not ideal for these DOFs, and I'll design a better set of blends ASAP.
Conclusion (bis): Current configuration on HAM3
X,Y,Z,RZ: 01_28 blend
RX,RY: 100mHz blend
Sensor correction ON on X,Y,Z.
Very nice, can you take some closed loop TFs so we can understand what is going on?