patrick.thomas@LIGO.ORG - posted 08:47, Tuesday 09 June 2015 (19010)
restarted h1ecatx1
Biweekly crash recovery. Burtrestored to 06/08/2015 00:10.
Biweekly crash recovery. Burtrestored to 06/08/2015 00:10.
Out of Science mode.
I flipped the intent bit because with Tuesday maintenance comes lots of noise.
Peter Shawhan, Andy Lundgren, Nutsinee Kijbunchoo
We did a first -- and successful! -- test of the "detchar" or "safety" hardware injections shortly after 6:00 PDT this morning, at the time recommended by Jeff (work permit 5262).
The detchar injections are a sequence of loud sine-gaussians at a range of frequencies, primarily intended to check for couplings from the GW strain channel to auxiliary channels. (See https://dcc.ligo.org/LIGO-G1500713 .) For now, at least, we are using a set of 14 frequencies logarithmically spaced from 30 Hz to 2000 Hz, each injected at 3 different amplitudes to try to get target SNR values, spaced 5 seconds apart. Here is the full list with times relative to the start time of the injection:
Matlab> GenerateSGSequence('H1','H1_ASD_at_1117710916.txt');
__time__ __freq__ __SNR__ __AMP__
0.50 30.0 25.0 2.22e-20
5.50 41.4 25.0 7.54e-21
10.50 57.2 25.0 2.86e-21
15.50 79.1 25.0 1.72e-21
20.50 109.2 25.0 1.52e-21
25.50 150.9 25.0 1.7e-21
30.50 208.4 25.0 1.97e-21
35.50 287.9 25.0 2.92e-21
40.50 397.7 25.0 3.73e-21
45.50 549.3 25.0 5.42e-21
50.50 758.8 25.0 6.95e-21
55.50 1048.2 25.0 1.1e-20
60.50 1447.9 25.0 1.71e-20
65.50 2000.0 25.0 2.68e-20
70.50 30.0 50.0 4.44e-20
75.50 41.4 50.0 1.51e-20
80.50 57.2 50.0 5.71e-21
85.50 79.1 50.0 3.44e-21
90.50 109.2 50.0 3.03e-21
95.50 150.9 50.0 3.41e-21
100.50 208.4 50.0 3.94e-21
105.50 287.9 50.0 5.85e-21
110.50 397.7 50.0 7.47e-21
115.50 549.3 50.0 1.08e-20
120.50 758.8 50.0 1.39e-20
125.50 1048.2 50.0 2.2e-20
130.50 1447.9 50.0 3.41e-20
135.50 2000.0 30.0 3.22e-20
140.50 30.0 100.0 8.88e-20
145.50 41.4 100.0 3.02e-20
150.50 57.2 100.0 1.14e-20
155.50 79.1 100.0 6.87e-21
160.50 109.2 100.0 6.06e-21
165.50 150.9 100.0 6.81e-21
170.50 208.4 100.0 7.87e-21
175.50 287.9 100.0 1.17e-20
180.50 397.7 100.0 1.49e-20
185.50 549.3 100.0 2.17e-20
190.50 758.8 100.0 2.78e-20
195.50 1048.2 100.0 4.41e-20
200.50 1447.9 75.0 5.12e-20
205.50 2000.0 30.0 3.22e-20
The file, on h1hwinj, is /data/scirun/O1/HardwareInjection/Details/config/Burst/Waveform/detchar_1117890580_3_H1.txt .
With H1 running in good low-noise mode, Nutsinee switched the intent bit to 'commissioning' and we first injected the sequence starting at 1117890580 with an overall scale factor of 0.25 -- so the target SNRs/amplitudes are a factor of 4 smaller than in the table above. Nutsinee didn't see anything obvious appearing in the live spectrum initially, but Andy looked at Omicron output afterward and say that it had picked up at least some of the louder signals.
We then injected the sequence again starting at 1117891250, this time with an overall scale factor of 1.0 . Nutsinee saw the signals clearly peak up in the live spectrogram, and Andy's quick check with Omicron showed many signals found with large SNR. The interferometer appeared to handle the injections fine, staying in lock. Afterward (and also in between the two injections), Nutsinee set the intent bit back to 'science'.
Note: In the future, we expect detchar safety injections such as these to be marked with the DetChar bit in the CAL-INJ_ODC bitmask, but for the test today we treated it as a burst injection -- it will be marked in ODC (and GDS-CALIB_STATE) as a burst injection, and should produce ODC-INJECTION_BURST segments in the DQ segment database.
I've done a few checks of the injections. The first attachment is the spectrum before the first injections were started compared to the spectrum just after the last one finished. The spectrum is the same before as after, so I don't think anything got rung up. Maybe we can check the violin modes more carefully. There was a fairly big glitch two minutes later (Omega scan) but I don't think it was related. The other four attachments are the injections of the first round of the injection set, done at normal gain. These are meant to have SNR of about 25, but that varies with the spectrum. Most look fine. However, the injections at 1 kHz and above are not correct. They look to be anti-aliased down, or maybe there's a saturation or something wrong with the actuation. We'll check our code to see if there's something wrong with the file we generated.
When Andy presented this on the ER7 call and talked about the higher-frequency injections not appearing correctly, Jeff asked if we were hitting the software limit at +-200 counts. That limit had been set based on some CBC injection studies; we don't really know the level at which unavoidable saturation in the software or hardware chain would set in. Duncan M quickly confirmed that the H1:CAL-INJ_HARDWARE_OUT_DQ channel was hitting +-200 for some of the injections. I took a look too and estimated what SNR value hits that software limit:At 549 Hz, SNR=80 hits 200 counts 759 Hz SNR=33 1048 Hz SNR=10.4 1448 Hz SNR<6.25 (saturated even for the weakest one we injected) 2000 Hz SNR<6.25 (saturated even for the weakest one we injected)So this tells us how much that (currently rather arbitrary) software limit would need to be relaxed to put in larger-SNR injections at those higher frequencies, if it's important to do so.
Jim told me that Guardian kept losing lock at BOUNCE_VIOLIN_MODE_DAMPING. After the lock loss I let Guardian took care of the locking up until this state and it lost lock as expected. I've just finish the initial alignment and will be doing the Bounce violin mode damping by hand. LOWNOISE_ESD_ETMY also known to have problems today so I will be doing the same thing when I get there. I have never run the code by hand so success not guaranteed. I'll do the best I can to get the ifo up for the 6AM hardware injection. Detcharians please stay tuned.
04:30 Locking again at LSC_FF. Guardian worked fine when requested LOWNOISE_ESD_ETMY. Intent bit switched to undisturbed after the range is stable.
Mostly quiet shift
16:00 Took over initial alignement from Cheryl, trying to lock
18:30 Finally locked
19:00 Robert to EY to bang on beam tube, lots of glitching -> particulate?
20:00 Evan breaks lock with a small change to Guardian
22:00 Lock reacquired, LLO is up too.
23:26 HFD is on site, out the gate at 23:55
I went out to watch the cleaning earlier in the day and returned after work had finished to reproduce some of the cleaning activities. I was on the phone with the operator who monitored DARM for glitches. I found that tapping the beam tube with metal like the water/vacuum nozzles produced large glitches, but brushing with the brushes did not. I found that the softer the instrument, the harder it was to make glitches. I was never able to make glitches with my fist, but there was nearly a one-to-one coincidence with metal taps. All glitches, according to Jim, the operator, were broad band and a couple of orders of magnitude above background. I had to wait quite a time for the spectrum to settle down before tapping again. The glitches were like delta functions, not like scattering shelves. There did not seem to be a difference between locations at a baffle and half way between baffles.
I suggested to Bubba and John that we might make fewer glitches if there was a polymer guard on the nozzles.
I guess that the important quantities for freeing metal oxide particles are either acceleration or change in curvature of the beam tube. The difference between soft and hard "hammers" is consistent with both of these hypotheses. I think that it is important to estimate the inter-site coincidence rate and propose that I mount an accelerometer and a shaker on the tube to study glitching as a function of frequency and amplitude. I suspect that there is a soft threshold in curvature change or acceleration, and that this will be fairly constant with time since the oxide layers should no longer be growing.
HFD reports a power outage (? I could barely hear him... gate phone,ftl) and they need to check some boxes. He's currently driving down the X-arm very slowly, so I've flipped the intent bit. Will revert when I see he's off site.
Elvis has left the building. HFD off site at 23:55.
The ISC_LOCK guardian now has a state HIGH_RANGE_ESD_ETMX, which is intended to be used to transition control of DARM from ETMY back to ETMX.
However, in the interest of maximizing the amount of coincident locking time with L1, this state has not yet been tested.
Pankow, Biwer, Thrane We restarted burst injections as LHO starting at GPS= 1117850177. We scheduled one ad hoc waveform to make sure tinj could find the burst waveforms after we switched waveform naming convention so that they end in .txt rather than .dat. This injection was successful: GPS=1117850363. Chris Biwer will be posting a similar entry on the LLO alog. The new burst injections are not all identical sine Gaussians. Each one is different. The upcoming schedule is: 1117854141 2 1.000000 hwinj_1117854141_3_ 1117856256 2 1.000000 hwinj_1117856256_3_ 1117869607 2 1.000000 hwinj_1117869607_3_ 1117878979 2 1.000000 hwinj_1117878979_3_ 1117906195 2 1.000000 hwinj_1117906195_3_ 1117912758 2 1.000000 hwinj_1117912758_3_ 1117929505 2 1.000000 hwinj_1117929505_3_ 1117949327 2 1.000000 hwinj_1117949327_3_ 1117959237 2 1.000000 hwinj_1117959237_3_ 1117966570 2 1.000000 hwinj_1117966570_3_ 1117973881 2 1.000000 hwinj_1117973881_3_
J. Kissel Using the DARM OLGTF model that we've updated the CAL-CS and GDS low-latency calibrations, I've generated an inverse actuation function for the hardware injections. This is an update to the filter designed for the LHO mini-run (see LHO aLOG 18115). The updated filter has been loaded into FM2 of the H1:CAL-INJ_HARDWARE filter bank, as well as the H1:CAL-INJ_BLIND filter bank, BUT NOT YET TURNED ON. It will be turned on tomorrow during maintenance. Recall that this filter should be assigned the same uncertainty as is true for the DARM Open Loop Gain transfer function (see LHO aLOG 18769) -- 50% and 20 [deg], mostly because of the frequency dependent discrepancy between the model vs measured DARM open loop gain transfer function. Indeed we are now quite certain that the model vs. measurement discrepancies / features seen at ~500, ~1000, ~1500 [Hz] are inaccurately modeled violin modes of the ETMY QUAD. Further, we're confident that we can *remove* these features from future actuation functions by retuning the ETMY UIM / PUM / TST control authority, which we will do immediately following ER7. Er, well, immediately following the vent that immediately follows ER7. So we expect future DARM OLGTFs and therefore inverse actuation function uncertainty to be much less. Details ----------- I've used the following DARM OLGTF model, /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Scripts/DARMOLGTFs/H1DARMOLGTFmodel_ER7.m with the following parameter set /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Scripts/DARMOLGTFs/H1DARMparams_1116990382.m to produce a new Inverse Actuation Function for the HARDWARE Injection path, ultimately created by and plotted with /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Scripts/DARMOLGTFs/CompareDARMOLGTFs.m The attached plots show bode plots of the design. As expected, because of the update to (a) the overall reduction of actuation scale factor (in [m/ct]), and (b) the frequency dependence of the actuation function to account for the new PUM / TST cross of ETMY, which is regrettably very feature full, the filter is significantly more complicated. The poles and zeros are toyModel.poles_Hz = [ pair(300,89.9) pair(300,89.9) pair(482,88.5) pair(1040,89.5) pair(3.2e3,0) pair(4e3,51) pair(6.5e3,60)]; toyModel.zeros_Hz = [0.01 0.01 pair(290,89) pair(310,88.9) pair(509,89.5) pair(1030,89.8) pair(3.2e3,70) 6.8e3]; The 300 [Hz] feature are the notches for the beam splitter violin mode (so we don't excite them via DARM), the ~500 and ~1000 [Hz] features are for the actual ETMY violin modes (which now leak through the L3 to L3 transfer function because of the inadequate roll off of the L2 to L3 transfer function), and the features at 2500, 3000, 3500, and 4000 [Hz] (which I did NOT both to include in the inverse filter) are to notch out further harmonics of the ETMY QUAD's violin modes. Finally, the > 4000 [Hz] poles are to roll-off the filter before the Nyquist frequency. The foton design string is zpk([0.01;0.01;5.0612+i*289.956;5.0612-i*289.956;5.95121+i*309.943;5.95121-i*309.943; 4.44181+i*508.981;4.44181-i*508.981;3.59537+i*1029.99;3.59537-i*1029.99;1094.46+i*3007.02; 1094.46-i*3007.02;6800],[0.523599+i*300;0.523599-i*300;0.523599+i*300;0.523599-i*300; 12.6173+i*481.835;12.6173-i*481.835;9.0756+i*1039.96;9.0756-i*1039.96;2517.28+i*3108.58; 2517.28-i*3108.58;3250+i*5629.17;3250-i*5629.17;3200;3200],1,"n")gain(1.27879e+09) where I've forced foton to use a gain of 1.14e17 [ct/m] at 100 [Hz], as shown in the plots, to avoid all the confusion about pole/zero normalization.
- locked when I arrived, ETMX ESD drive still "not quite off", counts in the hundreds
- went out of Intent and turned ETMX ESD drive really mostly off, counts less that 10, and range went up
- lock loss, relocked, and IFO very glitchy
- about 11AM local all glitches stop, I called Bubba, and started and investigation, around 1:30 local glitches from cleaning crew were confirmed, and Robert took over the investigation. Cleaning crew was at
- 17:39UTC, Ellie asked to have the ETMX lower RH turned down from 0.5 to 0.4, and it remains at 0.4
- ETMX roll mode damping gain turned up to 200, from 60, and turned on the -30dg filter, which seemed to work better today
- Evan changes to turn ETMX ESD drive really off automatically
- MEDM chages to ETMX ESD drive screen to have 3 colors for the ETMX ESD drive's 3 states
To Do:
- in the process of an initial alignment
This morning there was some confusion over whether or not the EX ESD was completely off. Since the latest technique for getting some more mpc's is to turn off the EX ESD after the ifo has achieved full lock, it is important for operators to know if it is all the way off or not. After talking with Evan, he cleared up some of my confusions and this is what I've found
There are three different states that the ESD can be in:
The confusion this morning arose from the ESD medm screen's "Active" light. The light was red but the Analog Monitors still had a few hundred counts to them. This is because the ESD DC Bias was OFF and the driver was on. To avoid this in the future, I added a third light to ESD Active. If the Active light is:
(Legend on the medm so you don't have to memorize that, screenshot attached)
Hopefully this will eliminate any future bewilderment.
Also, we updated the guardian to turn the EX ESD on and off as appropriate.
In LSC_FF, if the EX bias has been ramped to zero, the guardian will write 1 to H1:ISC-EXTRA_X_BO_3. This is then reset to 0 after a few seconds (by Beckhoff?). This will inactivate the driver.
In DOWN, if the analog readbacks for the driver are close to zero, the guardian will again write 1 to H1:ISC-EXTRA_X_BO_3. This is again reset to 0 after a few seconds. This activates the driver.
Occasonally it seems that the reset mechanism freezes, so that cycling BO_3 does not do anything. In this case, it usually works to do the following:
caput H1:ISC-EXTRA_X_BO_4 0
caput H1:ISC-EXTRA_X_BO_4 1
And then write 1 to H1:ISC-EXTRA_X_BO_3.
Glitches have been showing up in the IFO range since the lock that started at about 17:00UTC, and the rate of glitches has been about the same throughout the morning.
The crew took a break at 11:00 local time, and all glitches stopped. I verified that no one in the CR had made a change. Glitches resumed about 40 minutes later, which coincides with the crew starting to clean again.
20:22:00 UTC, beam tube cleaning starts.
The first IFO glitch in this lock happens within one minute.
Bubba has had the crew mark where they were working at the time.
Robert is heading down to investigate.
Glitch at 17:45:25UTC - H1:ASC-Y_TR_A_NSUM_OUT_DQ and H1:ASC-X_TR_A_NSUM_OUT_DQ gltich first.
The attached plot shows the tidal feedback to the ETM HEPIs. The available range is ±250µm. The maximum used drive so far was ~170µm. There seems to be a couple of times where the X end HEPI walked away between lock stretches.
Starting right around 21:00 local, the 25.4 hz (well, DTT says something like 25.37) peak showed back up, creating a nasty looking comb in the DARM spectra (first image). Unsure of what else to do, I turned the power down to 16 watts, the peak has now kind of subsided(second image). I'll wait to see if the peak settles down any more, then maybe turn the power back up.
Dan called in and helped me look for a PI with a template he had ready. There were 2 peaks rung up at 15516 and 15540, see attached plot. The main culprit causes the big bump in the RMS, the little peak next to it was also rung up more than Dan's quiet reference. We were trouble shooting when the IFO lost lock. Guardian is bringing everything back up.
Thanks, Jim!
The channel H1:IOP-LSC0_MADC0_TP_CH12 is the 64kHz-sampled IOP input for OMC DCPD A.
Recall that the frequency of this mode matches what Elli measured for our parametric instability.
The other approach would be to change your ring heater power. I guess you in the same situation as us from the fact that the previous step up in ring heater power was effective. So you need to increase your ring heater up from 0.5W per segment, which is what I think it was set to after your first observation of parametric instability. At Livingston if we increase the ring heater too much we then ring up a 15004Hz mode. There is a new wiki here for operators as we had an apparent change in the parametric gain after last vent and we are still in the process of finding a new operating point for the ring heater.
The two acoustic modes appear to ring up with a similar time constant see image, there is also a peak in DARM a bit further down ldvw link any idea if this is related? It's a bit big for an acoustic mode that is not ringing up.
I would guess that these two modes are ETMY and ETMX ringing up with the vertically oriented mode, as that is what we mostly see at Livingston. You could look at the transmission QPD channels for more information if you are interested.
At Livingston these channels are L1:IOP-ISC_EY_MADC1_TP_CH0-7 for X and Y end, pitch and yaw orientations can be derived.
Stefan, Kiwamu,
In this morning, DARM had many numbers of glitches which were visible in the DARM spectrum as wide band noise. We looked at various channels to see what caused the glitches, but we were not able to identify them.
The lock stretch which had a high glitch rate is the one starting at 2015-06-05 17:17 -sh to 18:00 -ish UTC. I attach an example time series of the glitches shown in OMC-DCPD A and B. We know that some of the loud ones were so large that they saturated the ADCs of the OMC DCPD.
Note that in the same lock stretch, we changed the DARM offset multiple times which changed the amount of the carrier light resonating in OMC. We do not think our activity with the DARM offset caused the high glitch rate.
Here is also a DARM spectrum and time series, as well as the summary page range graph. The glitches occur in the short science segment just before 18h.
We need to do some more thorough follow-up, but I can give some quick feedback that might give some hints. The initial signs seem to suggest that the problem might be due to input pointing glitches or alignment fluctuations in the PRC.
The loudest glitches in that short science segments were coincident with ASC-REFL_A_RF9_I_PIT_OUT_DQ, which looks like it was being used as the error signal for INP1 and PRC2 loops, feeding back onto IM4 and PR2. The corresponding YAW channel was also significant, but not quite as much so. We also saw ASC-REFL_B_RF9_I_PIT_OUT_DQ as somewhat significant, which looks like it was being used in both the INP1, INP2, and PRC2 loops and feeding back on IM4 and PR2.
So, we now start believing that these loud glitches were related to the cleaning activity on the X arm vacuum tube (alog 18992). Here I show glitch wave forms in time series from three glitch events (out of many) that were observed in this past Friday, 5th of June.
Typically the glitches were very fast and I am guessing that the glitch itself happens on a time scale of 10 ms or maybe less. Usually the OMC DC signals or the DARM error follows with a relatively slow oscillation with a period of roughly 100 msec which I believe is an impulse response of the DARM control. All the three events showed a power drop in the carrier light everywhere ( TRX, TRY and POP) indicating that the power recycling gain dropped simultaneously. For some reason POP_A_LF showed slower power drop which I do not understand. Also, in all three events that I looked at, the OMC DCPDs showed small fluctuation roughly 20 ms before the big transient happens.
(This one contains two glitch events apart only by roughly 200 msec)
(all the time series are high-passed with zpk([0], [40], 1) ). You can see a glitchy behavior in REFL WFS (as reported by TJ) as well as TRX QPD (and a little bit in TRY QPD).
Lock broke, we will keep it this way till the maintenance period is over or until we hear otherwise.