Following Keita's entry from last week, we are now running with the OMC in a lower-gain state to reduce the length actuation. First plot attached is a OLTF measurement of the OMC LSC loop while in low noise; the current Guardian settings bring us to the blue curve, with a UGF of about 15Hz. For the blue and green curves the OMC-LSC_SERVO_GAIN = 10, the red curve had a gain of 60 and a larger boost. We thought about turning off the boost entirely, which would give us the green curve -- for now we've left it on.
This gave us a good improvement in the upconverted noise around the OMC length dither line, see second figure which is the noise from late Friday night. (The areas of improvement compared to the blue reference, from Thursday morning, are annotated.) There is still some upconverted noise, see third figure, and we could reduce the OMC length gain even further.
To do: estimate the OMC length noise --> DARM coupling using Koji's noise measurements of the intrinsic length noise in the OMC to see how much length noise suppression we need.
PSL Status: SysStat: All Green, except VB program offline Output power: 31.9w Frontend Watch: Green HPO Watch: Red PMC: Locked: 5 days, 23 hours, 1 minutes Reflected power: 2.2w Transmitted power: 22.7w Total Power: 24.9w ISS: Diffracted power: 7% Last saturation event: 0 days, 8hours, 6minutes FSS: Locked: 0 days, 0 hours, 7 minutes Trans PD: 1.6v
SEI - hoping to get a new isolation filter installed on BS stage2.
SUS- nothing to report
CDS - ESD filters will be installed
3IFO - Tuesdat activities will include tagging and bar-coding of TCS and ISC items. Calum will be on site next week. Bubba is continuing to connect piping o containers for N2 purging.
FACILITIES - BTE cleaning is ongoing. Semi-annual lubrication of axial fans for the ventilation sytems will start today with the mid stations and may move to the corner statinos by today. Filters will NOT be removed.
TJ asked me about the BRS this morning, as it was tripping an alarm he had set. When I opened the overview, the numbers were rather large and swinging rapidly (over a minute it was going from -40000 cts to +40000 cts, the transition was a couple seconds. The attached dataviewer trend shows it has bee doing this since the 21st, starting about 15:00 UTC. No clue what happened.
A little bit more supporting data: I attach a time series of the drift monitor (the DC comparison between the autocollimator's fringe patterns for the balance mirror and reference mirror; in military green; H1:ISI-GND_BRS_ETMX_DRIFTMON) and control signal for the gravitational damping motor (in dark red; H1:ISI-GND_BRS_ETMX_DAMPCTRLMON). I suspect the suspension has drifted enough that fringe patterns have crossed causing the output of the sensor to go non-linear. This then rung up the suspension after the damper turned on thinking that the SUS had rung up. Then we get into a nasty feedback loop. Looks like the temperature in the XVEA has remained stable over the weekend (and for several days prior), so I doubt we can blame the temperature swings for the cause of the drift. For now, Jim and TJ have turned off the control software to let the suspension cool off. We'll do a more invasive investigation tomorrow morning.
I've attached a couple of plots showing the BRS_DRIFTMON channels. First one shows the 1-hr period when the BRS data started to grow large. It looks like some 'disturbance' set it off and then the damper took it away with a positive feedback loop because it was in the incorrect position.
The next plot shows the same channel over 3 days, starting from 2 days before the disturbance. It looks like the mean position was quite stable ( except for the odd missing dataset). Once the positive feedback loop got started, it drove the turn table harder and harder, generating heat and causing the mean position to shift slowly. Once things settle down, the mean position should return to ~ -2.9k counts. This is well within the linear range of BRS.
Due to the large amplitudes BRS reached before the software could be turned off, I'm not sure it will be sufficiently damped so it could be restarted tomorrow.
at 8:25 in an attempt to continue the locking sequence that had gotten "stuck" at engatge "ENGAGE_ASC". I selected "DC_READOUT_TRANSITION" to try and get it moving again. THis tripped the SUS RMS watchdogs on the ETMs. I was able to reset ETMY remotely but ETMX seems to need a PUM coil driver power cycle.
Filiberto power cycled ETMX PUM (and UIM accidentaly) and this cleared the UL RMS trip.
We should really just rip out these PUM driver RMS watchdogs, they're serving no protective purpose anymore -- unless there's something I don't know. It would be great if DetChar took a look at the many PUM driver watchdog trips that have happened over the past few weeks (using ${IFO}:SUS-${OPTIC}_BIO_L2_MON as the readback, an EPICs channel, so it should exist in the frames), and find out when the current goes over using the RMS current monitor, ${IFO}:SUS-${OPTIC}_L2_RMSIMON_UL_MON (and EPICs channel, so it should exist in the frames), and see if that corresponds to some actually dangerous level of current to the internal circuitry. If not, we rip this pesky thing out. Recall you can find all the information you could need in page 3 of the QUAD controls design description: T1100378. The only trickery that is poorly documented is that the binary input readback, ${IFO}:SUS-${OPTIC}_BIO_L2_MON, actually contains many status bits. Whether the UL, LL, UR, LR RMS current watchdogs are tripped are readback bits 2.^[1 5 9 13], which means you need to mask the bit word with the numbers 2, 32, 512, and 8192 to obtain the right information. The best "source" for this information is the QUAD's MEDM screen overview which has these masks, in concert with the first page of PUM driver's schematic, D070483.
model restarts logged for Sat 21/Mar/2015
2015_03_21 14:42 h1fw0
model restarts logged for Sun 22/Mar/2015
2015_03_22 11:28 h1fw1
both unexpected restarts.
The CDS web server cdswiki was dead this morning, and needed a reboot. It started normally, no explanation for the failure.
Evan, Lisa Evan started locking early in the afternoon. He collected 10+ successful locking sequences in a row without manual adjustment (WOW!), see first plot. However, in order to keep the interferometer locked on the operating point, he had to open most of the alignment loops, keeping closed only DHARD and BS. We realized later that it was due to some low frequency oscillation in the CHARD loop, which was triggered by closing the SRM/PRM loops. We decided to keep those loops opened for now. We then realized that the power build up has been consistently low for a while (second plot - the beginning of the plot corresponds toMarch 9 , when a manual adjustment of the ITMs brought the arm power to its maximum ~ 1100 counts), so we decided to go through the initial alignment and re-look at the ITMs position..and an earthquake happened.. P.S.: All of the above happened with DARM locked on RF.
This is a better plot with the trend of the arm power over the last 45 days. The build up started to be consistently low after the first week of Mach, with some attempts of bringing it back (but resulting in a not stable IFO, I am told).
Dan, Kiwamu,
Some reports from last night and a little bit from today.
(Non stationary behavior of noise floor in 20-100 Hz)
Last night we worked more on the L2 actuation in order to see what noise are below the ESD DAC noise. The recycling gain was as high as 27 and we could smoothly fully lock the interferometer multiple times. After we worked on the OMC trans fluctuation issue (see Dan's report), we transitioned to ETMY from ETMX. The L1 and L2 stages of ETMY were used. As we saw last time (alog 17367), the DARM spectrum showed a non stationary behavior in 20-100 Hz frequency band when small bias and quadrant voltages (< 100 V) were requested. The noise floor was breathing up and down on a time scale of ~10 seconds. We varied the ESD bias voltage to see if there is a "sweet spot", but because of the non stationary noise floor, we were unable to find it. On the other hand, having +380 V on ETMX ESD consistently brought us back to the previous noise floor. So this agrees with the hypothesis that we were limited by the ESD DAC noise. A funny thing is that changing the ETMY ESD bias did not seem to make the noise floor better or worse -- even with +380 or -380 V on the ETMY bias did not bring the noise floor visibly high.
At one point we noticed that the noise floor seemed to be correlated with DHARD YAW -- when the DHARD YAW deviates from the operating point, the noise floor in DARM seemed to go up. So we newly engaged a boost (to suppress 100 mHz variation) in DHARD YAW which seemed to somewhat reduce the non stationary behavior. We need to investigate this issue a bit more.
Here are some time to look at:
I attach a DARM spectrum starting at Mar-21-2015 8:30:00 UTC.
(Unable to damp ITMX roll mode)
Today I happened to be in the control room and so started locking. The recycling gain was as low as 24 today even though I went through the initial alignment. After reaching full resonance with RF darm, the roll mode of ITMX at 13.81 Hz was found to be as high as 10-12 m/sqrtHz in the DARM spectrum again (alog 17293). Initially I tried the same damping setting on ITMX, but I was unable to damp. Then I even tried the minus sign and +-60 deg phase rotators, but did not have a good luck. I wonder if this is related to the fact that the initial alignment today was far from optimum (i.e. low recycling gain).
I noticed that the X arm green PDH was not able to keep up with the longitudinal motion today. It kept unlocking every 5-10 seconds. I am not sure if this is due to high wind today. The roof wind monitors read 20 mph on average and reached 40+ mph at highest in this after noon.
I switched the ISI-ETMX stage 1 back to low blend, 45 mHz in X direction (which is exactly opposite of what Sheila's instruction says in alog 16583) without the BRS correction. This apparently helped the PDH locking as it became able to stay locked.
We are still getting RFM IPC errors on the LSC and ASC receivers of end station ISC channels. To gather statistics, I am running cron job on h1sysadmin0 which performs a DIAG_RESET on these models every minute to clear the latched IPC error. Currently the error rate is between 10 and 20 per hour.
model restarts logged for Mon 16/Mar/2015
no restarts reported
model restarts logged for Tue 17/Mar/2015
2015_03_17 05:42 h1fw0
2015_03_17 10:17 h1oaf
2015_03_17 11:39 h1pemcs
2015_03_17 11:55 h1lsc
2015_03_17 12:02 h1lsc
2015_03_17 12:06 h1dc0
2015_03_17 12:06 h1lsc
2015_03_17 12:08 h1broadcast0
2015_03_17 12:08 h1fw0
2015_03_17 12:08 h1fw1
2015_03_17 12:08 h1nds0
2015_03_17 12:08 h1nds1
2015_03_17 12:23 h1hpibs
2015_03_17 12:23 h1iopseib2
2015_03_17 12:23 h1isibs
2015_03_17 12:31 h1hpibs
2015_03_17 12:31 h1iopseib2
2015_03_17 12:31 h1isibs
2015_03_17 13:11 h1iopseib2
2015_03_17 13:11 h1iopseib3
2015_03_17 13:13 h1hpibs
2015_03_17 13:13 h1hpiitmx
2015_03_17 13:13 h1isibs
2015_03_17 13:13 h1isiitmx
2015_03_17 16:16 h1sushtts
2015_03_17 16:19 h1asc
2015_03_17 16:21 h1sushtts
2015_03_17 16:26 h1broadcast0
2015_03_17 16:26 h1dc0
2015_03_17 16:26 h1fw0
2015_03_17 16:26 h1fw1
2015_03_17 16:26 h1nds0
2015_03_17 16:26 h1nds1
2015_03_17 16:27 h1nds0
one unexpected restart, restarts of h1seib2,3 for unexpected timing error on h1seib2 chassis. Maintenance day, FE model changes with associated DAQ restarts.
model restarts logged for Wed 18/Mar/2015
no restarts reported
model restarts logged for Thu 19/Mar/2015
no restarts reported
model restarts logged for Fri 20/Mar/2015
2015_03_20 03:31 h1fw1
one unexpected restart.
Kiwamu, Dan
For a while now the OMC transmitted light has been flickering with an amplitude of ~20% of the full light level; this makes it look like the OMC is about to lose lock and generally puts everyone on edge. The first image is a ten second trend of OMC-DCPD_SUM to give a sense of the size of the excursions.
The RIN is dominated by a broad plateau from 1-4Hz with an ampltiude of about 0.01 RIN / rt{Hz}. see the second plot attached. Tonight we made some studies of this noise to pin down the source:
One thing to do is run BRUCO and see what turns up in the 1-4Hz band.
Really slow day. Eeryone either on vacation or at the LVC conference.
LOTS of time to play with the IFO. Locking is an issue due to some spiking low frequencies. Kiwamu is working on it.
06:15 P King out to LVEA near PSL rack
07:00 Cris and Karen in to LVEA
07:26 P King out of LVEA
08:26 Corey to MidY
10:00 Corey back from MidY and into the LVEA
10:30 Jodi and Gary back from Mids
11:11 Andres out to W Bay to retrieve tools
11:17 Andres out of LVEA
11:50 for the last 30 minutes there has been multiple seismic trips and consequently, TMS wd trips. No one is at the end station.
12:24 EX Pump Diff Pressure alarm
15:00 Dust monitor 15(Beer Garden) keeps going off.
16:00 weekend time
In the past two or three days, Sheila, Dan and I have worked more on switching of the ETM actuators from the ESD to L2 stage with the goal of reducing the ESD DAC noise (see Evan's latest noise budget). Last night, I was again able to reduce the ESD biases all the way to zero by actuating on the ETMY L1 and L2 stages (see previous attempts in alog 17267). The DARM spectrum improved in 30-200 Hz band as expected, but only by a factor of ~ 1.7. Lock was not so stable (which seemed to be related to a peak at 12.52 Hz) and it stayed locked only less than 10 minutes. I was not able to test different bias voltages yet.
(DARM noise)
Here are two DARM spectra, one with the 380 volts ETMX ESD bias (in purple) and the other with zero bias voltage (in red). The ETMY ESD had zero DAC voltage in all the quadrants and bias all the time. As you can see, the noise floor went down in 30-200 Hz band with a slight increase at around 20 Hz. Also a peak at 12.5 Hz has been prominent in this week (for example, look at alog 17355) and it looks like this peak rings up every time before we lose lock. The peak usually becomes clearly visible in the DHARD signals in time series several seconds before the lock loss.
(Actuation Scheme)
We used to switch off the ETMX ESD and switch on the ETMY L2 stage while keeping the ETMX L1 stage active (see alog 17267) in the last week. This time, we tried a different scheme in which we actuate the L1 and L2 stages of ETMY and we do not actuate ETMX at all. In a sense, this is a transition of the ETM suspensions from X to Y. Before trying the transition, we measured the transfer function of the L1 and L2 stages at different times in order to check the cross over frequency. The attached below is the ratio of the two transfer functions (i.e. L1 / L2) together with a LSC DARM open loop, L1 and L2 transfer function models:
As shown in the plot, the measurement can be mostly explained by the model. However, according to the model, the phase at 1.8 Hz touches 180 degrees and thus unstable at this frequency. Unfortunately our measurement incidicated that the ratio of L1/L2 exceeds unity gain at around 1.6 Hz and I noticed that this would be a potential risk of unstable loop. Indeed, this configuration gave me unstable DARM which rang up at 1.6-ish Hz as expected when I tried transitioning to ETMY. I decided to insert a lead filter around this frequency. With the lead filter (a simple zero-pole pair at 0.3 and 1.5 Hz respectively), the L1/L2 transfer function should now look like the following:
This has two cross-over points, one at 0.45 Hz and 1.8 Hz with phase mergines of 57 and 43 degrees. I installed the lead filter in FM8 of SUS-ETMY_L1_LOCK_L. This allowed a repeatable transition. The transition process is also coded in the ISC LOCK guardian.
Fred, Kiwamu,
So the 12.5 Hz oscillation happens not only when the interferometer is locked on DC readout but also when it is locked on ASQ. We had more than three times of lock losses today where we lost lock right after (typically in a few seconds) the 12.5 Hz peak rang up. Taking a close look, we noticed that a 17 Hz peak also rang up at the same time. This two-peaks-behavior seems consistent in each lock loss. Their peak heights typically show a slow modulation in the DARM spectrum (on the order of a few seconds) and eventually reaches as high as 10-13 Hz/sqrtHz (0.3 Hz fft-bandwidth) at which we lose lock. The attached is a screenshot of the DARM spectrum from a time when the interferometer was almost unlocking.
It turns out these 12 and 17Hz oscillations were my fault. Earlier in the week, we switched the HAM6 DC centering toplogy to use AS_A and the OMC degrees of freedom. But, I forgot to edit the part of the DRMI Guardian that sets the ASC input matrix. As a result, for the past two days we have been asking the DC3 centering loops to center the beam on AS_A, but we weren't giving them any input. With the correct DC centering loops on the instability does not appear.
We were able to replicate the 12Hz oscillation by disabling the AS_A centering loop before we increased the power. Also I was able to make it reappear by putting an offset into the AS_A pitch setpoint. At 10W input power, an offset of 0.1 counts was sufficient to excite whatever this mode is and break the lock. This is kind of a small offset - we should figure out how much margin we have during normal operations.
(It could be that we're sensitive to the spot position on AS_B, since any offset in the centering loops will also move the beam on AS_B. But we think that this noise comes in through DHARD, and that error signal is derived from AS_A. AS_B is used for MICH and SRC1.)
Attached is a trend around the time when we saw the oscilation grow during the first steps in the power-up sequence. The DC3 centering loops are turned on about halfway through the trend, and the spot on AS_A is immediately servoed to zero.
Slowly, over the past week, I have found a set of filter settings that damp the violin modes. In the tables below I identify the mode frequencies that are matched to each test mass; the matching was done based on which lines were suppressed or excited using bandpass filters in the L2_DAMP filter banks for each test mass. I also indicate which filter damps that particular line.
The filter settings are given at the bottom of each table. For all the L2_DAMP filter modules, FM1 is the bandpass filter (with +120dB gain in the pass band), FM2 and 3 are phase shifters (+/-60deg respectively), and FM4 is a 100dB gain. FM1 and FM4 are always on. FM2 and FM3 are used as indicated. The gain settings are included in the Guardian - the filters shouldn't change in the locking process, but I haven't updated the safe.snap files for the suspensions!
ITMX | |
f (Hz) | damped using... |
500.053 | MODE3 |
500.210 | MODE3 |
501.091 | MODE6 (still too high) |
501.253 | MODE3 |
501.450 | MODE3 |
502.620 | MODE3 (still too high) |
502.743 | MODE3 |
MODE3 | 0deg, -200 |
MODE6 | -60deg, +400 |
ITMY | |
f (Hz) | damped using... |
501.680 | MODE5 |
501.747 | MODE6 |
503.007 | MODE3 |
504.857 | MODE2 |
MODE2 | 0deg, -100 |
MODE3 | +60deg, |
MODE5 | -60deg |
MODE6 | +60deg |
ETMX | |
f (Hz) | damped using... |
504.870 | MODE6 |
505.585 | MODE6 |
505.708 | MODE6 (still too high) |
505.805 | MODE4 (still too high) |
506.921 | MODE6 |
507.157 | MODE6 |
507.389 | MODE6 (still too high) |
MODE4 | -60deg, -300 |
MODE6 | -60deg, +100 |
ETMY | |
f (Hz) | damped using... |
508.008 | MODE5 |
508.145 | MODE5 |
508.204 | MODE5 |
508.219 | MODE5 (still too high) |
508.583 | MODE5 |
508.659 | MODE5 |
1008.49 | MODE3 |
1009.03 | MODE3 |
1009.44 | MODE3 |
1009.49 | MODE3 |
~1484.5 | MODE6? |
MODE3 | +60deg, +100 |
MODE5 | -60deg, -100 |
MODE6 | -60deg, +10k (?) |
Unaccounted for: | |
f (Hz) | best guess |
501.606 | ITMY |
501.810 | ITMY |
503.119 | |
504.803 | |
507.192 | ETMX |
507.991 | |
508.288 | ETMY |
508.659 | ETMY |
The attached figure compares the noise spectrum from Friday and today. The first harmonics of the violin modes are now the dominant feature in the spectrum, along with 60Hz and the 2.5kHz BS butterfly mode.
I forgot to note the gains for the ITMY filters. They are:
Dan, Sheila, ITMX
While testing the vioin mode damping tonight, we had an abrupt lockloss, and for a short period my bandpass filter at 501.094Hz was sending noise x120dB into the ITMX L2 coils. This tripped the coil current watchdog, just like a previous event on ETMY a few weeks ago. (There are now limiters on all the violin mode damping filters...)
Unlike ETMY we can't reset this watchdog by toggling the RMS Reset epics channel. Anyone know how to clear this error?
We suspect something in hardware is preventing a software reset. In the CDS highbay, the SUS Independent Watchdog electronics box has red lights that won't reset when we use the 'Fault Reset' button.
I asked the question about how to clear it when I was designing an alert on the Ops Overview screen for it. As I recall, all you have to do is change the value in the field from a 1 to a 0 and then back again? Also, I think Stuart Aston told me that watchdog would eventually go away.
So, normally changing that value back and forth WOULD be the way to clear it according to Stuart. This did not work. He then suggested I simply power cycle the PUM so with Sigg's approval I did and that seems to have worked. The L2 RMS watchdog trip on ITMX has been cleared. On a side note, this 'SUS Ind WD' chassis is ONLY cabled to the BIO I/O chassis for ITMY. It IS NOT documented on any of the current drawings and it isn't commissioned. It's current state can't be reset with the button on the front so I assume that it would have to be power cycled to be cleared. That being said, it isn't really connected to anything.
Currently experiencing the same behavior with ETMX L2 UL. Toggling the RMS WD reset does nothing.
Also, the ETMX indicator on the ops screen does not indicate this fault (the border is still green).