In the continuing saga of the Ring Up of the BRS, I've turned off the damper. I turned it on yesterday thinking that 2 days was sufficient rest for the BRS to settle down, but Krisha reports that it probably needs more like 4-5 days, given current levels of excitement. It would be best if people avoided the end station, as the BRS is now completely uncontrolled.
The damper is controlled by a line of code that sets the variable TestData (or some variation of capitalization thereof). The normal line sets this equal to 2.5 - outvelocity. To disable there is a line immediately below that is commented out that sets TestData to just a constant (2.5). Uncomment this line and restart the code. Done.
Replaced Shutter Controller S1203610 with S1203612. Daniel reported CH1 on S1203610 is not functioning, thus we were controlling the shutter with CH2. I reconnected the shutter to CH1 of the new unit and changed the jumper settings to VS35 on both channels.
model restarts logged for Wed 25/Mar/2015
2015_03_25 14:33 h1nds1
2015_03_25 14:43 h1fw0
both unexpected restarts.
Loss of lock probably due to my work at EY on ESD drive. Following up on last nights report I went down to EY to investigate problems with the ESD. Much to my surprise aside from the unit being in the off state I could not find a problem. The HV supplies were on and at nominal current draw for no drive (3.2mA). Went in to check the ESD Chassis and all the lights were green. The only indication of a problem was the main indicating light for the unit showed it to be in the off state. I had disabled all of the DAC inputs to the driver so I turned it on. Went over to computer and enabled outputs and immediately saw a response on the OpLev. I am not sure what interlock could have shut the unit down but further investigation is warranted. I have left the unit on for now. please keep me Richard McCarthy informed if this or more problems arise.
When I was characterizing the ESD several years ago I noticed that the microprocessor that controls the device would turn off the system when a spike in either + or - direction on the 430 volts input occurs. You might want to check the tolerance on the input voltage of both the positive and negative input voltages.
Evan, Lisa
So, we leave the interferometer with:
After all of this we saw that the range was around 20 Mpc, even without the low noise ETMY ESD, and without any MICH/SRCL feedforward, so we might indeed have made some improvement.
As far as we can tell right now, the locking sequence tomorrow should work as usual.
Evan, Dan, Lisa
Dan and I wanted to leave, but Evan forced us to stay...so, here we are.
Dan and Evan, the experts in roll and bounce damping science present in the Hanford control room at mid-night, eventually succeeded in damping.
So we relocked to DC readout, right before the transition to low noise ETMY, and switched the coil drivers for PRM-PR2-SR2 (tested at least twice, now in the Guardian) and BS in their final low noise state (for BS we used the script which switches one coil at the time, so this transition is not included in the Guardian yet). We also switched ETMX, ETMY, ITMX, ITMY L2 to their final low noise state (state 3), but we are not including this step in the Guardian yet as we tested it only once (we modified the down state to go back to the acquisition mode for the quads).
We also investigated why the transition to ETMY stopped working by measuring the DARM open loop while actuating on both ETMX and ETMY at the same time.
As far as we can tell, the transition is not working simply because the ETMY ESD is not doing anything at all.
Please go to EY tomorrow morning and check the ESD driver.
Done, Could not find a problem aside from unit being off. See log entry
Evan, Kiwamu, Sheila, Dan, Lisa
We still haven't quite finished our list of "low hanging fruits", but we are getting close. The OMC is not shaking anymore after engaging the DARM boost and reducing the RIN in transmission of the OMC by nearly a factor of 10. Also, we switched the vertex control from POP to POPAIR, which has ~5 times more light, and in doing so we reduced the noise coupling from MICH to DARM. We could reach ~ 28 Mpc again without engaging the MICH feed-forward.
Sadly, we don't have any improvement in Mpc units because a few "technical problems" slowed us down, and we didn't have a chance to finish the coil driver transitions to low noise, improve the vertex length DOF loop cut-offs, and tune the MICH/SRCL feed-forward subtraction while on POPAIR.
In particular, a typo in the ISC_LOCK Guardian (switching of a filter in the "DAMP_M0" filter bank, instead of "DARM_DAMP_MO") badly tripped ETMY (thanks to Jeff who helped us recovering by phone), and caused both roll and bounce mode to run up insanely, and we can't lock. Our damping techniques with ifo unlocked are very (very) slow, so we are giving up.
Also, the transition to ETMY with low noise ESD worked well once today, but later attempts failed (not totally clear why, the only known difference in later attempts is that we engaged the DARM boost before making the transition).
The full list of things we did/tried is below.
Fixed the "shaking" OMC (in the Guardian)
We reduced the DARM motion between 1 and 6 Hz, as Dan reported here, and now the OMC is happy (and we are happy with him).
Transitioned to POPAIR (in the Guardian)
POPAIR has ~5 times more light than POP, so we decided to transition from the in vac diode to the in air one, to reduce the noise coupling of the vertex length DOF to DARM. Indeed, we could reach the same sensitivity as the other day (~28 Mpc) without any MICH feed-forward.
Close ALS shutters on ISCTEX/EY (in the Guardian)
Not clear improvement, but this is the beginning of the "closing beam diverter/shutter" campaign.
Test of SRCL cut-off (not in the Guardian yet)
The SRCL loop had 60 degrees of phase at 30 Hz, and a very modest cut off filter. We tested a more aggressive cut-off, which removed all the coherence we see between DARM and SRCL above 100 Hz. We tested this filter only once, so we are not including it yet in the locking sequence.
Test of BS coil driver switch in DRMI (not in the Guardian yet)
Kiwamu and Evan imported and adapted the script that Den wrote to do the BS coil driver switching one by one. It was tested in DRMI - 1f, and it worked. We haven't tested it in full lock yet.
Dan, Evan, Kiwamu, Lisa, Sheila (and earlier advice from Keita)
Today we turned on the DARM boost filter described here, to increase the suppression in the 1-6Hz band. The boost itself was no problem, but we realized that changing the phase of the open-loop gain around 10Hz changes the sign of the ETMY and ITMX bounce modes in the DARM error signal, which means the sign of the damping loops need to change as well. This led to a few hours of bounce mode excitement, but the new signs and phases are now in the Guardian.
The first plot is a comparison of the OMC TRANS intensity noise before and after the boost filter was engaged. The RMS before the boost was 2e-2 /rt[Hz], now it is 3e-3 /rt[Hz]. The OMC TRANS camera is much calmer.
The boost filter also reduced the coherence between the OMC length error signal and DARM. Yesterday, Keita pointed out that the OMC length loop was far, far too coherent with DARM between 1-6 Hz, and this noise was being upconverted around the dither line at 3.3kHz. The upconversion mechanism is not clear; if the OMC length is properly servoed the dither should not generate any RIN (or, very little) and should not upconvert any low-frequency RIN noise to sidebands around the dither frequency. But, that's what's happening. The new boost filter reduces the coherence at low frequency (second plot) and reduces the sidebands at 3.3kHz (third plot), but the coherence at low frequency and at 3.3kHz is not zero (second, fourth plots).
As a precaution against OMC LSC --> DARM coupling, we have reduced the gain of the OMC length loop even further: the OMC LSC boost is now off, and there is a 30Hz rolloff filter (FM3) in the OMC length loop. The UGF is a little more than 10Hz, as described by the green trace from the first figure in this entry.
Keita also recommended changing the dither frequency and changing the dither amplitude and OMC LSC loop gain. We haven't had a chance to do these things tonight. We need to investigate why this noise is being upconverted - note that the amplitude of the 3.3kHz dither line in the OMC RIN increased after the OMC LSC boost was turned off (third plot).
The good ETMY and ITMX bounce mode damping settings with the DARM boost ON are (in M0_DARM_DAMP_V):
The good settings with the boost OFF are:
This switching is handled by the Guardian.
By the way, an ongoing mystery is the source of a 300Hz line in DARM that is very large at the beginning of a lock but steadily decays to the level of the noise floor over a timescale of ~10-20min. The rate of decay is somewhat matched to the BS butterfly mode and the earlier twin peaks noise, but the line is present when we're locked on RF readout, so it doesn't seem to be coming from the OMC. We've checked the IOP inputs that are sampled at 65k and we don't see a line at 16384+/-300Hz that could be aliased, but we should check again at the very start of a lock when the 300Hz line is very prominent.
why use the error signal for bounce mode damping, when the DARM control signal is available?
the phase of G/(1+G) changes very little as the loop gain changes in the case that G>>1 (which should be the case with the Bounce/Roll RG filters on)
Rana,
This is a good point. We are planning to change the pick off point so that it takes the DARM signal from its output as you suggested.
J. Kissel WP #5118 After building the MEDM infrastructure for the calibration lines and optical gain compensation (a.k.a. "gamma"; see ) I realized a large fraction of calibration line path from its generation deep inside the GAMMA_LINE1_DEMOD block to its injection into the DARM loop has no readbacks or test points. Similarly, the calculation of each optical gain compensation coefficient has a ton of steps with out EPICs records to show the progress along the way. As such, I've installed several new EPICs records in the h1calcs.mdl (or, more accurately in the CAL_CS_MASTER.mdl and CAL_GAMMA_CALC_MASTER.mdl library parts) and installed an EPICs record / Test Point pair in the h1omc.mdl (again, more accurately in the omclsc.mdl library part), right before its injected into the DARM loop. See attached screen shots. I've chosen to store the new test point at the full 16 [kHz] in the frames, because I expect we'll want to make comparisons of the CALCS to OMC model delay over long stretches of time in the future to ensure the phase lag between generation and injection remains stable against, say, computer reboots. The new test point channel in the OMC will be ${IFO}:OMC-LSC_CAL_LINE_SUM_DQ where the already existing channel in the CALCS model is ${IFO}:CAL-CS_LINE_SUM_DQ remember that this is *sum* of all calbiration lines. The test point for each individual line, ${IFO}:CAL-CS_GAMMA_LINE${lineNum}_DEMOD_LO is *not* stored in the frames. We expect that the lines will be sufficiently far apart that a simple band-pass on the noise-free digital signal should be adequate to isolate each line if necessary. These new models have been compiled and committed to the userapps repo, and I'll install them tomorrow morning and restart the DAQ.
= Ops Summary =
7:00 - 8:00 Cris and Karen to LVEA
~9:00 Filiberto to LVEA near HAM4 (grab some stuff)
9:26 Karen to Mid Y and Cris to Mid X
11:19 Karen leaving Mid Y
11:50 Kyle to Mid X
12:30 - 13:00 ITMX and ITMY cable swap (Jeff K. Refer to alog17466 )
16:24 Jeff K. to log himself out at CDS highbay
It's been a quiet day today....
Scott L. Ed P. Chris S. Relocated lights and equipment to next section of tube, single door to X-1-7 double doors. Refilled D I water tank and cleaned 15 meters of tube this afternoon. Tube pressures monitored by control room operator during cleaning operations.
I added a simple script to /opt/rtcds/userapps/release/isi/h1/scripts/ called BRSswitch.py
You can use this to turn ON or OFF the BRS sensor correction. All this does is change the X2 and X3 matrix values in the STS2CART matrix so that when it is ON the the BRS signal is subtracted, and when it is OFF the signal stops there.
To run this script, navigate to /opt/rtcds/userapps/release/isi/h1/scripts/
Then:
python BRSswitch.py ON
or
python BRSswitch.py OFF
The purpose of this script is to eliminate confusion over how to turn it on or off and to allow for a method that can adapt to changes in the BRS system. As of now this script is specific to this BRS at EX, but can easily be made more general when/if there are more here or at LLO.
J. Kissel, F. Clara, E. Hall The commissioning vanguard had discovered that a portion of the ITMX M0 Top Driver's Binary Input/Output (BIO), which is control of the low-pass and test/coil enable switches, was malfunctioning a few days ago (see LHO aLOG 17449). This afternoon, Fil, Evan, and I ventured into the CDS highbay to diagnose the problem. After a cable-to-cable swap at the coil driver between ITMX M0 F1F2F3SD BIO cables and the ITMY R0F1F2F3SD BIO cables, and cycling both suspensions' top mass digital control and readbacks, we identified that *one* bit (the F1 bit) of the ITMX M0 F1F2F3SD BIO remained stuck (with the otherwise functional ITMY cables). This made us suspicious of the BIO chassis. However, due to time constraints, we couldn't diagnose further. Instead, we restored the nominal configuration -- and this FIXED the problem. Thus, happily, the problem is gone and all of ITMX driver channels can switch as expected, but, sadly, we don't know why our cable Chinese fire drill fixed the problem. ----------- Other details: Follow along using pgs 8 through 14 of D1100022 where the BIO connections happen. Remember that the top mass cabling -- even through the SUS have 6 OSEMs on the main and reaction chain -- is grouped in groups of four OSEMs. For a given QUAD, the break down is Set 1: M0F1 M0F2 M0F3 M0SD Set 2: M0LF M0RT R0LF R0RT Set 3: R0F1 R0F2 R0F3 R0SD thanks to the mechanical arrangement of the cabling in vacuum (see T1100327). What we saw initially: When we requested to switch the M0 "coil driver" (the main chain uses one-and-a-half top drivers) filter state request to go from State 1 to 2, it switched only the M0F1F2F3SD set, and it switched the test/coil enable bits instead of the filter state. This was reproducible over many attempts. All other functionality -- switching ITMX R0 state and test/coil enable was functional -- indicating a problem only with this cable/OSEM set. We confirmed that all cables were connected and secured screwed in as designed in D1100022 before doing anything. We swapped the cables at the coil driver, namely (on pg 9 of D1100022) we exchanged cables CAB_L1:SUS_ITMX-102 and CAB_L1:SUS_ITMX-103 in the SUS-C5-37C top driver position/chassis with the CAB_L1:SUS_ITMY-100 and CAB_L1:SUS_ITMY-101 in the SUS-C5-39C top driver position/chassis. With all cables connected in this fashion, the ITMY M0 and R0 BIO was entirely functional bhaving as expected. The ITMX M0 and R0 BIO was *more* functional, in that switching the filter state request actually switched the filter, EXCEPT that the F1 LP bit remained stuck. We then quickly assumed that this series of events was symptomatic of a problem with the ITMX BIO chassis, but we ran out of time to create enough test scenarios that we're confident. Running out of time, we re-exchanged the cabling, restoring everything to how it was, and to how its laid out in D1100022. However, after toggling the ITMX and ITMY BIO switches again, to confirm that functionality had been restored to ITMY and the same broken behavior was present on ITMX, we found that all problems had been fixed. YUCK. Well, YAY, but... YUCK. We'll keep an eye on this BIO chassis to see if it goes screwy again, but for now we consider the problem (dis-satisfyingly) fixed.
The BRS software was restarted at ~12:30PM. The device is currently disabled. The INMON signal is very clipped/saturated. The wind speed at EX is currently 20mph+. We still think there is a fault with the equipment. Krishna should be contacted.
Now the whole initial alingment procedure can be done using ALIGN_IFO, which will hopefully reduce confusion and the need to init different guardians. We have replaced the ISC DOF with the new diagnostic guardian on the ISC overview screen.
Also, the states which used to be called SRM ALIGN and OFFLOAD SRM ALIGN are now SRC ALIGN ect since they are really aligning both SR2 and SRM. Nutsinee and Ed are working on updating the procedure for operators.
I've edited the operator WIki instructions to reflect this change. If anyone catches something that I missed while trying to align/lock please let me know. -E
Laura, Josh, Andy, Joe, Detchar
We took a look at the 28 Mpc lock on 24th March to see if there was any evidence of arches that we have seen previously at LLO (for example LLO alog 17090).
The first attachement shows a histogram of the rate of glitches (omicron triggers) binned by the value of IMC-F at the same time. The plot looks pretty Gaussian, except for a few bins specifically between IMC-F values of -37.5 to -41.7 and -45.8 and -50. Between these values we see evidence of arches in DARM. The second attachment is a pdf file showing the arches in DARM lining up with values of IMC-F in the second bin. Here are links to many spectograms confirming the arches in the IMC-F bins outlined (spectrograms for IMC-F bin -37.5 to -41.7 / spectrograms for IMC-F bin -45.8 to -50). Each spectrogram time corresponds to a glitch as seen by omicron.
LLO has dramatically reduced the effect of these arches, see LLO alog 17191.
Also the value IMC-F has when these arches appear in DARM seems to drift, i.e. there does not appear to be a specific value of IMC-F which produces the arches (this is evident in the second attachement). This is not what we have seen at LLO, does anyone know what could be causing this?
Andy, Laura, DMac Here's a look at one of the whistles, and the putative cause as an RF beatnote with IMC-F. The idea is that the IMC VCO is beating against another (nearly) fixed RF oscillator, and that's what we see in DARM. In that case, the frequency track should look like the difference between IMC-F (calibrated in kHz) with some fixed value. In fact, the whistle clearly seems to match half of the frequency difference (black trace) rather than the full difference (red trace). Perhaps this hints at the type of coupling? Here, the fixed value is about -46.9 kHz (relative to whatever IMC-F's reference is), but we think this drifts during the lock. We can look in detail at some more of these, and find out.
Why not use the frequency readbacks of the oscillators and VCOs rather than the somewhat arbitrary IMC_F? The radbacks are only once per second, but they can still tell you which units are close in frequency.
I'd be happy to look at the readbacks, as long as they are recorded and available offsite. Can tell me what the channel names are?
I put some notes about these channels in the summary of the ISC F2F last fall. ("RF crosstalk" section)
The channel names and labels are in the attached code. There, I look at the time around the event that Andy sent around to the DetChar mailing list yesterday. It's not totally obvious what VCO is beating against the PSL VCO, the closest one is ALS COMM, but it's about 400kHz away at the time of the glitch.
The first plot shows the value of all the corner station VCOs in the ten minutes around the glitch, the second plot is a zoom around the PSL VCO frequency.
It seems like H1:SYS-TIMING_C_FO_A_PORT_11_SLAVE_CFC_FREQUENCY_5, which Dan has labeled as the PSL readback, is the best candidate. This seems to be roughly equal to (1/2)*IMC-F + 79.225 MHz, delayed by one second (i.e. I can line those two things up and they overlap). Is the one second delay due to some loop or is it just a quirk of the recording system? The whistles occur whenever this readback crosses 79.2 MHz, which is the fixed fiber AOM frequency from the talk referenced in Dan's notes from above.
Time Code Translator work at end stations, WP5116
Dave, Jim:
We racked up the missing TCT in EY, then connected it to one of the unused fiber pairs running between the VEA rack and the computer rack. We verified it was using ports 19 and 20 in the MSR (same as EX). Once connected the TCT synced to the time and the error LED on the MSR sender was extinguished.
We then ran 30 foot 50-Ohm coax, BNC terminated, between the TCT in the computer rack, through the wall, to the first port on the CER comparator. This was done at both EX and EY. The 1PPS port on the back of the TCT is a TNC connector, so we used a TNC-BNC adapter on that port for EX. We need to do the same for EY.
At EX, the comparator is now displaying a time of -22.2uS on the TCT port, roughly the time of flight between the buildings. We will revisit the TCT internal settings, they are most probably what was left from iLIGO.
Rebuild of ASC model
Daniel, Dave:
Daniel though h1asc would benefit from a clean build, install and restart. This was done, I'll see if this helped in the IPC errors.
PCAL filter modules, port from PEM file
Sudarshan, Dave:
Sudarshan discovered that I had forgotten to migrate the PCAL filters from when the CAL system shared a core with the PEM. He copied them from H1PEME[X,Y].txt to H1CALE[X,Y].txt, making a CAL_PCAL to PCAL name change in the process. The filters were then loaded into the PCAL models at both end stations.
The TNC to BNC adapter has been installed to connect the EY TCT to the timing comparator. Remaining work is calibration of the CsIII frequency standard, the time distribution system in the corner stations, and the TCT at the end stations.
Since we are frequently running out of range on the end station VCOs, when there is high wind, we switched the drive of the fiber AOM back to the IMC VCO. This was the original scheme. It has the disadvantage that the end station green locking now depends on the IMC to be locked. On the other hand, this is probably not a big deal, when working with the fully locked interferometer.
Kiwamu, Sheila,
We found that we had to lock ALS DIFF and COMM sequentially with this change, instead of in parallel as we have been doing. This morning we found that we could not transition the tidal to common with this change, so we reverted it so we are again using the fixed frequency RF source for the fiber AOM. We labeled both of the cables as well, so that next time the wind picks up we can put this back. We will probably not need high bandwidth common tidal with this modification, so we can try turning this bandwidth down and leaving off the IMC F offset, then later engaging it with a slow ramp.
Same story. Channel 1 opens and closes immediately. Next thing to check are cabling and slow controls.