A couple of plots from last night. We had an incredibly stable ~4h lock with 20 Mpc range. All the "standard" alignment loops were enaged (the only uncontrolled DOFs are comm/diff ITMs).
Recall that in this lock DARM was controlled by actuating on high noise ESD on ETMX.
Dan, Elli. EX ring heater power supply is turned off. Alll other ring heater power supplies are still on There is a coherence measured between EX magnetometer and DARM at 55Hz (see Gabriele's alog 17423), so we are checking whether turning off the ring heater power supply removes this line.
The 55MHz line didn't disappear, so we have turned off ITM ring heater power supplies also.
I've continued my work on improving BSC St2 blends, I think I have something fairly Hippocratic. I moved the St2 ellips down to 1.5 Hz and added a pole at 10 Hz. This affects the St2 CPS phase (~24 degrees at the blend frequency) but this doesn't seem to negatively affect the performance. I've talked to the commissioners about trying this during a lock, I think I can install and turn the new filters on without breaking anything.
First plot shows my modifications. Red is stock, green is the "old" again, blue is my current.
Second plot shows the performance. Pink is ground before, green is ground after, blue is before, dashed and solid are CPS and GS13 respectively, red is after. Does pretty good between 1 and 20 hz, doesn't seem to do any worse below 1 hz. I think all of the differences below 1 hz can be attributed to the ground.
Jeff suggested I check LLO' blends to see if they had made a similar change, and it turns out they have. A nice demonstration of convergent evolution. Attached plots show the comparisons. The LLO blend has more gain peaking, but keeps closer phase up to the blend frequency and has more high frequency roll off. My elliptics dont afftect the gain peaking, but don't roll off as fast around 10 hz, but at that point my performance plots show we are already hitting GS13 noise.
Sheila, Hugh
IFO lost lock ~1426utc, Richard suspects it was his testing of the ESD. Almost 45 minutes later, the ISCINF_LONG stepped to its rail, LIMIT set to 250000nm. And I mean stepped, ~1 second or less. This of course put a bit much demand on the HEPI loop. It spiked its output and pretty much instantly tripped the watchdog on the Actuator followed by the ISI shortly thereafter. The HEPI and ISI reisolated normally.
The first attached plot shows the lock loss. This 30 minute trend has the DARM OUT with the ISCMON_X_INMON quickly going to zero and the HEPI Outputs moving in response. No problems here, HEPI does fine. You also see the arm or something grab again as you can see the Tidal drive to the HEPI start again but this only lasted for a couple minutes.
The second plot attached zooms out to 2 hours, with the IFO lock loss seen on the ISCMON on the left. ~45 minutes later the DARM steps, this time stepping the ISCMON and tripping HEPI and about 8 seconds later, the ISI (not shown.)
In the third attachment, Sheila dug around in LSC/ALS space and put together some trends showing the spike into HEPI originating when the ALS-X_LOCK_STATE steps from level 2 to 6 (PLL Lock to Red Lock.) This is unrealistic as the Lock state dropped from Red Lock just ~90 seconds earlier. Notice the TIDAL_CTRL_INMON steadily growing (DARM not Cleared.)
Sheila is looking deeper into the Guardian feeling it may have failed to do something it possibly should have to prevent the DARM OUT from going to the rail and the unreasonable rapid transitions of the ALS LOCK STATE.
There were two things in place that should have prevented this but didn't.
In the continuing saga of the Ring Up of the BRS, I've turned off the damper. I turned it on yesterday thinking that 2 days was sufficient rest for the BRS to settle down, but Krisha reports that it probably needs more like 4-5 days, given current levels of excitement. It would be best if people avoided the end station, as the BRS is now completely uncontrolled.
The damper is controlled by a line of code that sets the variable TestData (or some variation of capitalization thereof). The normal line sets this equal to 2.5 - outvelocity. To disable there is a line immediately below that is commented out that sets TestData to just a constant (2.5). Uncomment this line and restart the code. Done.
Replaced Shutter Controller S1203610 with S1203612. Daniel reported CH1 on S1203610 is not functioning, thus we were controlling the shutter with CH2. I reconnected the shutter to CH1 of the new unit and changed the jumper settings to VS35 on both channels.
Same story. Channel 1 opens and closes immediately. Next thing to check are cabling and slow controls.
model restarts logged for Wed 25/Mar/2015
2015_03_25 14:33 h1nds1
2015_03_25 14:43 h1fw0
both unexpected restarts.
Loss of lock probably due to my work at EY on ESD drive. Following up on last nights report I went down to EY to investigate problems with the ESD. Much to my surprise aside from the unit being in the off state I could not find a problem. The HV supplies were on and at nominal current draw for no drive (3.2mA). Went in to check the ESD Chassis and all the lights were green. The only indication of a problem was the main indicating light for the unit showed it to be in the off state. I had disabled all of the DAC inputs to the driver so I turned it on. Went over to computer and enabled outputs and immediately saw a response on the OpLev. I am not sure what interlock could have shut the unit down but further investigation is warranted. I have left the unit on for now. please keep me Richard McCarthy informed if this or more problems arise.
When I was characterizing the ESD several years ago I noticed that the microprocessor that controls the device would turn off the system when a spike in either + or - direction on the 430 volts input occurs. You might want to check the tolerance on the input voltage of both the positive and negative input voltages.
Evan, Lisa
So, we leave the interferometer with:
After all of this we saw that the range was around 20 Mpc, even without the low noise ETMY ESD, and without any MICH/SRCL feedforward, so we might indeed have made some improvement.
As far as we can tell right now, the locking sequence tomorrow should work as usual.
Evan, Dan, Lisa
Dan and I wanted to leave, but Evan forced us to stay...so, here we are.
Dan and Evan, the experts in roll and bounce damping science present in the Hanford control room at mid-night, eventually succeeded in damping.
So we relocked to DC readout, right before the transition to low noise ETMY, and switched the coil drivers for PRM-PR2-SR2 (tested at least twice, now in the Guardian) and BS in their final low noise state (for BS we used the script which switches one coil at the time, so this transition is not included in the Guardian yet). We also switched ETMX, ETMY, ITMX, ITMY L2 to their final low noise state (state 3), but we are not including this step in the Guardian yet as we tested it only once (we modified the down state to go back to the acquisition mode for the quads).
We also investigated why the transition to ETMY stopped working by measuring the DARM open loop while actuating on both ETMX and ETMY at the same time.
As far as we can tell, the transition is not working simply because the ETMY ESD is not doing anything at all.
Please go to EY tomorrow morning and check the ESD driver.
Done, Could not find a problem aside from unit being off. See log entry
Evan, Kiwamu, Sheila, Dan, Lisa
We still haven't quite finished our list of "low hanging fruits", but we are getting close. The OMC is not shaking anymore after engaging the DARM boost and reducing the RIN in transmission of the OMC by nearly a factor of 10. Also, we switched the vertex control from POP to POPAIR, which has ~5 times more light, and in doing so we reduced the noise coupling from MICH to DARM. We could reach ~ 28 Mpc again without engaging the MICH feed-forward.
Sadly, we don't have any improvement in Mpc units because a few "technical problems" slowed us down, and we didn't have a chance to finish the coil driver transitions to low noise, improve the vertex length DOF loop cut-offs, and tune the MICH/SRCL feed-forward subtraction while on POPAIR.
In particular, a typo in the ISC_LOCK Guardian (switching of a filter in the "DAMP_M0" filter bank, instead of "DARM_DAMP_MO") badly tripped ETMY (thanks to Jeff who helped us recovering by phone), and caused both roll and bounce mode to run up insanely, and we can't lock. Our damping techniques with ifo unlocked are very (very) slow, so we are giving up.
Also, the transition to ETMY with low noise ESD worked well once today, but later attempts failed (not totally clear why, the only known difference in later attempts is that we engaged the DARM boost before making the transition).
The full list of things we did/tried is below.
Fixed the "shaking" OMC (in the Guardian)
We reduced the DARM motion between 1 and 6 Hz, as Dan reported here, and now the OMC is happy (and we are happy with him).
Transitioned to POPAIR (in the Guardian)
POPAIR has ~5 times more light than POP, so we decided to transition from the in vac diode to the in air one, to reduce the noise coupling of the vertex length DOF to DARM. Indeed, we could reach the same sensitivity as the other day (~28 Mpc) without any MICH feed-forward.
Close ALS shutters on ISCTEX/EY (in the Guardian)
Not clear improvement, but this is the beginning of the "closing beam diverter/shutter" campaign.
Test of SRCL cut-off (not in the Guardian yet)
The SRCL loop had 60 degrees of phase at 30 Hz, and a very modest cut off filter. We tested a more aggressive cut-off, which removed all the coherence we see between DARM and SRCL above 100 Hz. We tested this filter only once, so we are not including it yet in the locking sequence.
Test of BS coil driver switch in DRMI (not in the Guardian yet)
Kiwamu and Evan imported and adapted the script that Den wrote to do the BS coil driver switching one by one. It was tested in DRMI - 1f, and it worked. We haven't tested it in full lock yet.
Dan, Evan, Kiwamu, Lisa, Sheila (and earlier advice from Keita)
Today we turned on the DARM boost filter described here, to increase the suppression in the 1-6Hz band. The boost itself was no problem, but we realized that changing the phase of the open-loop gain around 10Hz changes the sign of the ETMY and ITMX bounce modes in the DARM error signal, which means the sign of the damping loops need to change as well. This led to a few hours of bounce mode excitement, but the new signs and phases are now in the Guardian.
The first plot is a comparison of the OMC TRANS intensity noise before and after the boost filter was engaged. The RMS before the boost was 2e-2 /rt[Hz], now it is 3e-3 /rt[Hz]. The OMC TRANS camera is much calmer.
The boost filter also reduced the coherence between the OMC length error signal and DARM. Yesterday, Keita pointed out that the OMC length loop was far, far too coherent with DARM between 1-6 Hz, and this noise was being upconverted around the dither line at 3.3kHz. The upconversion mechanism is not clear; if the OMC length is properly servoed the dither should not generate any RIN (or, very little) and should not upconvert any low-frequency RIN noise to sidebands around the dither frequency. But, that's what's happening. The new boost filter reduces the coherence at low frequency (second plot) and reduces the sidebands at 3.3kHz (third plot), but the coherence at low frequency and at 3.3kHz is not zero (second, fourth plots).
As a precaution against OMC LSC --> DARM coupling, we have reduced the gain of the OMC length loop even further: the OMC LSC boost is now off, and there is a 30Hz rolloff filter (FM3) in the OMC length loop. The UGF is a little more than 10Hz, as described by the green trace from the first figure in this entry.
Keita also recommended changing the dither frequency and changing the dither amplitude and OMC LSC loop gain. We haven't had a chance to do these things tonight. We need to investigate why this noise is being upconverted - note that the amplitude of the 3.3kHz dither line in the OMC RIN increased after the OMC LSC boost was turned off (third plot).
The good ETMY and ITMX bounce mode damping settings with the DARM boost ON are (in M0_DARM_DAMP_V):
The good settings with the boost OFF are:
This switching is handled by the Guardian.
By the way, an ongoing mystery is the source of a 300Hz line in DARM that is very large at the beginning of a lock but steadily decays to the level of the noise floor over a timescale of ~10-20min. The rate of decay is somewhat matched to the BS butterfly mode and the earlier twin peaks noise, but the line is present when we're locked on RF readout, so it doesn't seem to be coming from the OMC. We've checked the IOP inputs that are sampled at 65k and we don't see a line at 16384+/-300Hz that could be aliased, but we should check again at the very start of a lock when the 300Hz line is very prominent.
why use the error signal for bounce mode damping, when the DARM control signal is available?
the phase of G/(1+G) changes very little as the loop gain changes in the case that G>>1 (which should be the case with the Bounce/Roll RG filters on)
Rana,
This is a good point. We are planning to change the pick off point so that it takes the DARM signal from its output as you suggested.
J. Kissel WP #5118 After building the MEDM infrastructure for the calibration lines and optical gain compensation (a.k.a. "gamma"; see ) I realized a large fraction of calibration line path from its generation deep inside the GAMMA_LINE1_DEMOD block to its injection into the DARM loop has no readbacks or test points. Similarly, the calculation of each optical gain compensation coefficient has a ton of steps with out EPICs records to show the progress along the way. As such, I've installed several new EPICs records in the h1calcs.mdl (or, more accurately in the CAL_CS_MASTER.mdl and CAL_GAMMA_CALC_MASTER.mdl library parts) and installed an EPICs record / Test Point pair in the h1omc.mdl (again, more accurately in the omclsc.mdl library part), right before its injected into the DARM loop. See attached screen shots. I've chosen to store the new test point at the full 16 [kHz] in the frames, because I expect we'll want to make comparisons of the CALCS to OMC model delay over long stretches of time in the future to ensure the phase lag between generation and injection remains stable against, say, computer reboots. The new test point channel in the OMC will be ${IFO}:OMC-LSC_CAL_LINE_SUM_DQ where the already existing channel in the CALCS model is ${IFO}:CAL-CS_LINE_SUM_DQ remember that this is *sum* of all calbiration lines. The test point for each individual line, ${IFO}:CAL-CS_GAMMA_LINE${lineNum}_DEMOD_LO is *not* stored in the frames. We expect that the lines will be sufficiently far apart that a simple band-pass on the noise-free digital signal should be adequate to isolate each line if necessary. These new models have been compiled and committed to the userapps repo, and I'll install them tomorrow morning and restart the DAQ.
= Ops Summary =
7:00 - 8:00 Cris and Karen to LVEA
~9:00 Filiberto to LVEA near HAM4 (grab some stuff)
9:26 Karen to Mid Y and Cris to Mid X
11:19 Karen leaving Mid Y
11:50 Kyle to Mid X
12:30 - 13:00 ITMX and ITMY cable swap (Jeff K. Refer to alog17466 )
16:24 Jeff K. to log himself out at CDS highbay
It's been a quiet day today....
Scott L. Ed P. Chris S. Relocated lights and equipment to next section of tube, single door to X-1-7 double doors. Refilled D I water tank and cleaned 15 meters of tube this afternoon. Tube pressures monitored by control room operator during cleaning operations.
Laura, Josh, Andy, Joe, Detchar
We took a look at the 28 Mpc lock on 24th March to see if there was any evidence of arches that we have seen previously at LLO (for example LLO alog 17090).
The first attachement shows a histogram of the rate of glitches (omicron triggers) binned by the value of IMC-F at the same time. The plot looks pretty Gaussian, except for a few bins specifically between IMC-F values of -37.5 to -41.7 and -45.8 and -50. Between these values we see evidence of arches in DARM. The second attachment is a pdf file showing the arches in DARM lining up with values of IMC-F in the second bin. Here are links to many spectograms confirming the arches in the IMC-F bins outlined (spectrograms for IMC-F bin -37.5 to -41.7 / spectrograms for IMC-F bin -45.8 to -50). Each spectrogram time corresponds to a glitch as seen by omicron.
LLO has dramatically reduced the effect of these arches, see LLO alog 17191.
Also the value IMC-F has when these arches appear in DARM seems to drift, i.e. there does not appear to be a specific value of IMC-F which produces the arches (this is evident in the second attachement). This is not what we have seen at LLO, does anyone know what could be causing this?
Andy, Laura, DMac Here's a look at one of the whistles, and the putative cause as an RF beatnote with IMC-F. The idea is that the IMC VCO is beating against another (nearly) fixed RF oscillator, and that's what we see in DARM. In that case, the frequency track should look like the difference between IMC-F (calibrated in kHz) with some fixed value. In fact, the whistle clearly seems to match half of the frequency difference (black trace) rather than the full difference (red trace). Perhaps this hints at the type of coupling? Here, the fixed value is about -46.9 kHz (relative to whatever IMC-F's reference is), but we think this drifts during the lock. We can look in detail at some more of these, and find out.
Why not use the frequency readbacks of the oscillators and VCOs rather than the somewhat arbitrary IMC_F? The radbacks are only once per second, but they can still tell you which units are close in frequency.
I'd be happy to look at the readbacks, as long as they are recorded and available offsite. Can tell me what the channel names are?
I put some notes about these channels in the summary of the ISC F2F last fall. ("RF crosstalk" section)
The channel names and labels are in the attached code. There, I look at the time around the event that Andy sent around to the DetChar mailing list yesterday. It's not totally obvious what VCO is beating against the PSL VCO, the closest one is ALS COMM, but it's about 400kHz away at the time of the glitch.
The first plot shows the value of all the corner station VCOs in the ten minutes around the glitch, the second plot is a zoom around the PSL VCO frequency.
It seems like H1:SYS-TIMING_C_FO_A_PORT_11_SLAVE_CFC_FREQUENCY_5, which Dan has labeled as the PSL readback, is the best candidate. This seems to be roughly equal to (1/2)*IMC-F + 79.225 MHz, delayed by one second (i.e. I can line those two things up and they overlap). Is the one second delay due to some loop or is it just a quirk of the recording system? The whistles occur whenever this readback crosses 79.2 MHz, which is the fixed fiber AOM frequency from the talk referenced in Dan's notes from above.
Time Code Translator work at end stations, WP5116
Dave, Jim:
We racked up the missing TCT in EY, then connected it to one of the unused fiber pairs running between the VEA rack and the computer rack. We verified it was using ports 19 and 20 in the MSR (same as EX). Once connected the TCT synced to the time and the error LED on the MSR sender was extinguished.
We then ran 30 foot 50-Ohm coax, BNC terminated, between the TCT in the computer rack, through the wall, to the first port on the CER comparator. This was done at both EX and EY. The 1PPS port on the back of the TCT is a TNC connector, so we used a TNC-BNC adapter on that port for EX. We need to do the same for EY.
At EX, the comparator is now displaying a time of -22.2uS on the TCT port, roughly the time of flight between the buildings. We will revisit the TCT internal settings, they are most probably what was left from iLIGO.
Rebuild of ASC model
Daniel, Dave:
Daniel though h1asc would benefit from a clean build, install and restart. This was done, I'll see if this helped in the IPC errors.
PCAL filter modules, port from PEM file
Sudarshan, Dave:
Sudarshan discovered that I had forgotten to migrate the PCAL filters from when the CAL system shared a core with the PEM. He copied them from H1PEME[X,Y].txt to H1CALE[X,Y].txt, making a CAL_PCAL to PCAL name change in the process. The filters were then loaded into the PCAL models at both end stations.
The TNC to BNC adapter has been installed to connect the EY TCT to the timing comparator. Remaining work is calibration of the CsIII frequency standard, the time distribution system in the corner stations, and the TCT at the end stations.
Since we are frequently running out of range on the end station VCOs, when there is high wind, we switched the drive of the fiber AOM back to the IMC VCO. This was the original scheme. It has the disadvantage that the end station green locking now depends on the IMC to be locked. On the other hand, this is probably not a big deal, when working with the fully locked interferometer.
Kiwamu, Sheila,
We found that we had to lock ALS DIFF and COMM sequentially with this change, instead of in parallel as we have been doing. This morning we found that we could not transition the tidal to common with this change, so we reverted it so we are again using the fixed frequency RF source for the fiber AOM. We labeled both of the cables as well, so that next time the wind picks up we can put this back. We will probably not need high bandwidth common tidal with this modification, so we can try turning this bandwidth down and leaving off the IMC F offset, then later engaging it with a slow ramp.