model restarts logged for Tue 25/Nov/2014
2014_11_25 04:27 h1fw1
2014_11_25 11:03 h1pslfss
2014_11_25 11:03 h1psliss
2014_11_25 11:03 h1pslpmc
2014_11_25 11:06 h1broadcast0
2014_11_25 11:06 h1dc0
2014_11_25 11:06 h1fw0
2014_11_25 11:06 h1fw1
2014_11_25 11:06 h1nds0
2014_11_25 11:06 h1nds1
2014_11_25 11:07 h1nds1
2014_11_25 12:04 h1broadcast0
2014_11_25 12:04 h1dc0
2014_11_25 12:04 h1fw0
2014_11_25 12:04 h1fw1
2014_11_25 12:04 h1nds0
2014_11_25 12:04 h1nds1
2014_11_25 12:45 h1broadcast0
2014_11_25 12:45 h1dc0
2014_11_25 12:45 h1fw0
2014_11_25 12:45 h1fw1
2014_11_25 12:45 h1nds0
2014_11_25 12:45 h1nds1
2014_11_25 12:46 h1nds1
2014_11_25 19:33 h1fw1
two unexpected restarts. Maintenance day. PSL DAQ changes. DAQ restarts for PSL, DMT and Beckhoff.
conlog daily frequently changing channels report attached.
Evan, Kiwamu, Evans (Matt), Stefan - After finding a new alignment this afternoon, we realigned the DIFF beat note. - We halfway succeeded in scripting the CARM TR transition. The setup is now all done in the state "CARM_ON_TR". - The actual transition requires an ezcaservo (to suppress the voltage difference on the CM board while ramping). We struggled to implement it in Guardian. For now the following lines are required: ezcaservo -r H1:LSC-REFLBIAS_OUTPUT -g 0.000003 -f 1 H1:ALS-C_COMM_VCO_TUNEOFS -t 120 & sleep 5 ezcastep H1:LSC-REFL_SERVO_IN2GAIN -s '0.25' '+-1,28' ezcawrite H1:LSC-REFL_SERVO_IN2EN 0 ezcaswitch H1:LSC-REFLBIAS FM5 FM8 ON The last step engages a boost and a lead filter. Attached below is a loop transfer function of the CARM loop at sqrt(TRX+TRY)=1.5. - After that we moved the arm powers to sqrt(TRX+TRY)=1.5 (about 200nm). There we handed off to H1:LSC-ASAIR_A_RF45_Q. - This took a input matrix gain of 4e-7. We saw an oscillation kick in at 20 Hz, se next we will try at 2e-7.
Maybe the obvious question: Why are we not using the VCO tune servo which is already built into the VCO interface?
In light of yesterday's realization that POPAIR_B was saturating, and that DRMI locks much faster with a lower input power, here is a time series of the demodulated REFLAIR_A error signals during a DRMI lock acquisition attempt at 4 W.
The ADCs have a range of ± 215 counts (≈ 33 000 counts). The largest excursions shown are something like 17 000 counts on RF9I, so that means at 10 W we would instead expect excursions of ≈ 43 000 counts, which means we'd be saturating.
The PRCL trigger is not shown in the attached plot, because PRCL is always on during locking.
J. Kissel, M. Evans, D. Hoak, E. Hall, A. Staley As Dan describes briefly in LHO aLOG 15297, we want to think about how to implement ALS COMM, ALS DIFF, DARM, and CARM, very-low-frequency, tidal offloading global control to the ETM HEPIs in the front end. Currently we're using a guardian-run servo over EPICs, as described in LHO aLOG 15244. In this log, I describe our thoughts on how to do for the long term. After scouring the LSC model, we've found that ETM HEPI models are currently receiving the same LSC signals that the ETM QUADs are receiving. This is left over from when we were trying to attempt the distributed hierarchy. However, now that we've moved to an offloaded hierarchy for length control, we don't want to have to re-create the integrators that exist up the QUAD's signal chain, hope we get it right (and or the computer), and integrate once more to get the offloaded control signal we want. The REFL_SLOW path -- the digitized, slow output of the CARM Common Mode Board which takes the ALS COMM signal as its input -- was initially tossed up as an idea for the digital, very-low-frequency, COMM correction signal because it's currently hooked up through to the LSC CARM bank, but for the reasons described above, it's not exactly what we want for the long term. After discussing the options, we've decided that we need an additional path from the LSC model to each ETM HEPI AND we want to restore the OFFLOAD path from the TOP of the QUAD (straight) to HEPI. Why? - Currently, the digital IMC length control (the natural signal for controlling tidal drift) is only fed directly to MC2, and does not pass through the CARM or DARM banks. - We don't want to offload to, or directly control use the TOP mass of the QUADs because that stage of the QUADs have proven to have particularly bad length-to-angle coupling such that large longitudinal control signals just pitch the test mass over destroying the lock just as much as tidal forces would by themselves by reaching the range of the various VCOs. - For CARM, one might ask - why not offload to MC2's HEPI? - can't because PR2 is also in HAM3, and the low frequency displacements to the PRCL aren't the same as what's needed for the red laser frequency. Further, a nuance of how the green end station lasers are tied to the PSL -- with the *transmission* of the reference cavity, which doesn't change when the IMC to PSL VCO to REFCAV REFL servo is changed -- while the COMM / CARM feedback to the PSL via the IMC changes the *red* frequency to match the common motion of the arms, it *doesn't* change the *green* arm laser frequencies in common with the arms. - For DARM feedback, which will be sent through the suspensions because we need the bandwidth, will be offloaded on a few stages on its way up to HEPI. For CARM, we send most of the control through the MC2 SUS all the way to its TOP (because its a much better behaved SUS) and only the very low frequency would need to be send to the ends. So CARM and DARM need inherently different paths. - For DARM, one might ask - why offload straight to HEPI, why not ST2 of the ISI and up-through as we've snaked through the QUAD? - because, currently, the best configuration is not to use ST2 of the ISIs. The offload path would require the isolation loops to be ON in order to function. Given that the ISI configuration is still, and will probably be, under flux, let's just go straight to HEPI which has happily run with position loops only for quite some time, and we've found that it runs best that way. Given that we're still unsure what will work, we propose to create an additional, mini output matrix in the LSC model that takes in DARM, CARM, and IMCL, and spits out control to ETMX and ETMY HEPI. How much extra infrastructure is needed? - A new mini-matrix as described above, and the wires to make the connections from DARM, CARM, and IMCL to it and out to the top level. - Two new RFM IPCs between the LSC model and the two end station HEPI models. - Restore the M0 offload filter path and EUL2CART matrix that was removed in ECR E1300578, for the Longitudinal DOF only. - Two new end-station dolphin IPC connections between the QUAD models and the HEPI models. - Two new filter banks in the HEPI model (or at least after the output matrix, it could be in the LSC model) to match the frequency response of the direct CARM offloaded signal from LSC to the indirect DARM offloaded signals from SUS. - A summing node at the top level of HEPI. How painful will this change be? How much will it affect library parts? - The LHO LSC model is currently still hooked up to a library, but I'm not sure whether LLO is using it. So, though not under ECR control, a discussion might be nice before a change is made. - Because we can only imagine doing such direct LSC global feedback to the ETM HEPIs, we can add the new filters and summing node to the top level of the ETM HEPI models, and send signals into the common HEPI library part as is without modifying it (or requiring an ECR). - Even though we've removed the OFFLOAD path from all SUS, because we had wanted to re-create the total length control signal on the QUADs eventually (like is done for IMC-X), the longitudinal control signals are still passed to the top of the library part for the QUADs (thank god). Thus, only a top-level change is needed for the QUADs as well -- the EUL2CART matrix and IPC parts can all be added to the top-level of the ETMs only. Because we're only trying to change longitudinal, we shouldn't even need the full EUL2CART matrix (as originally described in T1100617) So in summary, not that bad. BUT -- we'll probably still wait until next week to get started with the changes given the limited staffing over the Turkey-day holiday. Thoughts and comments welcome, ESPECIALLY those that come in BEFORE we make the changes.
Maybe we should put together a design doc first. There are a couple of obvious questions:
A couple notes: 1) How is the IPC getting from LSC front-end to SEI end-station front-end? To do this same scheme at both sites, you will need to send it over the RFM from LSC to end-station ISC or SUS, and from there to the end-station SEI. This is a very slow loop, so I think you can adjust for the extra cycle delay. LLO's end-station SEI do not have GE FANUC RFM cards, so don't use them. 2) The LSC model is currently at the limit in getting RFM to the end-stations (and regularly fails to send properly). There is currently no real-time cycle overhead for an additional RFM broadcast (or two as it has to go to both ends). CDS has not been able to qualify newer, faster hardware that works without timing glitches.
Keita, Stefan Yesterday we noticed that our alignment put the spot way low on the ETMY. So today we tried to set the QPD offsets such that the TMSY well centres its green beam on ETMY. This allows us to use a co-aligned arm as good reference. 1) First we fine-tuned the PZT output matrix to well separate the POS and ANG DoF. We did that by dithering the desired DoF, and watching the camera shots in near and far field (ETMY scatter and TR_Y in the corner). Result: H1:ALS-Y_OUTPIT_MTRX: POS ANG 1 -1.2 PZT1 -0.12 1 PZT2 H1:ALS-Y_OUTYAW_MTRX: POS ANG 1 -1.25 PZT1 -0.1 1 PZT2 2) Next we moved the POS DoF until we were happy with the ETMY camera image. We x-checked that by watching the green reflected powers (ITMY was misaligned). We used this to set the QPD offsets: H1:ALS-Y_QPD_A_PIT_OFFSET 0 H1:ALS-Y_QPD_B_PIT_OFFSET 0 H1:ALS-Y_QPD_A_YAW_OFFSET 0.37 H1:ALS-Y_QPD_B_YAW_OFFSET-0.12 3) We then manually aligned the arm optics to this new beam position. We got an arm build-up of up to 1.09 (H1:ALS-Y_TR_A_LF_OUTPUT), about 10% more than before. Next we need to do the co-alignment. 4) A quick attempt to use the green WFS for that wasn't too successful. 5) We did a manual co-alignment of IR and green. We only got the green TR_Y to about 0.8, but that should be good enough for now. Kiwamu re-centered the DIFF beat note. Attached are snapshots of the ITMY camera and all the alignment values. Camera target values: X-center 336.8 Y-center 238.5
There is now a new filter in the H1:SUS-ETMX_L2_OLDAMP_P filter bank which is supposed to help avoid exciting the bounce mode when the OpLevs are on (in FM3 called Phase9.8).
We had previously used this filter module (sending OL PIT to L2) as a means of damping the bounce mode excitation. Now, we are hoping to prevent excitation by setting the phase of the usual OL filter to match that of the damping filter at the bonce mode frequency (roughly +90dg, see attached plot).
Finally got rid of the tired database and now have all the new useful channels. Take a look at the new pump controller medm screen. There are still a number of cables to run but all the ports available are at least in use now and database junk has been discarded. We still need to modify the board to power the amplifier for the remote pressure sensors at BSC2; we'll try to knock that out next week.
Attached is a shot of the new controller screen.
06:30 Cris into LVEA
08:00 LVEA is LASER HAZARD
08:00 OIB set to "Commissioning"
08:01 restarted Alarm Handlers
08:02 Restarted DTT session on projector 1
08:03 Restarted "hung-up" Operator2 workstation
08:25 port-o-potties on site for exchange
08:30 Corey out to LVEA
08:35 ISS diffracted power adjusted to ~7.6%
08:39 Aaron to EY for PEM work
08:40 HEPI HAM4 L4C watchdog accumulator reset - sat count 380
08:43 Keita and Daniel to EX
09:00 I'm running PSL DBB reports
09:06 Cris to EX
09:19 Travis into LVEA
09:25 Hugh to mazzanine to put HEPI pumpd in manual control. He'll be staying there and adjustin them manually
09:31 Karen into LVEA
09:45 Kyle into LVEA
09:52 Tour group into control room
10:37 Cris back
10:55 Karen to EX
11:04 Dave restarted FSS, ISS and PMC models to accomodate change in acq rate.
11:04 Dave restarted DAQ
11:13 Water delivery
11:34 Sudarshan into LVEA
12:02 Karen leaving EY
12:06 Corey out of LVEA
12:11 Sigg and company back from EX
15:20 Gerardo into H2 LASER enclosure
Hanford hauling indicates that they will not be running trucks on day shift or swing shift on Weds, Thurs or Fri of this week, nor over the weekend.
The button is intermittent because h1lsc is running at RCG2.8.3. The fix for this will be implemented when we upgrade to RCG2.9. In the meantime scripts should check if the button press was accepted, and try again if not.
Sudarsan, Peter, Dave: PSL model changes
The h1psliss, h1pslfss and h1pslpmc models were modified to downsample all 32kHz channels to 16kHz. This resulted in 10 32kHz channels changing to 16kHz with the distribution: ISS=1, FSS=4, PMC=5. The models were restarted and the DAQ was restarted. This resulted in a science frame size reduction of about 20MB per 64 second science frame (2%). The restarted models were burt restored to 9am PST today.
Dave: DMT broadcaster
Following last week's ISS model change, the H1BROADCAST0.ini file was changed to replace the obsolete H1:PSL-ISS_PD[A,B]_OUT_DQ with the calibrated H1:PSL-ISS_PD[A,B]_REL_OUT_DQ. Also the obsolete H1:LSC-OMC_DC_OUT_DQ was removed from the DMT.
Hugh, Dave: HEPI pump controller channels
Following Hugh's software change on the corner station HEPI pump controller EPICS system, a new EDCU INI file was added to the DAQ. Hugh has updated the MEDM screens and is updating the autoBurt.req file.
Dave: Restart of H1SLOW-F1FE EPICS gateway
Following the restart of h1ecatx1 this morning, subsequent restarts of the DAQ EDCU did not establish new connections to this IOC. I restarted the EPICS gateway between the EDCU and the Beckhoff IOCs which prompted the EDCU to re-establish a connection. It took several minutes for the reconnection to complete. Initially the number of disconnected channels went from 1,951 to 81, then took a further minute to reduce to zero.
Dave: DAQ restarts
To support the above work the DAQ was restarted three times this morning. On restart 1 and 3 the data concentrator was very slow on restarting (several minutes) while restart 2 proceded at the usual pace. While waiting for the slow restarts I had time to check for duplicate channels (none) and any errors in log files (none).
Jim, Dave: replace broken thunderbolt monitor at operator station
I collected the repaired Apple thunderbolt monitor from Alpha Computers this morning, Jim installed it at the operator station. We are looking into a tether cable solution for strain relief.
Daniel, Keita, Elli
This morning we worked on the ALS at Ex:
-We added autoalignment PZTs to the green WFS path. Mirrors ALS-M16 and ALS-M19 were taken out and and relaced with PZT mirrors. After adding the PZTs we realigned the green beam back onto the WFS.
-We removed two mirrors blocking the beam path upstream of ALS-M19. The mirrors weren't on the table layout and didn't appear to be doing anything.
-Swapped feedthorugh panels D1100434 on ISCTEX with replacement panels. The original panels had the wrong size screw spacing.
Two IR HR removed
There were two IR HR mirrors on the table that were not necessary any more, so I removed them and brought them back to the SQZ area cabinet.
Bug on the ISCTEX
Look what we've found there. I think the moth was attracted by the laser or maybe an LED inside, managed to crawl in from an opening under the white tape that covers a hole on the feedthrough panel, but couldn't find his/her way out and died.
I also found the same kind of moth in a bag of clean wipes. The entire bag went into the trash bin.
Aaron and I continued with PEM cabling at EY this morning. We have completed all long cables runs and started installing the permanent power supplies in the DC rack. We disconnected the power supplies powering the microphones in both the VEA and EBAY and placed them in the DC rack. Got a call from commissioners asking us to leave EY, before we could reconnect microphone power with permanent power supplies. Will go back tomorrow morning and power them up.
Microphone power was reconnected at EY this morning.
Matt, Dan
After Alexa gave us a quick alignment tutorial, we started by fiddling with the dither alignment of the X-arm. The PD thresholds for dithering across the baffle PDs were set to very small values, so small that the dither would declare victory before it even started. Matt has been working on the initial alignment, I'll let him say more about it.
We noticed that the green spot on the TMS-Y mirror, visible through ETMY, is very low. There are large offsets in the EY green QPDs that steer the beam to what looks like a bad place on the mirror. Zeroing these offsets (which is where they were for most of the last six months) would have forced us to begin a painful realignment procedure, so we left them as we found them. Historically it looks like people have set these values to maximize the buildup in the arms, but it's hard to believe that the optimal beam position is so far off-center.
With the arms well-aligned for green we aligned the corner optics - IM4, PR2, PRM, BS, SRM. The scripts for this went quickly, about five minutes.
Then we locked the DRMI, with the arms misaligned. Locking tonight was fast and robust; the DRMI would lock within a few minutes (rarely more than 5 minutes), and would remain locked for tens of minutes. There were many unexplained lock losses, so things are not as stable as we'd like, but we didn't see any mode hopping and the sideband buildup was consistently good. Over three hours of DRMI locking we didn't observe a significant drift in the alignment.
We went through the DRMI guardian and commented out lots of gain/switch settings that weren't needed, mostly for the DC centering. The script is now pretty fast, it goes from lock acquisition to ASC loops on in less than 30 seconds. We did not observe any unexpected long pauses. The duration of the script is dominated by ramping the SRM and PRM M2 gains, this takes 9 seconds total (4.5sec each).
We re-enabled the PRM and SRM2 offloading to M2. We found the SRM offloading was a little flaky and sometimes broke the lock. To make this a little gentler, we reduced the initial gain ramp to 1, and ramped the rest of the way (to 4.5) after the WFS turned on.
We measured the LSC OLTFs and found that MICH and PRCL didn't have much phase margin (~25deg each). We reduced the gain for these loops so that they both have >30deg. The MICH gain is now 3 (was 5) and PRCL is 15 (was 22).
Afterwards I mucked around with the OMC, I found a set of good misaligned values for ITMX and ITMY that minimized the RIN at the dark port when the other ITM was aligned (presumably this steers ghost beams away from the OMC). Following Gabriele's instructions I used the picos to center the beam on IMC WFS A and added some offsets to the IMC ASC loops; this reduced the intensity noise measured by the ISS second loops PDs by a factor of three or so. I tried to engage the ISS second loop but failed. The script posted here did not appear to work; I tried to follow these directions but the loop was unstable.
Plots attached are MICH, PRCL, SRCL loops before we reduced the gain to get some phase back in MICH and PRCL.
FWIW the misalignment settings for ITMX/Y that inject the least amount of RIN at the dark port (with a single bounce off the other ITM) are:
ITMX PIT | ITMX YAW | ITMY PIT | ITMY YAW | |
Aligned | 64 | -1.7 | 222 | -140 |
Nominal misaligned | 0 | -62 | 0 | 0 |
Better misaligned | -10 | -380 | 282 | -450 |
I didn't observe any change in the RIN spectrum when moving the PRM, SRM, or ETMs. This doesn't solve the problem - the OMC TRANS spectrum is still a little ratty - but it seems to be dominated by RIN noise that is coming from the input beam, which is a job for the ISS.
The ISS second loop scripts are not 100% fail proof. If they fail once, try a couple of times more. Normally the switch one script works more than 50% of the times.
Well, Gabriele is right -- running the ISS secondloop script with Sudrashan this afternoon was problem-free.
For posterity, I've been using Sheila's DRMI OLTF templates for measuring loop gains: /ligo/home/sheila.dwyer/DRMI/DRMI_MICH_OLG.xml and related.
Relieving the arm VCO signals doesn’t work once the COMM and DIFF loops are engaged. (The DIFF loop will fight against the HEPI feedback for example.) Therefore I made a similar HEPI offload servo for the COMM loop.
I use MC_F as the error signal and feed back to the common end station HEPIs.
guardian code:
class HEPI_RELIEF(GuardState):
@fault_checker
def main(self):
servo_gain = -.3
offset_ramp_time = 2
offset_chan_1 = 'HPI-ETMX_ISCINF_LONG'
offset_chan_2 = 'HPI-ETMY_ISCINF_LONG'
readback_chan = 'IMC-F_OUTPUT'
ezca.write(offset_chan_1+'_TRAMP',offset_ramp_time)
ezca.write(offset_chan_2+'_TRAMP',offset_ramp_time)
self.hepi_servo = cdu.Servo(ezca,offset_chan_1+'_OFFSET',readback=readback_chan,gain=servo_gain,control2=offset_chan_2+'_OFFSET',gain2=1);
@fault_checker
def run(self):
self.hepi_servo.step()
return True
attached file shows servo with wrong sign, then sign flipped. Then an unrelated lockloss.
So right now when just the arms are locked on ALS, they each feed to their own hepi. At some point the ALS_COMM guardian turns on the HEPI feedback and the arms turn off their feedback.
One last thing to mention: These offsets will walk away over time and should be zeroed periodically. It should probably be written into the DOWN state of one of the guardians.
Jeff, Dan, Alexa, Evan, Matt
To illustrate the need-for / effect-of the feedback to HEPI from ALS_COMM, I attach two plots.
The first shows the start of an IR lock in the arms. The ALS_DIFF path is turned on about 1/3rd of the way through. Once the differential degree of freedom is out of the arms, the tidal drift begins to dominate the ALS green control signals, and the VCOs run away. The rail at 7V in end-X is where we lose the lock.
The IMC control signals
In the second image, I plot a series of locks around the time when Nic implemented his ezca servo code. After some missteps for gain tuning the offloading of the COMM signal to HEPIkeeps the VCO in range.
Nic's code uses MC_F, but for a permanent digital implementation we think that MC_L is a better choice. The current plan is to add a PCIE channel to the LSC front-end module to send MC_L (the input of the LSC-MC filter) bank to the endstation HEPIs.
A set of scripts was created last week to allow commissioners to switch SEI platforms to SEI’s preferred testing configuration at the end of their shift.
The scripts are very simple for now:
Those new scripts were put in the svn so they can be remotely adjusted, and improved (made more generic, make classes/methods, etc…):
/ligo/svncommon/SeiSVN/seismic/BSC-ISI/H1/Common/Misc/Test_Configuration_Scripts/
To run those scripts:
$ cd /ligo/svncommon/SeiSVN/
$ python Switch_To_Test.py (at the end of the commissioning shift)
$ python Switch_To_Comm.py (when commissioning resumes)
Kiwamu used those scripts at the end of his shift on 10/28/14. JimW and I added the senscor switching capability on 10/29/14, which does not change the way to run the scripts.
These scripts are now executable. They are run by navigating to the folder and typing ./Switch_To_Comm.py or ./Switch_To_Test.py . They currently control HEPI Z and ISI X sensor correction for the X arm BSC's.