We turned the ETMY ring heater off (.55 W --> 0 W top and .55 W --> 0 W bottom) at 21:00 UTC.
Change was made since we seem unable to damp ETMY Mode 27 (18041 Hz).
Keita Daniel
We were able to engage the outer loop ISS after the mode cleaner was locked and power up to 50 W in full lock. The third loop was intended to be engaged around 10 W, but due to a scaling error in the arm powers, the gain was too low. We were able to adjust the gain, when we reached 40 W. And then power up to 50 W.
How to engage the outer loop:
How to engage the third loop:
This is now coded on the guardian. However, the arm power scaling error needs to be fixed with an LSC model reboot.
New features in the digital ISS model:
We did not test turning on the boost or turning off the AC coupling of the outer loop. Changing the ISS main gain is problematic due to electronics offsets. The same goes for the boost stage.
OLTF of the third loop at 40W, with the outer loop engaged with AC coupling (first attachment, red and blue).
Some notes about power scaling:
I increased the heat in the LVEA yet again by 1ma (to 6ma) on heaters 1A,1B,4C & 5C . We also now have electric as opposed to pneumatic actuators on AHU 1 & 2 face/bypass dampers which serve the LVEA.
J. Kissel, K. Izumi Summary -- PCALY optical follower servo (OFS) has been saturated since the Saturday, Sep 10th 2016 power outage (see LHO aLOG 29592). Recovery required a quick OFF/ON cycle of the OFS loop from the MEDM overview, which is now complete. All lines are now confirmed back in the spectrum. The story: Kiwamu noticed that the 331 Hz calibration line was missing from the wall spectrum once we powered up/reduced the noise enough to see it at the expected amplitude. We looked into the MEDM control screen and found the excitations still happily going out to the DAC. We checked and confirmed that the 1083 Hz line was also missing in the DARM spectrum, implying something was wrong in analog land. Finally, I noticed that the OFS error signal, normally displayed on the PCAL overview (H1:CAL-PCALY_OFS_PD_OUTMON) looked abnormally low and quiet. After trending for the past 6 days, seeing the sudden change of behavior at 19:30:48 UTC (12:30p PDT) on Sep 10th. and a little bit of aLOG scouring, I put two-and-two together that the power outage caused the failure. Knowing that turning the OFS OFF then ON is a common "guess-and-check" method of fixing problems with PCAL, I performed this time honored tradition. Normal PCAL functionality was restored immediately after re-closing the loop. Easy fix! Perhaps we need a PCAL line monitor like we have for bounce, roll, and violin modes, so we can have DIAG_MAIN set up an alarm once we're at sufficient DARM noise levels.
I have reset the HEPI L4C watchdog counters for HAM4 and ITMX. See attached screenshot from before reset.
This closes FAMIS request 7071.
Andy, Josh We have found RF beatnotes in the lock from Sep 12 (alog 29606), from 100 Hz up to Nyquist in DARM. We have only been able to find them in DARM so far, not in any of the usual auxiliary channels that typically see them stronger (POP/ISS/REFL). These are a beatnote of the PSL VCO frequency with an apparently fixed line at 79131700 Hz. TJ points out that because the AOM is double-passed, it would be good to check for something at double that frequency as well. As a reminder, we find the beatnote frequency by doing a linear fit of IMC-F to the PSL VCO readback (which has the correct absolute frequency but is only sampled once a second). Then we plot abs(f_VCO - f_offset) over DARM and choose the offset to make them line up. In the past, we've seen various multiples of this frequency show up (i.e. harmonics like the second or fifth). This time there's a multiplier of one. Attached are two of these fits. Note that we sometimes see a mirror of the line around something like 4800 to 5000 Hz - maybe it's the OMC dither?
The subsystems all reported no problems or issues after yesterday’s maintenance window. CDS continues working on the card reader access installation. Commissioners are working on the ISS Second Loop upgrade. CDS will assist as needed. Peter King reported a small leak (perhaps <1 drop per day) in the manifold. The PSL team is working on an upgrade of the manifold to eliminate these leaks. Kyle needs time at End-X to bake out the new RGA. Ops will notify Kyle when conditions allow End Station access. FMC installed an upgrade to the HVAC damper controls. Bubba reports a noticeable improvement in system function. There will be an all hands safety meeting at 15:00 in the LSB Auditorium. At the meeting Chandra will give an overview of View Ports.
There was an issue with the GDS calibraiton filters uploaded yesterday (see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29651). These filters were too long and it was found that over the period of ~hours the GDS calibration pipeline would fall behind real time by ~minutes. I have re-made these filters and significantly shortened the length of the correction filter for DELTAL_RESIDUAL in particular. Attached are updated plots of the filters and a spectrum comparison for some ER9 data (115207641601152076544) between the output of the GDS calibration pipeline with the new filters and the output of the front-end CALCS model. The filters are checked into the calibration SVN under:
aligocalibration/trunk/Runs/PreER10/GDSFilters/H1GDS_1157896856.npz
These filters were made with calibration SVN version #3301.
The GDS calibration pipeline was restarted on the LHO DMT machines to use these new filters at 1157908016.
| Work Permit | Date | Description | alog |
| 6157.html | 2016-09-13 09:28 | Activity: Re-flange RGA using proper vacuum fasteners, i.e. remove and replace factory supplied fasteners, on RGA (if still present) - will require venting the RGA volume while isolated from the X-end via one closed 2 1/2" metal valve*remove redundant O-ring valve which is in-series with the 1 1/2" metal pump port valve and install a turbo pump in its place*Pump RGA volume with leak detector and helium leak check RGA and turbo fittings*Wrap RGA components with aluminum foil and heat tapes so as to be prepared to Bake out RGA (to be done later under separate WP)*Shut down pumps after leak testing is complete | 29665 |
| 6156.html | 2016-09-13 08:42 | Activity: Facilitate Safety tour and Tuesday maintenance Lasers on Tables and Ports closed. | |
| 6155.html | 2016-09-13 07:42 | Activity: New outer loop model w/ sum, norm and RIN modules | 29654 |
| 6154.html | 2016-09-12 14:49 | Activity: Update user accounts on h1brsex and h1brsey to match each other. | 29663 |
| 6153.html | 2016-09-12 14:42 | Activity: Update h0vaclx to add X1 PT140 BPG402 gauge. Requires DAQ restart. | 29657 |
| 6152.html | 2016-09-12 14:22 | Activity: Add Stage 2 SEI Configuration nodes for all of the BSCs. Stage 1 nodes already exist. I will create one node to start and cycle through the states to make sure that all is well before creating the others. | 29655 |
| 6151.html | 2016-09-12 11:54 | Activity: GC core router will be rebooted and have transceivers swapped to upgrade from 1Gb to 10Gb connection with ESnet. Wireless and Internet connectivity will be disrupted during the short outage. CDS and LDAS internal networks will be unaffected, along with the phone system. | |
| 6150.html | 2016-09-12 08:25 | Activity: Safety Audit Committee to tour the LVEA and End Stations. | |
| 6149.html | 2016-09-09 13:43 | Activity: model update (mode RMS monitor point, add offset to PLL output gain adjustment) will require DAQ restart | 29629 |
| 6148.html | 2016-09-09 13:32 | Activity: Create an HWS SDF to report any setting errors with any of the hws epics systems (h1hws[ex,ey,msr]). Will require DAQ restart to trend the SDF statistics. A cleanup of the hws autoburt.req files is also needed. | |
| 6147.html | 2016-09-09 13:20 | Activity: Forklift backup PSL chiller (Wallace) from OSB Receiving to Mechanical Building for testing. | 29566 |
| 6146.html | 2016-09-09 10:31 | Activity: Update the front-end model for the outer loop ISS, fix inverted BIO, add filters for PD diodes to account for whitening | 29570 |
| 6145.html | 2016-09-08 16:06 | Activity: Move backup chiller from Mechanical Building to Chiller room. Will go through the LVEA. | |
| 6144.html | 2016-09-08 15:50 | Activity: Drive down to EX and come back while seismic level is high and IFO locking is difficult. No entry to the building. This is for KAGRA guests. | |
| 6143.html | 2016-09-08 10:36 | Activity: Give two KAGRA scientists a quick LVEA tour while initial alignment is done. | |
| 6142.html | 2016-09-08 08:28 | Activity: Reinstall ISS Servo chassis in PSL-R1. Cabling for Photodiodes 5-8 at HAM2 will also be reconnected to new chassis. | 29570 |
| 6141.html | 2016-09-07 10:54 | Activity: Power down h1fw0 and h1fw1 to document BIOS settings. Shutdowns will be staggered to prevent data loss. | |
| 6140.html | 2016-09-07 09:31 | Activity: Offload ITMY HEPI Position Bias to Load Springs: DeIsolate HEPI, Adjust DSCW Springs to bring free position to servo'd spot. Reisolate platform. | 29671 |
We now see ETMY PI 18059 Hz (aliased down from 47477 Hz) MODE 28 ring up. It's been successfully damped several locks now with gain +10000, phase +90. These settings have been added to the guardian and SDF.
I believe this is the same pringle mechanical mode as the 18056 Hz which was originally seen back in July; there's no peak at 18056 Hz now and I was able to ring and damp this by driving ETMY ESD equally in opposite quadrants.
We've also been battling MODE 27 ETMY 18041 Hz (aliased down from 47495 Hz) the past two nights. Wonderfully, Slawomir just posted a study on these modes earlier this evening. Indeed, we do see both modes ringing up essentially simulataneously (attached: spectrum of three snapshots during the ring up last night. 18041 was being actively damped and 18059 had no damping).
So far we've been unsuccessful at damping MODE 27. MODE 27 ring up very rapidly but for the past few months we've been quickly passing over it during the transient before it had a chance to ring up very much; the recent ring heater changes have shifted the optical mode so that's no longer the case. We had three tries to damp tonight, once with the usual BP + Damping filter set-up, once with PLL, once trying both. All were unsuccessful and we lost lock due to this PI. During this last lock, MODE 27 grew just enough that the drive started to saturate, consequently ringing up MODE 28. I was able to damp the latter by decreasing the gain of the former (and thus not saturating). During this time I tried PLL but it couldn't turn it around. After both seemed damped, MODE 27 rapidly rang up and broke lock. We need to find a smart way to handle stepping the gain to avoid saturation while balancing the need for strong drive. The PLL can do this (we can set a fixed number of counts out so as to avoid saturation) but the PLL doesn't seem to lock onto the modes until too late in the game. Perhaps we consider sending the OMC signal to the PLL for a try?
The fast shutter test that checks that the shutter fires by look at the GS 13s at the time of a lockloss where we had more than 14kW gave us one false alarm tonight.
(It correctly reported that the shutter did not fire in a lockloss where the circulating power dropped). Not enough of the power was sent to the AS port for H1:SYS-MOTION_C_SHUTTER_G_TRIGGER_VOLTS to get close to its threshold of 2 Volts. (screenshot attached)
We probably want to increase this threshold to avoid false alarms so people don't get in the habit of ignoring the warning. For now I have increased it to 25kW, and committed to the SVN.
Lately we have been aligning DRMI by hand before letting the ASC come on because the ASC engagement has been rough and caused many locklosses.
Tonight I watched WFS signals while Corey alinged by hand (and the BS drifted in pitch while cooling off from a 50W lock). It looked like AS45Q was a better signal for the BS than the 36 WFS, with much less sensitivity to other optics. Hoping that this would help DRMI ASC acquire more smoothly, I tried it out by eyeballing the relative gain. (where ASB36Q using a 2 in the input matrix, we used a -100 for AS45Q).
This seems to be working quite well, we have acquired lock with bad alingments about 3 times durring these earthquakes, and let the gaurdian handle the ASC without aligning by hand.
We need to switch back to AS36 before we do the CARM offset reduction.
I've moved the DRMI ASC offloading, fast shutter testing, and waiting for BS ISI out of the CARM on TR guardian state (this state was getting crowded with things that are not related to CARM.) There is now a new state called DRMI_ASC_OFFLOAD that switches MICH to AS36, waits for the ASC to converge, tests the fast shutter if needed, and waits for the BS ISI to isolate stage2. This happens before CARM on TR.
The BS was using AS36BQ durring DRMI locks, and switching to AS36AQ before increasing power. The build ups are better in DRMI at the zero crossing of AS36AQ, so we are now switching from ASB45Q to ASA36Q after DRMI ASC has convereged, before starting the CARM offset reduction. We no longer need to change anything about the MICH ASC before we increase power, we are using the high power matrix from the begining.
Keita, Sheila, Terra
We have had 3 locklosses today caused by saturations of the ISS 1st loop. There are 2 main messages from this alog:
Earlier today, Keita and I had a look at the logic in the ISS autolocker. It seems that it watches the channel H1:PSL-ISS_TRANSFER1_B_OUTPUT, if this is above H1:PSL-ISS_SAT_THRES (currently set to 5 V) for more than 10 seconds, it opens the first loop. In the third attached screnshot you can see this in action right before one of our locklosses. At just after -14 seconds, TRANSFER1B (bottom row) becomes large and the AOM DRIVER monitor (top right) flatlines. The PD sums become slightly more noisy for 10 seconds, after which the ISS is opened by the autolocker. This causes the PD sums to become much noisier, and the hugh glitch in power visible in IM4 TRANS. Is there any gentler way to deal with a saturation?
Corey, Sheila
Tonight we have very low ground motion (useism band 10^-1 um/second earthquake band 10^-2um/second)
We have been using awggui to add an 8000 count sine wave at 0.1 Hz to the BS M1 LOCK L filter bank (which has a gain of 0.1). This is helping us to catch fringes more often tonight, and we are manually ramping it off once DRMI locks. This would be a good idea for operators who find themselves havign trouble locking DRMI in low ground motion.
To do this, use a terminal and type awggui. Set it up like the attached screenshot. It is a good idea to start with the gain (bottom middle) set to 0, hit set/run, then ramp the up to 8000 by hitting set in the gain box.
How does one manually ramp off this excitation? I set gain from 8000 down to 0 & the ramp time from 1 to 10 and then hit the Set button, but I wasn't sure if this did anything (the excitation was still seen on the CDS Overview for H1SUSBS).
I couldn't find instructions on killing/clearing testpoints in the wiki, but luckily Kiwamu was here and he knew the command:
> tpclear 31 *
This cleared the excitation.
Sheila let me know that I could have taken care of the excitation by taking the gain from 8000 to 0 and then EXIT-ing the awggui session. Will try this next time.
(Sheila, Terra, Jenne, Matt, Lisa)
Locking Notes:
Had some grief with H1 locking.
Before the issues above, managed to get some high power locks to Terra for PI work.
SEI_CONFIG State Change Due To EQ
Additionally, while attempting to lock we had a 6.0 EQ in Colombia which peaked at 1um/s. I was a few minutes slow, but took the SEI_CONFIG to EARTH_QUAKE_V2 before the noise got to 1um/s.
Facilities Note: Noisy Air Conditioning Unit (one closest to entrance) was making a bit of noise. Bubba was contacted, and this should be addressed tomorrow. (he said we could turn it off and open the hallway doors if it was too noisy....but we never got to that point.).
We are having another EQ now (a 6 in the Solomon Islands). We were riding out the first waves OK (we got through the peak of 0.3um/sec on the BLRMs) and then tried EQ mode v2. I think that switching to earthquake mode blew the lock. :(
We switched back to windy mode because ALS looked better that way, and made it to DRMI ASC by the time the R waves hit and blew us out of lock. Switched back to windy more at 8:22 UTC, when the EQ band blrms were back to 0.2-0.1 um/second. The attached screenshot shows a couple of trials of switching back and forth between EQ and windy more while trying to lock. It seems that at this level of ground motion, EQ mode v2 adds too much angular motion in 0.1-0.3 Hz for ALS to handle. (see JIms alog)
Other than the EQs the ground is quiet, no wind no useism.
[Jenne, Daniel, Ed, JeffB]
While recovering the IFO, Jeff and I could not engage the WFS centering for the AS WFS which needs to happen for us to engage the DRMI ASC. After some confusion about what the problem was, Daniel suggested that probably the high voltage supply for the fast shutter was off. We went to the CER mezzanine, and indeed both the OMC PZT and the fast shutter's HV were both off. This likely happened as a result of some vac/Beckhoff restarts that happened today, which probably triggered the interlock that shuts off the HV when the pressure in HAM6 is too high. As soon as we turned on the high voltage, everything was good again.
So, on the one hand, this is not a situation we expect to run into very often. On the other hand, we should probably put a test into the DIAG_MAIN guardian indicating that the HV is off so that we don't spend an hour confused again. There is a test in DIAG_MAIN for the OMC PZT's HV, but that doesn't start looking until we're close to DC readout. We could just use this, since both HVs will be turned off at the same time.
The fast shutter test that checks that the shutter fires by look at the GS 13s at the time of a lockloss where we had more than 14kW gave us one false alarm tonight.
(It correctly reported that the shutter did not fire in a lockloss where the circulating power dropped). Not enough of the power was sent to the AS port for H1:SYS-MOTION_C_SHUTTER_G_TRIGGER_VOLTS to get close to its threshold of 2 Volts.
We probably want to increase this threshold to avoid false alarms so people don't get in the habit of ignoring the warning. For now I have increased it to 25kW, and committed to the SVN.
Evan H., Jenne, Matt, Kiwamu,
We locked the interferometer at 50 W with the latest ring heater setting (RHX = 0.5 W, RHY = 2.5 W, 29588). So far the interferometer has been locked at 50 W for roughly 1 hour.
P.S. we have now having difficulty damping a PI mode (mode2, 15520 ITMX) and decreased the PSL power back to 30 W.
[DRMI lock on POP sensors]
As reported yesterday (29601), we had a difficulty in switcing the sensors from the 3fs to 1fs earlier today (29603). In the end, I manually executed the sensor switch process one by one and for some reasons this was successful. I then measured the open loop transfer functions of the DRMI LSC degrees of freedom, but they looked OK. See the first attachjment. This may be due to that we did not wait for long enough time to let the new CO2 setting settle (29603) for lock acquisition ([CO2X, CO2Y] = [500 mW, 1000 mW]). Not sure.
Also, in lock acquisition, I manually kept aligning PR3 when the interferometer was at ANALOG_CARM in order to maintain the lock.
[CO2 tuning at 50 W]
This is not well tuned, but the below is an OK CO2 tuning for 50 W which gave us a 30% imbalance at the AS port OSA.
[CO2X, CO2Y ] = [300 -400 mW, 0W]
This time I did not spend time for tuning the dCO2 at 20 W or 40 W. Maybe I should have done that to collect more data points.
[PIs]
MODEs 17 and 27 needed a sign flip. Mode 27 seems tricky -- every time when it rang up we needed to flip the control sign.
The PI difficulties are probably my fault. The phases may not have been set properly after the power glitch.
After damping PIs (Terra is writing about this) we tried to take the IFO back to 50 W, but lost lock within a few minutes. I'm not sure why we lost lock, it didn't seem to be PI or the usual sidebands tanking problem, there was a glitch in CHARD and DHARD a second before lockloss (1st attachment).
We spent some time relocking and found that we had some problems durring the latter CARM offset reduction steps which were similar to what we had last night. We measured the DARM loop at RF DARM and at the state CARM 15 pm, it seems fine (2nd attachment).
It is possible that our relocking difficulties were due to bad alignment, or the new TCS. As we were about to try to relock the 3rd time, we got hit by an EQ so we stopped for tonight.
Here is some analysis of the sideband imbalance.
Synopsis -- Overall, adjustment of the CO2 lasers I did at 50 W improved the sideband imbalance at the AS port
However, the resulting CO2 setting does not match what we expected from the previous test (29585). We expected the final CO2 settings to be [CO2X, CO2Y] = [100 mW , 0 mW], but we ended up with [400 mW, 0 mW].
The 45 MHz seems to prefer a high CO2 contrast of roughly 400 mW (CO2X minus CO2Y) regardless of the ring heater settings so far, and this prevents us from further reducing the common lens.
[The sideband evolution as seen by the OSA]
The below shows a plot of the OSA raw output. I plotted several scans from different times, each of them is separated almost by 10 minutes in time.
Also, here is a rough time line of what I have done in this lock.
Here is another plot showing how the sideband amplitude evolved as a function of time.
It is evident that increasing CO2X helped reducing the imbalance. Before I started changing CO2X, there was a slow trend in which the imbalance kept decreasing.
Here is another trend plot.
The carrier recycling gain stayed at 29 at 50 W. The sideband build up or ASAIR90 seemed to have reached some kind of equilibrium after approximately 1 hour or so. Also, looking at ASAIR RF 45 Q, I don't see any change in the DARM offset point -- hopefully this is an indication of a stable optical gain. The Pcal line at 331.9 Hz was too small that it was burried below intensity noise. So I could not directly check the DARM optical gain.
The intensity noise coupling changed as a function of time at 50 W. Here is a plot showing several coupling transfer functions from different time.
The measurement time were set to identical to the ones showed in above or 29637. Here is a plot of the transfer coefficient at 400 Hz as a function of time.
As you can see, the coupling became worse at the beginning for about 20 minutes or so. Then it came back to a value as small as the very begining. Although high frequency above 100 Hz seems to have settled to a small coupling, the low frequency part looked worse at the end. See the first attachment.
As for the HWS signals, while the HWSX signal seems reasonable, the HWSY reports a twice large self heating.
Here are some conclusions from comparison of the predicted ITM lensing (outputs from TCS-SIM) and HWS outputs.
The below shows a comparison between the single-pass defocus measured by HWSX and that predicted by simulator (with some coefficient fitting similarly to 27330):
I added a constant offset to the simulated defocus in order to plot it on top of the measured defocus. I removed the ring heater component that was unfortunately in the middle of settling to the equilibrium in the simulator due to the unscheduled model restart (29592).
The below is a same plot, but for the ITMY substrate defocus.
As noted in the legend, the component from the self heat needed to be increased by a factor of two.
The stability of AS90 was apparently better with the combination of the new ring heater setting and dynamically-decreased CO2s.
The below plot shows a comparison of this lock stretch (Sep. 12) and two lock stretches (Sep. 09) from a day before we started changing the ring heater setting.
The two lock stretches from Sep. 09 both showed AS90 decreasing monotonically as a function of time. This was something we have been suffering from (29486) and has been an issue since the begining of this month (29457) at which we considerably changed the interferometer alignment.
In contrast, AS90 settled to a value pf 600 counts on a time scale of 30 minutes or so in the new lock stretch. This lock was stable until PI modes started ringing up. The carrier recycling gain was slightly lower than the previous two stretches. I don't know if this due to misalignment or new ring heater setting.
Here are the evolution of the wavefront gradient as a function of time as seen by the Hartman sensors.
The starting time is set to Sep 12 2016 00:30:30 UTC at which the interferometer was in the middle of powering up to 50 W. The gradients are initilized such that the gradients are zero at dt = 0.
Change seems effective. We just had ~2 hour lock during which we passed through the RH change transient peak and all PIs are dampable; we're no longer sitting over the 47kHz mechanical modes that were undampable this morning. We've been slowly changing modes over to PLL damping (as opposed to just static BP + phase filters). The potentially problematic Mode 2 (15520 Hz on ITMX) became unstable during the lock but remained damped with the PLL. We've confirmed that some too-high damping gain, even below ESD saturation values, will indeed make damping efforts ineffective; something along the way must be saturating. Damping using the PLL solves this as it outputs some fixed gain that we can keep below this 'bad' gain level (~30k cts).