I investigated the HWS glitching issue in more detail. I ran the following tests to try to reproduce the glitching seen previously:
take command (number of ring buffers = 4): no glitchingtake command (number of ring buffers = 1): no glitchingtake command (number of ring buffers = 1): no glitchingtake command (number of ring buffers = 4): no glitchingRun_HWS_ITMX and Run_HWS_ITMY at the same time: no glitchingSince we couldn't reproduce the glitching after everything restarted, Kiwamu and I decided that we create some Guardian diagnostics to run continuously on the HWS data. The following channels will be monitored and will issue warning in the case of aberrant behaviour:
Daniel, Terra, Jim, Dave WP5790
A new model was installed on the final core of the h1lsc0 front end machine. Its name is h1omcpi, dcuid=9, rate = 64kHz.
A PI_MASTER.mdl file change was also made.
The new h1omcpi model was installed and started. The h1susitmpi, h1susetmxpi and h1susetmypi models were rebuild and restarted. The DAQ was restarted.
Unfortunately this precipitated an instability in h1fw0 which we are investigating.
Darkhan, Kiwamu,
We found that the optical follower servo (aka OFS) for the Pcal Y had stopped running. Pressing around some buttons on the medm screen, we could not make it back up running. We are suspecting some kind of hardware issue at the moment. According to trend data, it seems that it stopped running at around 18:30 UTC on Mar 18th for some reason. We will take a look at the hardware in the next Tuesday during the maintenance period.
J. Kissel, E. Hall, [R. Savage via remote communication at LLO] This was re-discovered again tonight. Opened FRS ticket #5241 so we don't forget. Email sent to E. Goetz and T. Sadecki suggesting they address on Monday 4/4. Further information from our re-discovery: Looks like Kiwamu had left the calibration line amplitude in a low / off state. We restored the line amplitudes to nominal, and it caused excess noise in the DARM spectrum (lots of lines, a little bit of elevated noise floor). We don't see any sine-wave-like excitation in the OFS, TX, or RX PDs with a single calibration line on at reasonable amplitude (which is contradictory to elevated noise in DARM). Rick suggests: - Check the AOM. - Check the shutter. - Check that the laser hasn't tripped.
Related alogs: 25932 and its comment
Yesterday, we started the preloading test. The CO2 lasers now have an additional common power of about 250 mW, corresponding to an additional common substrate lensing of roughly 31 uD.
Two major conclusions so far:
We need to decide what optimization we need to for this common lensiong. For testing purpose, we will keep running with this pre-loaded configuration for lock acquisition and for full lock with 2 W PSL.
After vent test, current pressures at aux carts: HAM 7/8: 2e-5 Torr HAM 9: 3e-5 Torr HAM 11/12: 4e-5 Torr Continue to monitor pressures to detect any potential outer oring leaks.
John, Chandra Vented each annuli system currently being pumped with aux carts to see which are leaking into diagonal volume (inner oring leak). Attached is plot showing leaks in HAM 7/8 and HAM 11/12 systems, each containing seven annuli. HAM 9 did not leak when vented.
Dave B rebooted the H1HWSMSR machine yesterday. Unfortunately, an investigation of the HWSX images shows that they are still glitching for an unknown reason.
Nutsinee, Kiwamu,
As requested by Aidan, we have restarted the cameras and the computer yesterday after the LHO ISC meeting. We remotely shut the cameras by pressing the buttons on the medm screens. Then we went to the MSR and pressed the reset button to restart the computer. Aidan reported that the corrupted image issue has gone before we shut off the cameras. After rebooting the computer and cameras, we restarted the HWS codes using the old tmux sessions.
Also Aidan discovered that when the images were corrupted it reported an anomalously high pixel mean peak values and in fact it exceeded the theoretical limit with the 12 bits data = 4096 counts. This motivated us to set up a diagnosis test in the DIAG guardian as reported in 26243.
Checked the PSL Chillers per FAMIS#4143.
Aoplogies; I went in to do my weekly reset of the FE Watchdog last night and on my way out i noticed that the Xtal chiller level was at minimum so I added 250ml to bring it upo to the max line. I neglected to aLog this.
120mL seems like a good bit of water for only an ~12 hour period. It's conceivable that Ed would have had to add 250mL Wednesday night since we did fix a water leak on Tuesday and had to drain the laser head circuit in order to do so. We had to add more water to the chiller to get it operational, but only added enough to do this (was not topped off). Corey having to add 120mL the next morning is somewhat concerning.
I have attached 3 and 7 day trends of the relative humidity of the HPO box and the PSL enclosure laser room. Our activity on Tuesday is obvious, and once we put the lid back on the HPO box the RH slowly increased (over the course of ~17 hours) in the box until it reached its previous level. I see nothing that to me obviously indicates that we have a bad water leak, although the slow RH increase in the HPO box could indicate we still have a slow water leak. Will continue to monitor.
[Sheila, Jenne]
We measured the PRC1 ASC loops, and adjusted the suspension compensation such that we could increase the UGF to about 0.3 Hz for each. This included sending the ASC control signals to M3 of the PRM, in addition to M1. There's room to increase the gain, but we want to maintain our heirarchy with the PRC2 loops of 2Hz ugf from last night. See screenshots for new loop TFs.
We tried to modify the way the MICH ASC loops come on, so that we're less rough with the turn-on, but were unsuccessful. The idea was to have FM2 (0.03:0) off but FM3 (:0.03) on so that we don't have a pure integrator. We planned to turn the input on with zero gain, then ramp up the gain. Once the loop is on, we were going to engage FM2 to cancel FM3's pole and give us a pure integrator. This kept kicking us out of lock though, so we've backed out these changes, and just put a sleep in after the MICH loops come on, before any other loops come on. We did however make the MICH yaw loop use the -20dB FM1 initially and then ramp it off, in the same way that the pitch loop does. We also increased FM1's ramp time from 2 sec to 4 sec. This seemed to help enough that we're leaving it for now, and may come back to it in the future.
The "msboost"s in the CHARD loops are causing loop oscillations now, so we've commented them out of the guardian. We'll have to revisit why this is happening, and fix it (refl wfs phasing due to preloading?). This may need to be addressed before we can power up.
The environment is starting to get fussy (high winds and moderate useism), so we're going to leave things as-is for now. Locking should all be the same as usual through DC readout. We make no warranty for power-up, since we haven't had a chance to try it.
Plan: Get to Part3 or DC readout, measure phasing of refl wfs by shaking mcl and checking that signals all in I phase (hopefully). Should be okay with the new preloading, since it's all fine with regular hot IFO. Fix if needed. Then try powering up!
Krishna, Michael,
We left Seattle at 10 AM and reached LHO at 2ish PM. Hugh helped us unload the BRS-2 vacuum can and other parts into the 'loading bay' at the Y-End-station. Tomorrow morning, we will clean them and move them in and begin assembly, which is expected to take 2-3 days and commissioing/debugging may take a week. We will try to complete activities in the VEA by mid-day to minimize impact on commissioining.
1545 - 1600 hrs. local -> To and from Y-mid Exhaust check valve bypass opened, LN2 at exhaust after 3:39 mins with LLCV bypass open 1/2 turn Next CP3 over-fill to be Fri. before 4:00 pm hrs. local
Following the grouting of the HAM5&6 HEPI piers, and in light of JeffK's work untangling my designs (in LHO alog 26002), I wanted to review the design of the HAM 4/5 S0-1 feedforward. To do this I took a new set of measurements for HAM's 4 and 5 so I could see how similar the different transfer functions were. The measurements I took were: the transmissibility from the FF L4Cs on St0 to the St1 GS13s with the ISI in both damped and isolated states, and the actuator to GS13 tfs in both the isolated and damped states. I got all 24 measurments for HAM4, but I only got the all of the Z measurements for HAM5, so I looked at that dof filter design.
Jumping straight to the punchline (first plot, calibrated ground and GS13 spectra for HAM4) I have a new Z FF filter that does a lot better and makes more sense, installed on HAMs 4&5, and I'm working on similar redesigns for the remaining dofs. The two traces to really look at are brown and light blue, the GS13 measured displacement with the old and new FF filters. Compared to dark blue (FF off), the old FF filter mostly improved motion between 20 and 30 hz and only worked well with a gain of .5, something I couldn't explain when I installed it. The new filter works better from a few hz to almost 60 hz.
The second plot is the transmissibilties for HAM 4 (blue) and 5 (green) in both damped (solid) and isolated(dashed) states. The damped states are very similar, though I'm not sure why there is a gain difference. The Isolated tf's are more different, but probably it's probably due to some differences in the feedback loops, I should probably take a look at that.
The third plot is the stiffness tf, color key is the same, HAM 4 (blue) and 5 (green) in both damped (solid) and isolated(dashed) states. Again the damped states are similar, the isolated measurments are a little more different. Importantly, the differences are consistent with the transmissibility measurements, which means the filter design ends up looking the same between the chambers, as shown in my next plot.
The fourth plot is the ideal filter design for HAMs 4&5. This is done following the process outlined in JeffK's 26002 alog, where the ideal filter is roughly speaking the transmissibility over the stiffness. One of the other things I wanted to look at was what state the ISI needed to be in when designing the filter, as I thought that I had found in the past that the ISI needed to be isolated for both the transmissibility measurment and the stiffness measurement. It looks like it doesn't much matter as far as the final filter design is concerned. The isolated design is a little different at a few hz, though it's hard to see because the data is crappy. The data is much cleaner with the damped transfer functions. UGF's for this loop are on the order of 35hz, so feedback is not doing a lot at 10 hz, that probably explains why the isolated and damped filters end up looking the same, and why they differ more at lower frequency. Also nice to see that the filters for HAM 4 & 5 should be the same, regardless of the either ISI's state. The cleaner data for the damped configuration is a strong argument for using that configuration for my future redesigns.
The fifth plot is a comparison of the HAM4 ideal filters(blue and green), the old filter (red), and the new filter (black). You can see that the black line matches green and blue very well between 10 and ~40 hz, but red only matches well in phase right at about 25 hz. This explains why the old filter only did well at 20-30hz. The gain mismatch at 25hz also explains why the old filter needed a gain of .5 to work. The new filter works well from a few hz up to almost 60 hz, with a gain of 1.
I still have my data and the script that I used for the orginal design, Jeff rehashes it in his log. I still don't understand the differences, what I screwed up originally. I think the approach used both times was the same, but I clearly did something wrong the first time around. I'm in the process of repeating this for the other dofs.
The last plot is HAM5, comparing the old FF filter to the new. Red, blue green are with the new filter, brown, pink and light blue are with the old filter.
I've look at all dofs now, X&Y needed to be redone as well, but the filter ended up just looking like the Z filter with a different gain. Attached are gnd STS, FF L4C and ST1 GS13 spectra show the performance before and after for both HAMs in X&Y. Solid lines are after, dashed are before. Similar to Z, these dofs show marked improvement. I've started this process with HAM2, which will likely apply to 3&6 as well, but those chambers use the HEPI L4Cs and the coherence isn't as good.
I did a common CO2 test with the fully locked interferometer. Unfortunately the interferometer could not stay locked due to the SRC1 and 2 pitch loops running away.
The interferometer was fully locked with 8 W PSL (10 W was not stable and in fact it lost lock once due to the angular instability 26208). ASC loops were all closed. No LSC feedforward engaged. DARM line was injected at 331.9 Hz monitoring the cavity pole and optical gain. The line was provided by the LSC lockin oscillator and not by Pcal Y.
Since it lost lock, I set the CO2s back to nominal. I will check the data later.
Inconclusive. The test lasted only for 17 minutes and this is too short to say something conclusive.
The goal of this measurement was
The first goal was not achieved because the interferometer unlocked before reaching the predicted optimum. Regarding the second goal, it seems that the common lensing affected the SRC loops. This is something we kind of expected.
[The test]
The first two attachment show the trend of various channels during the measurement.
[Cavity pole]
With the line injected at 331.9 Hz, I was able to track the change in the cavity pole frequency. It is shown in the third attachment together with the simulated substrate lensing of ITMX and ITMY. The time scale of this plot is the same as those for the first two attachments. The cavity pole measurement is only valid in the time band roughly between t = 2100 sec to t = 3800 sec. The cavity pole increased at the beginning and went down afterwords. One might think that the highest cavity pole frequency is the point where the SRC couples to DARM maximally, but I can not support that hypothesis. If one looks at the lensing of ITMs, they were slowly settling to 10 and 7.5 uD for ITMX and ITMY respectively in 2000 - 2500 sec while the cavity pole kept increasing. Immediately after that the CO2 lasers started dominating the lensing and pulled the lensing upward rapidly. It does not look like that the ITM lensing went across the optimum SRC coupling point given the behavior of the lensing. It rather looks as if it was caused by some other effect (perhaps some ASC loops ?).
[Stefan, Sheila, Jenne, Kiwamu, Hang]
Tonight ended up being a night of retuning the PRC2 loops, to make them sensible. Now they have UGFs around 2 Hz, rather than ~20mHz.
The thought was that perhaps this 0.4Hz oscillation that we regularly see is a result of not holding the PRC2 loops tight enough. If the PRC isn't following the motion of the SOFT loops (which move when we power up due to radiation pressure), then we're getting a modulation in the amount of power coupled in to the arms from the PRC. This power modulation makes the SOFT oscillation even worse. If we force the PRC to follow the arm pointing, then we won't have this power modulation, so we will hopefully only have to deal with "regular" angular instabilities.
In the end, we tuned the shape of the PRC2 pitch and yaw loops while sitting at ASC_PART3 by compensating the PR3 suspension resonances. This made the loops beautifully stable, so we cranked up the gain. For the PRC2_Pit loop, we use the new shape (although low gain) for DRMI_ASC. For the PRC2_Yaw loop, we use the old loop shape and gain during DRMI_ASC, and then use the new loop shape for the main ENGAGE_ASC steps. Recall that all DRMI ASC loops except for MICH are turned off after offloading the alignments, and are left off for the carm offset reduction sequence. Also, during DRMI_ASC, the PRC2 loops actuate on PR2, while they actuate on PR3 in the final state, so it's not quite as weird as it sounds to be using different shapes at different states.
See the attached PRC2 open loop measurements for the OLG after our retuning and gain increases. All of these sequences have been added to the guardian.
Now however, when we try to go from 12W to 15W, we see an oscillation in PRC1 Yaw. We should probably retune the PRC1 loops, including actuating on the lower mass in addition to the top mass of the PRM (right now we only push on the top mass). We leave this for another day.
ASC plan for tomorrow: Run Hang's A2L script, and find SOFT offsets that put us at our actuation nodes. Hopefully this will reduce the angular effects of increasing the power, and help us get back to 20W.
Other things tonight:
1) I added filters with a gain of -1 to the roll mode damping for ETMY and ITMY, so that the gains for all of the loops will be positive from now on. This is in the guardian and should make things easier for people who need to damp roll by hand when it is rung up badly. I will work on doing this for bounce as well but that is not finished.
2) We had 2 locklosses in DRMI on pop tonight, looking at them I saw that we were switching the coil drivers while the DRMI sensors were still ramping. I added a timer to make sure they don't switch until the ramp is over now.
Jenne and stefan also made some bug fixes for the code that switches off the soft loops when their outputs are too large.
It seems like the problem we have now is really the PRC1 Y loop going unstable, which is not too suprising since this is cross coupled with the PRC2 loop which we changed last night.
The attached screenshot shows on the left the optical levers, POP build up and PRC control and error signals. The left panel is for a time when we rang up the half Hz CSOFT instability which is visible in the optical levers, and the pop build up somewhat. The second panel shows what happened after we increased the gain in the PRC2 lops, we see nothing in the optical levers but the PRC1 Y loop is unstable. This is the same problem kiwamu had later in the night.
As Jenne said it should be relatively easy to fix the PRC1Y instability.
ITMY ring heater was left on for an overnight measurement. The upper and lower segments of the ring heater were set to 1 W (i.e. 2W in total). The interferometer is aligned but in the down state. I started the HWS codes on h1hwsmsr because they were not running. I have not updated the reference images for the HWS codes this time. Therefore they use whatever the reference images that are in ~/temp/. The ring heater will be automatically switched off at around 6 am in local time by a script running on opsws4.
The HWS code should have been running in a single tmux session containing two windows. Clearly this is too easy to circumvent. I'll talk to Jamie about seeing if we can get this set up as a daemon process instead with the new version of the code.
JimB and I are working on this issue at the moment. We are trying to get rid of tmux sessions by implenting monit. As of yesterday, we succeeded in running the hartman codes under the managmenet of monit. The implementation is still underway and about 80% done at the moment.
h1fw0 continued to be unstable. First I power cycled h1fw0 but this did not help. Then I powered down h1fw0 and h1ldasgw0 (in that sequence). When I powered up h1ldasgw0 it went through an extensive set of startup diagnostics. I then had to manually mount the QFS file system and NFS export it. Finally I powered up h1fw0. Its been running for 5 minutes, so too early to tell if we have fixed this.