We kept losing lock at DHARD WFS, and Jenne and I spent a lot of time figuring out why. Here are some notes about things that we tried:
- We tried going to RF DARM, and running the DHARD WFS engagement by hand
- It seemed like turning on the small negative gains for DHARD (normal for the state) was bad
- we tried positive gains instead, and they seemed ok for the engagement
- this is REALLY WEIRD, we didn't do anything to DHARD or AS WFS today that would cause this
- we tried walking DHARD around a bit to see if we were very far off in the alignment, but that didn't seem to be a problem
- error signals were close to zero and the AS camera looked good
- We also held in down for a bit to check the AS dark offsets in case they had somehow changed significantly
- Corey SDFed these changes in observe here, and I SDFed them in safe
- they were not largely different, so we realized that couldn't be the issue
- We tried keeping DHARD with a flipped sign and proceeding with locking
- we then lost lock a few times next at CARM OFFSET REDUCTION- it looked like the DHARD error signals got very big very fast in this state causing a lockloss
- as a note, we were skipping over the shutter ALS states so we could use the green alignment as another witness for what was going on
- we were worried the green alignment was very bad somehow
- This led us to think that maybe the DHARD signal before CARM OFFSET REDUCTION was just junk, maybe the offset was too big for the signal to be meaningful, and the sign flip was just an indication of that
- then we tried NOT engaging DHARD, and just going to CARM OFFSET REDUCTION, which is scary because the camera gets very wobbly, but it didn't necessarily cause a lockloss
- I then tried moving DHARD by hand again
- I found that we could then engage DHARD with the correct gain sign, so that seemed promising
- we moved on and started losing lock at CARM TO REFL
- at this point Jenne had to leave
- I eventually figured out that the lockloss occurred after the ramp from TR CARM to REFL9.
- essentially the buildups would get very noisy as we ramped and then the suspensions would saturate shortly after
- This reminded me of the exact same problem that Jenne and I had to solve previously. I couldn't find an alog, so either we didn't write one (boo) or I forgot the correct search terms to find it
How I solved the problem:
- I ran through CARM to REFL by hand
- essentially, I set the LSC-TR_REFLAIR9_OFFSET by hand based on LSC-TR_REFLAIR9_INMON (inmon was about 0.17 so I made the offset -0.17)
- then the state sets up some filters and the ramping input matrix, just dropped those lines of code into the guardian shell
- I found that ramping partway made the buildups a bit fuzzy but less fuzzy (I vaguely remembered trying halfway ramps with Jenne the last time we had this problem)
- by "partway" I mean that the TR CARM intrix value is 1, and the final REFL9 value needs to be 2, so I ramped TR CARM to 0.75 and REFL9 to 0.5
- Then, I remembered that we needed to reduce the offset I set above by hand
- essentially, we think that whatever that offset is, it is too far away from where we think we are in the carm offset space, so we make it smaller and bring the arms closer to resonance
- The offset was set to be -0.17 to cancel out the inmon signal
- I stepped down towards zero in 0.01 steps
- each time, I saw the PRG increase accordingly, and the buildups got less fuzzy
- then I ramped another bit, so TR CARM to 0.5 and REFL9 to 1
- reduced the offset a little more, ramped TR CARM to 0.25 and REFL9 to 1.5
- reduced the offset a little more and then finished the ramp
- by the time I did the final ramp, the offset was down to -0.1
- Then I was able to complete the rest of the carm to analog procedure by letting the guardian run
Some final notes about minor mistakes I made after:
- after running ENGAGE ASC FOR FULL IFO, I manualed to PARK ALS VCO and SHUTTER ALS, then manualed to SOFT LOOPS
- I forgot that we go to DARM OFFSET after full IFO ASC, so the OMC guardian couldn't lock the OMC and we stalled in PREP DC READOUT
- manualed to DARM OFFSET and then back to PREP DC READOUT
Apologies for missing this detail in my original alog, but we had run two successful initial alignments. The first was run as usual after the end of the maintenance day. Green alignment converged properly, arm alignment looked good. We ran a second after the early problems with DHARD, thinking that maybe the alignment was poor. Same results, good green convergence, but problems with DHARD.
In general, the alignment of the arms and corners looked very good through this whole process. When I stepped in DHARD P and Y, I was making very small steps overall to correct the alignment, steps on the order of 0.01 or 0.03. In general, the buildups and the camera showed good alignment. The green arms also stayed well aligned when the DHARD signal engaged properly, which is telling me that our initial alignment was doing the correct thing for the ITMS. Is it possible that the CARM offset was too large? I have been wondering this, because if the alignment was decent but the WFS signal was junk because we were too far from resonance that might explain this behavior, and it lines up with the fact that the better fix was reducing the carm offset more before moving to REFL. I'm not sure if that makes sense though.
As a side note, DRMI was mostly locking very quickly throughout the night.