[Jenne, Ibrahim, RyanS, Sheila, Robert, TJ, Elenna, Jennie, Oli, Jim, others]
Something is different with the IFO today, and it's not great. Since we haven't pinpointed what the problem is, we don't really know if it is related to our mediocre range last night and struggles to relock after that, or if it is new from maintenance day. So far, the solution to get back to (almost) NLN has been to put in *massive* offsets in AS_C. This did enable us to get locked (and avoided the ASC ringup that was causing locklosses this afternoon), but it left us with a large scatter shelf in DARM. Robert had the good suggestion that we see if, once locked, we could walk the offsets back to their nominal values of zero; doing this caused an ASC ringup, and it's probably the same thing that we'd been seeing throughout the day. So, indeed going toward the nominal offset of zero for pit and yaw on AS_C is not an okay place right now.
We began to narrow in on AS_C since, during initial alignment, it is in-loop for SR2 and SRY alignments, and it was causing SR2 to be pulled very far in yaw. We were seeing this visually on the AS_AIR camera, which looked really unusually terrible. In the end, Sheila hand-aligned SRY, and then put offsets into AS_C such that the ASC now servos to that point. But, the offsets used are +0.46 in pit and -0.85 in yaw, so really, really big. However, with these in place, we were able to let DRMI ASC run, and it ran fine.
Since that worked (the large AS_C offsets), we let the IFO relock the rest of the way, and it kept on working. Per a suggestion from Elenna from earlier in the afternoon, after we completed REDUCE_RF45 I manual-ed to ADJUST_POWER and increased the power 5W at a time (waiting for ASC to mostly converge between steps) until we were at 60W, then I manual-ed back to LOWNOISE_ASC. After that, I just selected NomLowNoise and guardian finished things. We ended up 'stuck' in OMC_WHITENING, although the violins were coming down with the damping that guardian was doing. It was around here that I tried ramping down the AS_C yaw offset with 10 sec ramp times to see if it would reduce the scatter shelf that we saw in DARM. See first attachment, with the scatter shelf circled. My first step was to -0.80, second step was to -0.70, and we had an ASC ringup and lockloss at this step. I wasn't sure after the first step, but definitely as we were ramping to the second step (before we lost lock) RyanS, Robert, and I all agreed that the scatter shelf was getting worse, not better. But, we lost lock before I could put the offset back.
We still don't understand *why* we need these large offsets in AS_C, just that we do. Since I hadn't yet SDF-ed them, we had 2 locklosses from ENGAGE_DRMI_ASC when the nominal zero-offsets were used in AS_C. I have since saved the AS_C offset values, and the offset switch being on, in both the safe.snap and observe.snap SDF files. The second and third attachments show these.
We've been trying to think through what from maintenance day could possible have had anything to do with this, and the only things we've come up with are the h1asc reboot and the grounding of SUS racks. From Jeff's alog about the h1asc changes, it seems quite clear that those are just additions of monitor points, and that shouldn't have had any effect. While we were unlocked after our one successful attempt, Robert went into the LVEA and un-grounded those two racks that Fil grounded earlier today. Robert is quite sure that he's checked the grounding of them in the past without any consequence to our ability to lock, but just in case we've got them undone for tonight. So far, that does not seem to have changed the fact that we need these crazy offsets in AS_C. Just in case, Robert suggests we consider rebooting h1asc tomorrow to back out other changes that happened today (even though those shouldn't have anything to do with anything).
For right now (as of ~7:20pm), my best plan is to see if we can get to Observing, and should a GW candidate come through, have the detchar and parameter estimation folks exclude all data below ~80 Hz. This is a really very poor solution though, since this huge scatter shelf will be quite detrimental to our data quality and could cause our search pipelines to trigger on this scattering.
Related thought, but not yet thoroughly investigated (at least by me), is whether our problems actually started sometime yesterday or last night, and aren't at all related to maintenance. As partial 'evidence' in this direction, I'll note that kappa_c has been dramatically low for the last two Observing segments last night. We haven't gotten all the way to NLN tonight yet (OMC_WHITENING didn't finish before we lost lock), but kappa_c looks like it might be even lower now than yesterday. The 4th attachment shows kappa_c for last night's locks (screenshot taken before we locked today). So, maybe we've had some clipping or severe degredation of alignment in the last day or so. Jim checked all of the ISIs, and nothing seems suspicious there.
Last thing - since we had run the dark offsets script earlier this afternoon (probably was a red herring) and I saved all of the SDFs in the safe.snap, we will have diffs for Observe.snap. All diffs that have _OFFSET in them should be accepted (and screenshot-ed).
Oh, also, the squezing was not at all good when we locked. I suspect that it's because the squeezer alignment isn't matched to this bad-but-working alignment of the rest of the IFO. Sheila should be able to log on in a little while and can hopefully take a quick look to see if there's anything to do.
Adding screenshot that summarizes/ supports what Jenne is saying above.
A little over 1 day ago, there was a drop in optical gain. There isn't an indication of a shift in alignment of SR2, or the OMs at that time. AS_C QPD is in loop, so that signal being zero only shows that we weren't using any offsets at the time. The drives on OM1 + OM2 show that their alignment also didn't shift at the time. Since they are used to center the beam on AS_A and AS_B, their drives would need to shift if there is a large shift in the real beam position on AS_C. You can see that for today's lock, where there were large offsets on AS_C, the alignment of SR2 is different, as well as the alignment of OM1 + 2 (indicating that the alignment onto AS_C is different). We might be clipping on the OFI, or elsewhere at the AS port.
Here's an image of the AS camera:
- before optical gain dropped: https://lhocds.ligo-wa.caltech.edu/cr_screens/archive/png/2024/04/21/05/nuc30-2.png
- after the optical gain dropped but before the offsets: https://lhocds.ligo-wa.caltech.edu/cr_screens/archive/png/2024/04/22/05/nuc30-2.png
- tonight with offsets in AS_C: https://lhocds.ligo-wa.caltech.edu/cr_screens/archive/png/2024/04/23/22/nuc30-2.png
you can toggle through these three in different browser tabs to see that the alignment didn't seem to shift until we added offsets. So all indications are that these offsets aren't good, and we should not be using them, except that they appear to have allowed us to lock the IFO.
Editing to add another screenshot: As Ibrahim checked several times last night, the osems indiczte that SR3 + PR3 haven't moved. In early Tuedsday lock where the optical gain first dropped, the power recycling gain was unchanged, with the introduction of offsets last night it seems slightly lower (and the optical gain also took another step down).This attachment shows that the offsets moved OM3+ OMC, which I did not expect (since the AS centering beams fix the beam axis arriving on OM3, I wouldn't have expected the AS_C offsets to have moved these optics). But they didn't move for the TUesday morning low optical gain lock.
Jennie W, Sheila
Sheila suggested I use the oplevs for SR3, BS, ITMY, ITMX. ETMY, ETMX to compare our alignment out of lock right before we started locking the main IFO (ie. after locking green arms). I chose two times we were in FIND_IR (state 16 in ISC_LOCK).
One of these times was after a lockloss from a very high range lock from Monday afternoon (~157 Mpc).
GPS reference time in lock = 1397864624 (16:43:26 PDT)
GPS reference time in FIND_IR presumably before our current weird alignment/locking problems = 1397868549 (17:48:51 PDT)
The other time was in our last locking sequence Tuesday morning before maintenance Tuesday started. We did not get to NLN but fell out before this at ENGAGE_ASC_FOR_FULL_IFO (state 430 in ISC_LOCK).
GPS reference time in FIND_IR = 1397918790 (07:46:12 PDT)
The main OPLEVS which changed more than 1 micro-radian were ITMY PIT (down by 1.24 microradians)
ETMY PIT (up by 1.02 microradians)
ETMY YAW (down by 1.14 microradians)
ETMX PIT (up by 2.35 microradians)
ETMX YAW (up by 3.21 microradians)
While SRY was locked (last step in initial alignment), we moved the OFI as much as we could, by putting large offsets in the actuator filter banks. No discernable power level changes are visible. This isn't surprising given how weak the OFI actuators are, but seemed worthy of eliminating a possiblity.
