Ryan S, Keita K, TJ S
After the lock loss at 0427 UTC (2027PT), Ryan S started an initial alignment. Green went as usual, but he couldn't get Xarm IR to go. I'll put more details below, but the issue was that the change I made earlier today to change the LASER_PWR guardian to use the IM4 trans channel (alog87581) upon Sheila's findings that gain scaling based on IMC REQUEST caused gain reductions after the power outage (alog87545), caused the power scaling to go to -1.4 after the lock loss. We reverted this change and will have to make it a bit smarter to use this channel in the future.
More details:
Ryan and I initially thought it was an odd alignment and chased that with the usual checking of suspension alignments, checking temperatures, etc. We just couldn't get any flashes to show up on LSC-TR_X_NORM_INMON. On the front FOM, POP90 was visible and noisy, both things that shouldn't be at this point of initial alignment. We eventually thought that maybe it was a dark offset issue since the TR_X channel wasn't quiet at 0, so we ran the dark offset script. This didn't help but I've attached the SDF screenshots. While looking at this channel more, Ryan noticed that it would dip below 0 frequently, almost as if it was flashing but into the negative values. Sign flip somehow? We tried to understand this for awhile and eventually gave up to call the top person on the call list, Keita.
While we were describing the issue to Keita and starting to look into where a sign flip could have happened, Ryan noticed that the power scaling was at -1.4. Since this should just scale with the IFO input power, it shouldn't ever be negative. I plugged in 2.0 and, then everything looked normal again. Looking at the last attachment, IM4 trans goes negative when we lost lock and powered down. The power scaling adjusted as such, and stopped adjusting when the rotation stage stopped moving, as it was supposed to do. If we want to use this channel, we'll have to add some logic to avoid this.
The rest of initial alignment went ok, I had to manual over the PRC offload check because the high useism. DRMI and PRMI aren't locking now, it looks like due to ground motion, but I'll have it try another initial alignment again and report back.
I'm putting the observatory mode to useism and stopping trying to lock. I've had another initial alignment and numerous small DRMI locks, but it just won't hold longer than a minute. Flashes look great, but there is just too much ground motion to hold. Maintenance day starting soon.