IFO is still not locked. Daniel and Austin (during and post-shift) and thankfully now, Camilla have all tried helping but so far we don't know what the problem is. Here are some of the leads.
Initial Alignment is not, and has not been working since 12PM today (and from last week's initial alignment, this seems to be a known issue). Firstly, MICH_BRIGHT and MICH_DARK do not lock according to guardian (but they seem to lock according to Austin and Daniel at the start of the shift). The behavior of the MICH trace is oscillating between the threshold (4000 cts) and 0, which is odd. After getting MICH "as locked as possible", we (Daniel, Austin and I) manualled over to SRC locking, which also failed in a similar way. Initial alignment did not work at all. According to Daniel, the gain had to be bumped 10x to lock MICH but then the init_alignment just failed completely afterwards (everything swinging). I tried touching the various documents' manual alignment sliders (BS, SRM, SR2) to no avail. Even during SRC manual locking, the differential in the "high highs" and "low lows" was nowhere near what it trended as. With zero indications of the problem after parsing through the code and attempting to call other people, Camilla offered help.
With this help, we could not get locked still so we thought to try normal locking without any confirmation that initial alignment works. Somehow, this worked. IFO is currently powering up. DRMI locked easily while bumping the BS during Engage_DRMI to better the usual signal traces.
In diagnosing the issues with the help of past alogs and trends, the last time MICH offloaded successfuly was June 26th. It failed on the 1st, 3rd and today. While the locking issue seems to have been fixed, we still have not fixed the initial (incl. manual) alignment locking problems. Expecting to be locked within the next 30 minutes now.
Special thanks to Austin, Daniel and Camilla for volunteering their time to assist with this eventful eve.
Summary:
ASC-AS_A seems to have been badly clipping, but the LSC signal came from the two quadrants on which only a tiny fraction of the beam power was falling. A bad recipe as alignment-length sensing coupling is maximized.
Details:
In the 1st attachment (same time window as Daniel's plot, but more channels), the right-most column shows the AS_A_DC_NSUM was about half of AS_B_DC_NSUM. (FYI this is never the case in e.g. full lock.)
The second column from the right shows AS_A_DC_PIT_OUT16 was basically 1 except some transient responses (circled in red), which means that all of the power was on quadrants 1 and 2. If you look at AS_B_DC_PIT and YAW on the same column you can see that the beam was not clipping on AS_B.
2nd and 3rd column from show that most of the signal was in quadrants 1 and 2 (unsurprising, but they were in I phase not Q as was pointed out by Daniel). However, for whatever reason, quadrants 3 and 4 seem to have somewhat better (but not much better) demod phase, therefore Q3 and Q4 were actually larger than Q1 and Q2.
This resulted in AS_A_RF45_Q_SUM signal mostly coming from Q3 and Q4 (circled in cyan).
I cannot remember if any kind of DC centering is done during initial alignment, but if not, we need one.
Also measure AS_A and B RF45 phase in full lock (maybe we can check this using calline or violin?).
Here some traces when we tried to lock the bright Michelson. A couple observations:
Turns out that there's no DC centering on AS WFS during BRIGHT MICH in the initial alignment. See the first attachment.
45MHz demod phase was OK for full lock with 60W power for both AS_A and AS_B. 2nd and 3rd attachment. I don't know if that's the case for lower power. But the demod phase is clearly off for initial alignment, and I don't want to bump up the LSC gain by a factor of 10 to compensate.
What we need is: