Notes to self on transition from etmx locklosses we've been having recently (6 Jan 2024), and other lockloss investigations.
POP18 looks quite wobbly. Do we have clipping again? Haven't really checked anything w.r.t. this, so perhaps something to come back to.
A few weeks ago, we'd been having trouble with ALSX noticing that it was locked, so we renormalized it's PD around noon Pacific on 13 Dec 2023. The next lock, it already looked like the renorm was unneccessary, ALSX trans was about 1.15. Stayed at that level over many locks for many days.
Dec 20th 02:15 UTC, first time I see ALSX trans getting to 1.2. After that, pretty consistently at that level. So, probably not related to this week's troubles?
The first lock this morning that was using guardian automatically lost lock during the state Transition_from_etmx, but the lockloss was about 21.5 sec after the swap away from ETMX. So, I thought we should be able to do the swap and sit there, if the locklosses were due to something else like the ETMX bias ramping, or something (not that I know why that should cause a lockloss).
Ibrahim was able to stop and wait for 10+ mins at LOWNOISE_ASC. We ran LOWNOISE_COIL_DRIVERS with guardian, then in the guardian shell did all the prep stuff in Transition_from_etmx, then as a block did lines 5086-5089 to actually do the swap. Lost lock during the 5 sec ramp swapping away from ETMX, so this didn't work.
Checking some ETMY filters (since Sheila et al checked ETMX yesterday). Looks like h1susetmy foton file was last loaded in April 2023 (according to the date of the last _install.txt file in the filter archive), so before O4 began, so really there should be no possiblitiy of changes there, and indeed I don't find any changes.
* ETMY L2 Lock only filter used is FM7 "Qprime", has not changed shape in a long time (just picked a few older foton files out of the filter_archive).
* ETMY L1 Lock only filter used is FM10 "UIMcomp", also not changed in a long time.
* ETMY L3 lock filters FM5, FM8, FM9, FM10. But these are all flat to pretty high freq, before they are notches / stopbands, so no effect really on our low freq stability issues. Also, no changes.
h1susitmx foton also last loaded April 2023, before O4, so shouldn't be any changes there.
Lockloss around 1388535878 (6 Jan 00:24:20 UTC) was about 2 mins after the swap to ETMY and ITMX.
Lockloss around 1388533427 (5 Jan 23:43:29 UTC) was only about 10 sec after the swap.
Seems like we're just quite delicate and susceptible to wind + microseism after this swap?
In case we were running into lack of control authority due to being in the lownoise coil driver state, we tried (without much hope, but trying anyway) to skip lownoise_coil_drivers and go straight from lownoise_asc to transition_from_etmx. We lasted a little more than 5 sec past the swap, but not much more than that, so this isn't an avenue toward different diagnosis path.
The wind is supposed to start getting better after 4pm today, so I wonder if the best path forward is to just wait out the environment? I wonder if (as Ibrahim has said) this level of microseism plus wind is just untenable for us to reacquire lock with.
