Over roughly 7hrs, we had 2-consecutive ETMx Glitch locklosses taking H1 down from OBSERVING.
After that last lockloss, have had 2-consecutive LOWNOISE_ESD_ETMX locklossess while trying to get H1 back. An alog search on LOWNOISE_ESD_ETMX lockloss issues were noted in alog82912. Since we are having a similar situation (high microseism and frequent locklosses at this state), I made changes to LOWNOISE_ESD_ETMX in ISC_LOCK.py by increasing the (1) ETMx ramp time & (2) a timer after this step (see attached):
- etmx_ramp = 60 (this was 20)
- self.timer['ETMswap'] = 60 (this was 30)
Change Save & Loaded for ISC_LOCK and we are currently on the way to see if we can get through LOWNOISE_ESD_ETMX (or have a 3rd Consecutive lockloss here).
Well, that didn't work. H1 had a 3rd-consecutive lockloss within a couple seconds of going through LOWNOISE_ESD_ETMX. (Posted situation on mattermost & consulting the Call List for assistance.)
So, I am restoring the (1) etmx_ramp back to 20 (from my 60) & (2) self.timer['ETMswap'] back to 30 (from 60).
Attached is the restored values noted above.
Below you can see what happened---lockloss happened FAST within 4-seconds of starting LOWNOISE_ESD_ETMx. Jim wsa first on the call list and he asked if his ETMx ESD limit steps happen---they did not even get to try---that step is the LAST step for LOWNOISE_ESD_ETMx.
Here is the ISC_LOCK for this lockloss:
2025-03-09_17:35:40.042953Z ISC_LOCK [TRANSITION_FROM_ETMX.run] timer['ETMswap'] done
2025-03-09_17:35:40.092335Z ISC_LOCK [TRANSITION_FROM_ETMX.run] ADS convergence check
2025-03-09_17:35:40.195243Z ISC_LOCK EDGE: TRANSITION_FROM_ETMX->LOWNOISE_ESD_ETMX
2025-03-09_17:35:40.195907Z ISC_LOCK calculating path: LOWNOISE_ESD_ETMX->NOMINAL_LOW_NOISE
2025-03-09_17:35:40.196027Z ISC_LOCK new target: LOWNOISE_LENGTH_CONTROL
2025-03-09_17:35:40.201276Z ISC_LOCK executing state: LOWNOISE_ESD_ETMX (558)
2025-03-09_17:35:40.206296Z ISC_LOCK [LOWNOISE_ESD_ETMX.enter]
2025-03-09_17:35:40.216682Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] switching ESD back to ETMX
2025-03-09_17:35:40.217217Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] ezca: H1:SUS-ITMX_L3_LOCK_L_TRAMP => 60
2025-03-09_17:35:40.217751Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] ezca: H1:SUS-ITMX_L3_LOCK_L_GAIN => 0
2025-03-09_17:35:40.218187Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] ezca: H1:SUS-ETMX_L3_LOCK_L_TRAMP => 60
2025-03-09_17:35:40.218678Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] ezca: H1:SUS-ETMX_L3_LOCK_L_GAIN => 1
2025-03-09_17:35:40.219270Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] ezca: H1:SUS-ETMY_L3_LOCK_L_TRAMP => 60
2025-03-09_17:35:40.219711Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] ezca: H1:SUS-ETMY_L3_LOCK_L_GAIN => 0
2025-03-09_17:35:40.219923Z ISC_LOCK [LOWNOISE_ESD_ETMX.main] timer['ETMswap'] = 60
2025-03-09_17:35:44.717933Z ISC_LOCK [LOWNOISE_ESD_ETMX.run] Unstalling IMC_LOCK
2025-03-09_17:35:44.853115Z ISC_LOCK [LOWNOISE_ESD_ETMX.run] USERMSG 0: IMC_LOCK: has notification
2025-03-09_17:35:45.067039Z ISC_LOCK JUMP target: LOCKLOSS
2025-03-09_17:35:45.070405Z ISC_LOCK [LOWNOISE_ESD_ETMX.exit]
2025-03-09_17:35:45.140996Z ISC_LOCK JUMP: LOWNOISE_ESD_ETMX->LOCKLOSS
2025-03-09_17:35:45.141251Z ISC_LOCK calculating path: LOCKLOSS->NOMINAL_LOW_NOISE
2025-03-09_17:35:45.142460Z ISC_LOCK new target: DOWN
2025-03-09_17:35:45.150074Z ISC_LOCK executing state: LOCKLOSS (2)
2025-03-09_17:35:45.150466Z ISC_LOCK [LOCKLOSS.enter]
2025-03-09_17:35:45.154072Z ISC_LOCK [LOCKLOSS.main] ezca: H1:GRD-PSL_FSS_REQUEST => READY_FOR_MC_LOCK
2025-03-09_17:35:45.267242Z ISC_LOCK JUMP target: DOWN
2025-03-09_17:35:45.271745Z ISC_LOCK [LOCKLOSS.exit]
2025-03-09_17:35:45.331052Z ISC_LOCK JUMP: LOCKLOSS->DOWN
2025-03-09_17:35:45.331185Z ISC_LOCK calculating path: DOWN->NOMINAL_LOW_NOISE
None of the first 3 LOWNOISE_ESD_ETMx (558) locklosses have an IMC tag. (3rd one had a WINDY tag & we did transition back to SEI_ENV = USEISM at 1721utc (about an hr ago).)
All 3 have the REFINED tag.
A Look At Microseism with comparson to Elenna's plot in Nov when the microseism season was starting and here we are a few more months into it. We've certainly had worse days, but we are in the middle of a noisy microseism period at about 600 counts since last night.
STATUS: Have been holding at MAX_POWER [520]
Here is a look at the last LOWNOISE_ESD_ETMx (using an Elenna template).
For the 4th locking attempt, time settings were restored (from change I made), and on this one I held at MAX POWER [520] for about 50min while reading through alogs, looking at previous lockloss to post in the alog and then eventually just continued on, and this time, H1 made it thru LOWNOISE_ESD_ETMx. Attached you can see how things looked going from TRANSITION FROM ETMX [557] to LOWNOISE ESD ETMX [558]....in contrast to my lockloss alogged earlier, this one looks like sort of like the plot on left from Elenna's scopes posted here.
