TITLE: 01/07 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Once our locking issues were addressed after being worked on during the previous shift, smooth sailing back to observing for most of the evening.

• 00:01 Lockloss from LOWNOISE_LENGTH_CONTROL
• Success with fixing issues progressing past TRANSITION_FROM_ETMX; see alog75225 and comments for details
• 00:45 Reached OMC_WHITENING
  • Violins are rung up, starting to damp
• 01:05 Reached NLN
  • Accepted one SDF diff on h1susetmx (attached)
• 01:07 Started observing
• 03:05 Lockloss - alog75228
• 04:04 Reached NLN
• 04:08 Started observing

Lockloss @ 03:05 UTC - 2:00 into lock stretch, no obvious cause.

Things were running smoothly once H1 locked after fixing the TRANSITION_FROM_ETMX lockloss issues, so I'm unsure the cause of this one. LSC-DARM showed first motion.

TITLE: 01/07 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:
IFO is still LOCKING and has been down due to microseism and troubleshooting

Update on ETMX Transition failures:

After a lot of investigation with Shela, we may have found the issue, and Ryan S is going to see if we are able to get to observing.

In short, we think that a cal line filter was incorrectly accepted to be on in the safe.snap SDF table. Additionally, Sheila added a new "missing filter" that was in the supposedly identical L1 Lock filter for EX/EY but was not in EY.

The investigation, details and screenshots are found in alog 75225

LOG:

None

Sheila, Ryan S, Ibrahim

Continuation of Transition from EX Failures - The Lockloss causing state

• Sheila came on in the afternoon to help - this time investigating EY L1 and L2 saturations/overflows. 
• She found that during this state, which is known to be more delicate, the DAC values to get quite high, sometimes saturating and sometimes (the running idea is). Causing locklosses given already high microseism.
• Even during the last successful lock - at this stage, DAC2 values were still high.
• The filter for this drive is supposed to be a copy of the EX actuator, which does not see such high values, but maybe some difference contributing to instabilities.
• Screenshot: Sheila writes “This is a successful lock acquisition from the other day at the TRANSITION_FROM_ETMX state. The drive to EY L2 is large at the microseism frequency, not enough to saturate the 20 bit DAC though. When we complete the transition to ETMX, the drive is not as large. The EY + IX actuator is supposed to be a copy of the low noise EX actuator, but perhaps there is some difference in the filters. I will look for that now. In recent locklosses it looks like this L2 drive is even larger, probably because the ground is moving more.”
• This seems to support the hypothesis that something is slightly wrong, causing instabilities that would otherwise be invisible and unnoticed if not for the extremely high microseism and wind.
• What might be slightly wrong?
  • There are filter differences between the so-called same filters (EY vs. EX L1)
  • Sheila added a new filter (screenshot 2), which did successfully stop the counts from getting as high as they did in EY. Screenshot 3 comparison
  • Sheila noticed that the EX L3 Cal lines were erroneously on during the transition, when it should have been off
  • Further investigation by Ryan S found that L1, L2 and L3 Cal line were all on, incorrectly.
  • Ryan S investigated this and found that it was wrongly accepted into SDF and therefore, when PREP_FOR_LOCKING turns it off, SDF_REVERT turns it back on. It should only be turned on during TURN_ON_CALIBRATION_LINES.
  • We think these two issues may be the cause and we'll see during the next lock acquisition if they are
• Ryan S is investigating further.

TITLE: 01/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 9mph 5min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.65 μm/s
QUICK SUMMARY:

Troubleshooting is continuing for the locklosses at the TRANSITION_FROM_ETMX state. Wind and secondary microseism are still very high.

Closes FAMIS 26225

Laser Status:
    NPRO output power is 1.819W (nominal ~2W)
    AMP1 output power is 67.9W (nominal ~70W)
    AMP2 output power is 137.5W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN

PMC:
    It has been locked 30 days, 23 hr 15 minutes
    Reflected power = 16.9W
    Transmitted power = 109.4W
    PowerSum = 126.3W

FSS:
    It has been locked for 0 days 0 hr and 30 min
    TPD[V] = 0.9213V

ISS:
    The diffracted power is around 2.2%
    Last saturation event was 0 days 1 hours and 43 minutes ago

Possible Issues: None

Closes FAMIS 26485

T240 Centering

2024-01-06 13:24:12.266253

There are 11 T240 proof masses out of range ( > 0.3 [V] )!
ITMX T240 1 DOF X/U = -0.891 [V]
ITMX T240 1 DOF Y/V = 0.448 [V]
ITMX T240 1 DOF Z/W = 0.545 [V]
ITMX T240 2 DOF Y/V = 0.354 [V]
ITMX T240 2 DOF Z/W = 0.387 [V]
ITMX T240 3 DOF X/U = -0.913 [V]
ITMY T240 3 DOF X/U = -0.394 [V]
ITMY T240 3 DOF Z/W = -1.336 [V]
BS T240 3 DOF Z/W = -0.353 [V]
HAM8 1 DOF X/U = -0.34 [V]
HAM8 1 DOF Z/W = -0.526 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = 0.04 [V]
ETMX T240 1 DOF Y/V = 0.032 [V]
ETMX T240 1 DOF Z/W = 0.018 [V]
ETMX T240 2 DOF X/U = 0.014 [V]
ETMX T240 2 DOF Y/V = -0.009 [V]
ETMX T240 2 DOF Z/W = 0.07 [V]
ETMX T240 3 DOF X/U = 0.043 [V]
ETMX T240 3 DOF Y/V = -0.036 [V]
ETMX T240 3 DOF Z/W = 0.049 [V]
ETMY T240 1 DOF X/U = 0.147 [V]
ETMY T240 1 DOF Y/V = 0.191 [V]
ETMY T240 1 DOF Z/W = 0.259 [V]
ETMY T240 2 DOF X/U = -0.007 [V]
ETMY T240 2 DOF Y/V = 0.231 [V]
ETMY T240 2 DOF Z/W = 0.116 [V]
ETMY T240 3 DOF X/U = 0.281 [V]
ETMY T240 3 DOF Y/V = 0.182 [V]
ETMY T240 3 DOF Z/W = 0.204 [V]
ITMX T240 2 DOF X/U = 0.256 [V]
ITMX T240 3 DOF Y/V = 0.27 [V]
ITMX T240 3 DOF Z/W = 0.208 [V]
ITMY T240 1 DOF X/U = 0.188 [V]
ITMY T240 1 DOF Y/V = 0.169 [V]
ITMY T240 1 DOF Z/W = 0.058 [V]
ITMY T240 2 DOF X/U = 0.111 [V]
ITMY T240 2 DOF Y/V = 0.252 [V]
ITMY T240 2 DOF Z/W = 0.199 [V]
ITMY T240 3 DOF Y/V = 0.151 [V]
BS T240 1 DOF X/U = -0.097 [V]
BS T240 1 DOF Y/V = -0.281 [V]
BS T240 1 DOF Z/W = 0.19 [V]
BS T240 2 DOF X/U = 0.018 [V]
BS T240 2 DOF Y/V = 0.103 [V]
BS T240 2 DOF Z/W = -0.039 [V]
BS T240 3 DOF X/U = -0.09 [V]
BS T240 3 DOF Y/V = -0.262 [V]
HAM8 1 DOF Y/V = -0.291 [V]

Assessment complete.

STS Centering

2024-01-06 13:28:25.261111
There are 2 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -4.117 [V]
STS EY DOF Z/W = 2.8 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.551 [V]
STS A DOF Y/V = -0.899 [V]
STS A DOF Z/W = -0.466 [V]
STS B DOF X/U = 0.462 [V]
STS B DOF Y/V = 0.913 [V]
STS B DOF Z/W = -0.433 [V]
STS C DOF X/U = -0.613 [V]
STS C DOF Y/V = 0.798 [V]
STS C DOF Z/W = 0.393 [V]
STS EX DOF X/U = -0.151 [V]
STS EX DOF Y/V = 0.032 [V]
STS EX DOF Z/W = 0.079 [V]
STS EY DOF Y/V = 0.19 [V]
STS FC DOF X/U = 0.269 [V]
STS FC DOF Y/V = -0.931 [V]
STS FC DOF Z/W = 0.725 [V]

Assessment complete.

IFO is LOCKING but losing lock at TRANSITION_FROM_ETMX (probably due to high wind/microseism)

MX VEA Temperature Investigation:

• Alarm shows that H0:MX_VEA_202D_DEGF temperature has gone too low, triggering an alarm
• Further investigation by Dave and I, trending all 4 sensors shows that two of the thermometers increased in temperature, while the other 2 decreased (one of these gave the alarm)
• The fluctuation of the temperature seems to be erratic and only started 18 hours ago, around the same time the wind picked up
• Further trending with the MX wind sensor shows that the fluctuations followed the erratic wind pattern
• This indicates that there is a draft making its way through the room with the 4 sensors, increasing the temp on one side and decreasing it on the other side.
• This may mean that a door is open and needs to be closed.
• The wind is dying down and the alarm is no longer active, but this should be physically investigated.
• The wind went back up past 25mph and then the alarm came on again - this is further evidence for the “something is open/ajar” guess.
• Screenshot below

GraceDB Query Failures 

• GraceDB Query Failures flashing 5 or 6 times since the start of the shift
• Checked in with LLO and they are experiencing the same thing, implying that this is a server problem and not an H1 problem.

Transition from EX Failures - The Lockloss causing state (alog 75215)

• After losing lock at EX, managed to make it back to the same place (alog 75214)
• Jenne came online to investigate what would be causing these locklosses
• We are pausing at LOWNOISE_ASC (18:37 UTC) until we have an idea of what’s happening. We were able to stay here for a while.
• Another lockloss happened when Jenne was trying to troubleshoot the faulty guardian state. We lost lock the moment we tried running the EX transition part of the code (alog 75218)
• Still troubleshooting but not coming up with any reason other than this state being more susceptible to high microseism and wind.
• Jenne’s Idea: Perhaps the Low Noise coil driver is putting us in a more delicate state so what if we skip it temporarily? We tried this and it caused another Lockloss (alog 75222)
• I will attempt to get through this state again but if the issue keeps happening, I will put H1 in DOWN. 
  • This is because the wind is forecasted to go up even more until the evening when it is supposed to die down (by EVE shift)

Minor:

• Nuc24 Reboot by Dave (Was working in the control room but not externally)
• Nuc33 FCIR Trans B Cameras Bluescreened - Collectively agreed (Jenne, Dave, Ibrahim) that they can be fixed at a later date (not high priority)
• OpenEye Crashed and had to be restarted

Notes to self on transition from etmx locklosses we've been having recently (6 Jan 2024), and other lockloss investigations.

POP18 looks quite wobbly.  Do we have clipping again?  Haven't really checked anything w.r.t. this, so perhaps something to come back to.

A few weeks ago, we'd been having trouble with ALSX noticing that it was locked, so we renormalized it's PD around noon Pacific on 13 Dec 2023.  The next lock, it already looked like the renorm was unneccessary, ALSX trans was about 1.15.  Stayed at that level over many locks for many days. 

Dec 20th 02:15 UTC, first time I see ALSX trans getting to 1.2. After that, pretty consistently at that level. So, probably not related to this week's troubles?

The first lock this morning that was using guardian automatically lost lock during the state Transition_from_etmx, but the lockloss was about 21.5 sec after the swap away from ETMX.  So, I thought we should be able to do the swap and sit there, if the locklosses were due to something else like the ETMX bias ramping, or something (not that I know why that should cause a lockloss).

Ibrahim was able to stop and wait for 10+ mins at LOWNOISE_ASC.  We ran LOWNOISE_COIL_DRIVERS with guardian, then in the guardian shell did all the prep stuff in Transition_from_etmx, then as a block did lines 5086-5089 to actually do the swap.  Lost lock during the 5 sec ramp swapping away from ETMX, so this didn't work. 

Checking some ETMY filters (since Sheila et al checked ETMX yesterday).  Looks like h1susetmy foton file was last loaded in April 2023 (according to the date of the last _install.txt file in the filter archive), so before O4 began, so really there should be no possiblitiy of changes there, and indeed I don't find any changes.

* ETMY L2 Lock only filter used is FM7 "Qprime", has not changed shape in a long time (just picked a few older foton files out of the filter_archive). 

* ETMY L1 Lock only filter used is FM10 "UIMcomp", also not changed in a long time.

* ETMY L3 lock filters FM5, FM8, FM9, FM10.  But these are all flat to pretty high freq, before they are notches / stopbands, so no effect really on our low freq stability issues.  Also, no changes.

h1susitmx foton also last loaded April 2023, before O4, so shouldn't be any changes there.

Lockloss around 1388535878 (6 Jan 00:24:20 UTC) was about 2 mins after the swap to ETMY and ITMX.

Lockloss around 1388533427 (5 Jan 23:43:29 UTC) was only about 10 sec after the swap.

Seems like we're just quite delicate and susceptible to wind + microseism after this swap?

In case we were running into lack of control authority due to being in the lownoise coil driver state, we tried (without much hope, but trying anyway) to skip lownoise_coil_drivers and go straight from lownoise_asc to transition_from_etmx.  We lasted a little more than 5 sec past the swap, but not much more than that, so this isn't an avenue toward different diagnosis path. 

The wind is supposed to start getting better after 4pm today, so I wonder if the best path forward is to just wait out the environment?  I wonder if (as Ibrahim has said) this level of microseism plus wind is just untenable for us to reacquire lock with.

Sat Jan 06 10:08:28 2024 INFO: Fill completed in 8min 22secs

Lockloss at TRANSITION_FROM_ETMX. Despite microseism and wind, the lock acquisition was smooth until this point.

The last 8 Locklosses have been from this state.

TITLE: 01/06 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: None (Cancelled OWL)
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 25mph Gusts, 18mph 5min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.77 μm/s
QUICK SUMMARY:

IFO is down due to high microseism - will attempt to lock (but will go back to down if there's no chance)

MX Temperature is showing a red alert on the alarm handler - will investigate

TITLE: 01/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Microseism/Wind
SHIFT SUMMARY: Challenging evening fighting excess ground motion from wind/microseism and consistent locklosses after reaching full power during lock acquisition.

• H1 relocking following lockloss during TRANSITION_FROM_ETMX
• 00:58 Reached CHECK_VIOLINS_BEFORE_POWERUP, staying here and damping violins at 2W DC until wind calms
• 05:28 Wind actually got worse (gusts up to 40mph at times), then slightly better; trying now to powering up
• 05:43 Lockloss from MAX_POWER (state had finished two minutes prior)
• 06:14 While relocking, had poor alignment (CHECK_MICH_FRINGES ran twice), starting initial alignment
• 06:38 Alignment finished, starting lock acquisition
• 07:13 Lockloss from TRANSITION_FROM_ETMX, same place as seen earlier today
• 07:52 Lockloss from LOWNOISE_COIL_DRIVERS (one state sooner than seen earlier today)
  • Acquisition was going smoothly until this point, violins were elevated but not extremely

H1 is currently DOWN until environmental conditions improve and/or further investigation into the instability seen after powering up.