Over the past 2 days I've been seeing lots of IMC locklosses right after a main lockloss while ISC_LOCK is in READY. This has been happening 10+ times between relocks. The IMC guardian seems to lock for a split second in its "AQUIRE" state then as soon as it moves on to "BOOST" it loses it. It looks the the MC2 transmision maybe isn't stable, then the BOOST state turns on MC2 lock filters and ramps gains. We also see the bottom stage of MC2 SUS saturate during this. Some are from FSS oscillations but not all.

Lockloss @ 11/02 20:08 UTC after 20 minutes locked

Lockloss @ 11/02 18:38 UTC after only 33 minutes locked

Sat Nov 02 10:10:55 2024 INFO: Fill completed in 10min 51secs

Janos, Dave:

VACSTAT version 1.10

Following several recent false positive VACSTAT alarms due to noise in a single gauge (usually BSC3 PT132) we have made a change to the code to mitigate this.

The new alarm logic is:

• If a single gauge trips, increase the sensitivity of the other gauges. Then if at a later time a second or more gauges trip, send the cell phone alarm to the vacuum team.
• If multiple gauges trip at the same time, send the cell phone alarms to the vacuum team.

The alarm system was modified to alert the vacuum team if the channel H1:CDS-VAC_STAT_GLITCH_NUM_GAUGES is 2 or greater.

My cell phone alarm will continue to sound on a single gauge trip, because tripped gauges are latched out from the sytem and we need to reset any false positives to restore the nominal functionality.

The new code was installed Friday afternoon (15:40 01nov2024 PDT).

Design reasoning:

In the event of an in-chamber gas release the local gauge will see a large signal. Gas migration to neighboring chambers will result in smaller signals there. Increasing their trip sensitivities will increase the chance of seeing a multi-chamber event.

Lockloss @ 11/02 17:13UTC after only 9 minutes locked

Lockloss @ 11/02 15:17 UTC after 4 hours locked

TITLE: 11/02 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 16mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.26 μm/s
QUICK SUMMARY:

In Observing and have been Locked for 3hours 15mins. Secondary microseism is coming down and wind is just over 20mph

TITLE: 11/02 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: TJ
SHIFT SUMMARY: The IMC has been very unhappy and the FSS has been glitching frequently this shift. HLID picket fence seismometer has also been glitching out this afternoon (Tagging SEI), after the locklosses the IMC would lose it again and relock after reaching READY this kept happening this evening. We just locked PRMI as of 05:00 UTC.

The left display of the Operator WS is flickering, unplugging and plugged it back in fixes the issue but only for like an hour.

LOG: No log

Lock#1:

• 23:29 UTC ENGAGE_DRMI lockloss
• 23:42 UTC CARM_OFFSET_REDUCTION lockloss #3, IMC and ASC_AS_A lost lock at the same time, zoomed in
  • 00:25 UTC Observing
• 00:52 UTC lockloss 31 minute lock :(
  • Lots of FSS glitches after this lockloss, the IMC kept locking then the FSS would glitch and kill it in ready. I put ISC_LOCK in down till it locked, this happened 6 times before it would hold it, then it lost it as soon as I moved on from DOWN, I watched the FSS_FAST channel grow until a glitch killed it seconds later. Then the 8th time was the charm and it held.
  • Despite these inital issues relocking was fine after the IMC actually held, automated relock in just over an hour
  • 01:58 UTC Observing
• 04:17 UTC lockloss 2:24 lock
  • The IMC locked and relocked a few times again in READY
  • DRMI doesn't look as good now

04:17 UTC, the IMC lost lock less than 100ms after ASC_AS_A, no IMC tag though and the FSS didn't really look glitchy.

Closes FAMIS26509, last checked in alog79454. It looks as expected, the lack of switching in the recent past was the vent I believe.

00:53 UTC

IMC and ASC_AS lost it at the same time, looks like some kind of fast ringup in the FSS_FAST channel. FSS_OSCILLATION and IMC tag on the lockloss tool.

TITLE: 11/01 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Ran a handful of tests today to try to narrow down further the FSS glitches that we've been seeing (80993, 80990 ). We recently started relocking with only minor changes, hopefully they will help.
LOG:

• Lost lock at Transition_From_ETMX, looks to be another one of the PSL glitches, losing light at AS_A before the IMC unlocks (attachment 1).
• Made it to the state before low noise when we lost lock again. We will pause here to run some tests for the laser glitching.
• 1534 - IMC locked, ISC_LOCK in IDLE at 2W
• 1644 - IMC brought to Offline
• 1842 - ISS unlocked
• 20:25:04 - Sheila reduced the common mode gain by 3dB
• 2201 - Started locking again
• Lock loss at carm reduction. Possible laser caused (attachment 3).
• Scratch that last note, had a lock loss in the same place, starting initial alignment.
• 2257 Initial alignment

LVEA zone 1B has been slowly getting warmer over the past 2 days. Not a worry yet, but something we should keep an eye on.

TITLE: 11/01 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 19mph Gusts, 10mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.39 μm/s
QUICK SUMMARY:

• Secondary microseism is decreasing
• Wind is slightly elevated
• We're back trying to lock after the PSL/IMC investigations this afternoon
  • Back to back CARM_OFFSET_REDUCTION locklosses
  • Initial alignment starting at ~23:00UTC