Sat Nov 02 10:10:55 2024 INFO: Fill completed in 10min 51secs

Janos, Dave:

VACSTAT version 1.10

Following several recent false positive VACSTAT alarms due to noise in a single gauge (usually BSC3 PT132) we have made a change to the code to mitigate this.

The new alarm logic is:

• If a single gauge trips, increase the sensitivity of the other gauges. Then if at a later time a second or more gauges trip, send the cell phone alarm to the vacuum team.
• If multiple gauges trip at the same time, send the cell phone alarms to the vacuum team.

The alarm system was modified to alert the vacuum team if the channel H1:CDS-VAC_STAT_GLITCH_NUM_GAUGES is 2 or greater.

My cell phone alarm will continue to sound on a single gauge trip, because tripped gauges are latched out from the sytem and we need to reset any false positives to restore the nominal functionality.

The new code was installed Friday afternoon (15:40 01nov2024 PDT).

Design reasoning:

In the event of an in-chamber gas release the local gauge will see a large signal. Gas migration to neighboring chambers will result in smaller signals there. Increasing their trip sensitivities will increase the chance of seeing a multi-chamber event.

Lockloss @ 11/02 17:13UTC after only 9 minutes locked

Lockloss @ 11/02 15:17 UTC after 4 hours locked

TITLE: 11/02 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 16mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.26 μm/s
QUICK SUMMARY:

In Observing and have been Locked for 3hours 15mins. Secondary microseism is coming down and wind is just over 20mph

TITLE: 11/02 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: TJ
SHIFT SUMMARY: The IMC has been very unhappy and the FSS has been glitching frequently this shift. HLID picket fence seismometer has also been glitching out this afternoon (Tagging SEI), after the locklosses the IMC would lose it again and relock after reaching READY this kept happening this evening. We just locked PRMI as of 05:00 UTC.

The left display of the Operator WS is flickering, unplugging and plugged it back in fixes the issue but only for like an hour.

LOG: No log

Lock#1:

• 23:29 UTC ENGAGE_DRMI lockloss
• 23:42 UTC CARM_OFFSET_REDUCTION lockloss #3, IMC and ASC_AS_A lost lock at the same time, zoomed in
  • 00:25 UTC Observing
• 00:52 UTC lockloss 31 minute lock :(
  • Lots of FSS glitches after this lockloss, the IMC kept locking then the FSS would glitch and kill it in ready. I put ISC_LOCK in down till it locked, this happened 6 times before it would hold it, then it lost it as soon as I moved on from DOWN, I watched the FSS_FAST channel grow until a glitch killed it seconds later. Then the 8th time was the charm and it held.
  • Despite these inital issues relocking was fine after the IMC actually held, automated relock in just over an hour
  • 01:58 UTC Observing
• 04:17 UTC lockloss 2:24 lock
  • The IMC locked and relocked a few times again in READY
  • DRMI doesn't look as good now

04:17 UTC, the IMC lost lock less than 100ms after ASC_AS_A, no IMC tag though and the FSS didn't really look glitchy.

Closes FAMIS26509, last checked in alog79454. It looks as expected, the lack of switching in the recent past was the vent I believe.

00:53 UTC

IMC and ASC_AS lost it at the same time, looks like some kind of fast ringup in the FSS_FAST channel. FSS_OSCILLATION and IMC tag on the lockloss tool.

TITLE: 11/01 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Ran a handful of tests today to try to narrow down further the FSS glitches that we've been seeing (80993, 80990 ). We recently started relocking with only minor changes, hopefully they will help.
LOG:

• Lost lock at Transition_From_ETMX, looks to be another one of the PSL glitches, losing light at AS_A before the IMC unlocks (attachment 1).
• Made it to the state before low noise when we lost lock again. We will pause here to run some tests for the laser glitching.
• 1534 - IMC locked, ISC_LOCK in IDLE at 2W
• 1644 - IMC brought to Offline
• 1842 - ISS unlocked
• 20:25:04 - Sheila reduced the common mode gain by 3dB
• 2201 - Started locking again
• Lock loss at carm reduction. Possible laser caused (attachment 3).
• Scratch that last note, had a lock loss in the same place, starting initial alignment.
• 2257 Initial alignment

LVEA zone 1B has been slowly getting warmer over the past 2 days. Not a worry yet, but something we should keep an eye on.

TITLE: 11/01 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 19mph Gusts, 10mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.39 μm/s
QUICK SUMMARY:

• Secondary microseism is decreasing
• Wind is slightly elevated
• We're back trying to lock after the PSL/IMC investigations this afternoon
  • Back to back CARM_OFFSET_REDUCTION locklosses
  • Initial alignment starting at ~23:00UTC

DQ Shift summary:

• LHO was observing on average 48% of the time through out the week, with Wednesday only 1.9% (Issue with damping the violin mode at the beginning of the day, not sure what caused the detector not locking in the later half of the day)
• There was an obvious trend in the BNS range at the beginning of the lock: low BNS range (140-150 Mpc) going up to ~160 Mpc starting on Thursday; correlated with elevated broadband high-frequency noise; no clear correlation with glitch rate
• There were periods (1 - 2 hours) of elevated broadband 10-15 Hz noise throughout the week, especially with the short observing segments.
• Daily Hveto with usual suspects, except for Thursday: H1:PEM-CS_ACC_PSL_PERISCOPE_X_DQ (100 - 300Hz glitches)
• Line feature most obvious on Friday with ~ 1050, ~1160, ~1200, and ~1270 lines on the spectrogram. Fscan line count stays around 650 throughout the week. 
• Sunday BNS range with extended periods staying around 150 Mpc, seems to have some correlation with some REFLAIR, VAC, PSL channels. Obvious broadband noise.
   

More daily details can be found at: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20241014

We measured the TTFSS OLG, I didn't save the data but will attach several photos here. There is a feature that looks like it could be unstable at 1.68MHz, there are about 3 untiy gain crossings around 500 kHz.  The FSS has a gain between the test points that mean that unity gain is at -10dB in this measurement. 

67253 Craig mentions that the FSS OLG was 350kHz, we'd need to reduce the gain by 3dB to get back to a 350 kHz ugf.

We lost lock at some higher states in the relocking process due to the laser glitching* issue that we have been having. At this point we decided to pause locking for now to run some tests that were discussed during a meeting yesterday. The current plan is to hold at 2W with the ISS, FSS, and IMC locked. If this configuration is found to be unstable or we see excessive glitching still, then we will start to remove just the ISS or FSS, etc to narrow down the problem.

During this time, other, approved, light maintenance activities can take place.

*The issue may not be the laser itself, but I'm unsure what else to call this at the moment.

Yesterday we went back to the higher range CARM configuration, in which the fast/IMC locklosses first appeared  80938.  Overnight we had two observing locklosses tagged with the IMC, neither of which had glitches visible in the FSS fast mon channel, 1414411965 (1st attached screenshot), and 1414381791. 

This morning I changed the gains by 6 dB in the other direction, so we further limit the range available to the frequency actuator. (CARM sliders and ndscope) Our microseism is around 5um/second right now, slightly above the 90th percentile, but there was no wind and no earthquakes this morning.  At times the IMC splitmon was reaching -8V, it's range is +/- 10V so this is about as far as we can go in this direction, and might be kind of marginal.

We've since had one lockloss 1414426971, this is tagged as commissioning because we were in commissioning time at the moment, but no one was actively doing anything at the time so this wasn't caused by commissioners.  There was a glitch in the FSS fast mon for this lockloss, although it was below the threshold for the tag to be applied, and around the time of that glitch the drive to the ISS AOM becomes noisy.  We were not saturating the IMC splitmon at the time of the lockloss so that doesn't seem to be the cause.