Reports until 00:27, Friday 15 September 2023
LHO General
ibrahim.abouelfettouh@LIGO.ORG - posted 00:27, Friday 15 September 2023 - last comment - 10:20, Friday 15 September 2023(72894)
OPS Eve Shift Summary

TITLE: 09/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 143Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 22:53 UTC

Continued Lockloss (alog 72875) Investigations (Continued from alog 72876)

The same type of lockloss happened today during TJ’s Day shift. Instead of continuing exactly where I left off by inspecting the EX saturations. I will briefly trend the lockloss select (as done yesterday and today with TJ with separate “this issue” locklosses. From the LSC lockloss scopes (Screenshot 1), we can clearly see that about 92ms before any of the LSC channels saw the lockloss, H1:LSC-DARM_IN1_DQ saw it first. From speaking with TJ the day earlier, this is a channel that goes back to the OMC DCPDs (if I recall correctly).

Before hunting the actuator down, I zoomed into the channel and saw that this channel’s bumpy behavior started building up at 4:45:47 (Screenshot 2) - a second before that lockloss. This second picture is just a zoom on the tiny hump seen in the first screenshot.

 

And unfortunately, there was not enough time to continue investigating but will be picking this up next week. We essentially found that there is one particular channel, related to OMC DCPDs that has a build-up followed by a violent kick that knocks everything from time to time, causing locklosses. What I would like to know/ask:

  1. Are these just causing glitches, whereby the most powerful “spasms” are the lockloss causing ones? Or do these species of glitches always cause locklosses?
  2. How is this related to EX saturating 1.5 seconds before? This didn’t happen in today’s lockloss, yet there was a similar preliminary kick in this channel prior to the lockloss.
  3. Which actuator/suspension is this one related to? What more can we find out by looking at it? This is probably easy to find out (and also the most important) - I just ran out of time.
  4. How long has this type of lockloss causing glitch been happening? This would involve just trending that channel with various lockloss times, but also trending it with various BLRMs glitch times to investigate the mutual exclusivity of these two events. Is there a correlation high enough between these events that can inform a causation? Do these things sometimes cause EX saturations (the really violent ones like yesterday’s) or is that just a coincidence?
  5. Is this nature of spasm build-up indicative of anything to do with electronics, software etc.? A question for the experts.

Most of these questions hit on the same few pieces of evidence we have (EX saturation - potential red herring, OMC Channel kick - a new area to investigate) and the BLRMs glitch incidence (the evidence that it wasn’t external).

Other:

3 GRB-Short Alarms

Many glitches but 0 lockloss causing ones

LOG:

Start Time System Name Location Lazer_Haz Task Time End
20:22 EPO Oregon Public Broadcasting Overpass N Setting up timelapse camera 20:59
Images attached to this report
Comments related to this report
camilla.compton@LIGO.ORG - 10:20, Friday 15 September 2023 (72896)

Tony, Oli, Camilla

Good lockloss investigations Ibrahim. The lockloss tool shows these ETMX glitches in the ~2 seconds before the lockloss in the "Saturations" and "Length-Pitch-Yaw" plots. I think ETMX moving would cause a DARM glitch (so the DARM BLRMs to increase) or vice versa, DARM changing would cause ETMX to try to follow.  Faster ETMX channels to look at would be H1:SUS-ETMX_L3_MASTER_OUT_UL_DQ, 16384Hz vs 16Hz. You can see the framerate of the channels using command 'chndump | grep H1:SUS-ETMX_L3_MASTER_OUT_' or simular...

Plot attached shows L1, L2, L3 of ETMX all see these fast noisy glitches but the OMC and DARM channels show a slower movement. Can this us tell us anything about the cause?

See similar glitches in:

Images attached to this comment