Mon Nov 11 10:17:12 2024 INFO: Fill completed in 17min 9secs
Jordan confirmed a good fill.
After discussions with control room team: Jason, Ryan S, Sheila, Tony, Vicky, Elenna, Camilla
Conclusions: The NPRO glitches aren't new. Something changed to make us not be able to survive them as well in lock. The NRPO was swapped so isn't the issue. Looking the timing of these "IMC" locklosses, they are caused by something in the IMC or upstream 81155.
Tagging OpsInfo: Premade templates for looking at locklosses are in /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/PSL_lockloss_search_fast_channels.yaml and will come up with command 'lockloss select' or 'lockloss show 1415370858'.
Updated list of things that have been checked above and attache a plot where I've split the IMC only tagged locklosses (orange) from those tagged IMC and FSS_OSCIALTION (yellow). The non IMC ones (blue) are the normal lock losses and (mostly) only once we saw before September.
Sheila and I went out to the floor to plug in the cable to run frequency noise injections (AO2).
I switched us to REFL B (79037) only to run the injections. I first ran the frequency noise injection template. Then, I disenaged the 10 Hz pole and engaged the digital gain to run the ISS injections (low, medium, and high frequency). I was fiddling with the IMC WFS pitch injection template to get it to higher frequency for a jitter injection when we lost lock.
Next step is to at least use the intensity and frequency measurements to determine how the noises are coupling to DARM (would be nice to have jitter too, but we'll see).
Commit 464b5256 in aligoNB git repo.
These results show that above 3 kHz, some of the intensity noise couples through frequency noise. There is no measurable intensity noise coupling through frequency noise at mid or low frequency.
The frequency noise projection and coupling in this plot are divided by rt2 to account for the fact that frequency noise is measured on one sensor here, but usually is controlled on two REFL sensors. This may not be correct for high frequency (roughly above 4 kHz), where CARM is gain limited.
code lives in /ligo/gitcommon/NoiseBudget/simplepyNB/Laser_noise_projections.ipynb
TITLE: 11/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 4mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.33 μm/s
QUICK SUMMARY:
Ryan C was called last night at 9:24 UTC.
When I walked in Sheila was in the Team speak chat, so maybe she was helping out with getting it locked first thing in the morning.
NLN reached at 15:37 UTC
Observing reached at 15 :40 UTC
H1 called for assistance at 09:29 UTC after the initial alignment timer expired from the IMC unlocking an not relocking, we were in PREP_FOR_SRY. I took us to down and the IMC locked and I then went back into locking where DRMI locked very quickly and looked pretty decent.
lockloss at 10:07 UTC from LOWNOISE_ESD_ETMX, doesn't look like an IMC lockloss
11:03 observing
TITLE: 11/11 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
IFO is LOCKING at CHECK_AS_SHUTTERS
Overall a smooth shift where we were locked for two thirds of the shift until the PSL glitch most likely took us down. Since then, the glitch has been happening without much gap for relocking. We've had gaps of 20 or so mins without glitching but that's been the maximum so far since LL. Lockloss alog 81181.
Other than that, we dropped out of OBSERVING very briefly due to a TCSY CO2 Laser SDF diff that fixed itself.
LOG:
None
Closes FAMIS 26016. Last checked in alog 80912.
Changes from last check:
Lockloss seemingly PSL caused judging from IMC and ASC-A signals losing lock within 10ms (attached). About a 700ms after the glitching started, we lost lock. Interestingly, IMC is not glitching for minutes post-LL as it usually does, with IMC locking and staying locked in the new acquisition.
Spoke way way (way) too soon. The glitch has been happening since the lockloss, which is much longer than usual. I've been staying in DOWN and attempting to lock if we pass 5 mins without glitching, which happened twice only to lose lock to it around DRMI or FIND_IR...
Watching the channels closely for when it stops so I can actually lock. The screenshot shows that the longest time without glitching since LL is that first 19 mins.
TITLE: 11/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
The morning was a little shakey, but we persevered through the problematic IMC glitches to get to a lock by 22:35 UTC
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 21:41 | SAF | Laser | LVEA | YES | LVEA is laser HAZARD | 08:21 |
| 18:30 | PEM | Robert | LVEA | Yes | Getting more spools of cable | 18:39 |
| 18:48 | PEM | Robert | CER | No | Checking cables | 20:10 |
| 21:29 | BBSS | Ibrahim | Staging Building | N | BBSS inventory | 00:29 |
TITLE: 11/11 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 4mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.37 μm/s
QUICK SUMMARY:
IFO is in NLN and OBSERVING as of 22:35 UTC
Recovered from a massive 6.8 EQ. Nothing else of note.
TITLE: 11/10 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 8mph Gusts, 4mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.40 μm/s
QUICK SUMMARY:
After the 6.8M Earthquake passed I started relocking
Unfortunately:
IMC PSL-FSS _FAST MON Issues present while trying to relock.
Currently Relocking , ISC_LOCK @ Power_25W
6 days ago there was an increase in H0:VAC-EX_FAN1_570_2_ACC_INCHSEC
Both H0:VAC-MR_FAN1_170_1 & 2 ACC_INCHSEC saw an increase in noise in the last 3 days.
Sun Nov 10 10:11:37 2024 INFO: Fill completed in 11min 33secs
https://earthquake.usgs.gov/earthquakes/eventpage/us7000nr0v/
ISI Watchdog trips:
BS: Stage 1 & 2
ITMX 1 & 2
ITMY 1 & 2
ETMX 1 & 2
ETMY 1 & 2
HAM 8
HAM 7
This lockloss seems to have the AOM driver monitor glitching right before the lockloss(ndscope), similar to what I had noticed in the two toher locklosses from this past weekend(81037).
03:45UTC Observing
In 81073 (Tuesday 05 November) Jason/Ryan S gave the ISS more range to stop it running out of range and going unstable. But on Tuesday evening, Oli saw this type of lockloss with again 81089. Not sure if we've seen it since then.
The channel to check in the ~second before the lockloss is H1:PSL-ISS_AOM_DRIVER_MON_OUT_DQ, I added this to the lockloss trends shown by the 'lockloss' command line tool via /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/psl_fss_imc.yaml
I checked all the "IMC" tagged locklosses since Wednesday 6th and didn't see any more of these.
This happened before we fixed the NPRO mode hopping problem, which we did on Wednesday, Nov 6th. Not seeing any more of these locklosses since lends credence to our suspicion that the ISS was not responsible for these locklosses, the NPRO mode hopping was (NPRO mode hops cause the power output from the PMC to drop, the ISS sees this drop and does its job by lowering the diffracted power %, once the diffracted power % hits 0 the ISS unlocks and/or goes unstable).
Reran all O4b locklosses on the locklost tool with the most up to date code, which includes the "IMC" tag.
I used the lockloss tool data since the emergency OFI vent (back online ~23rd August) until today and did some excel wizardry (attached) to make the two attached plots, showing the number of locklosses per day tagged with IMC and without tag IMC. I made this plots for just locklosses from Observing and all NLN locklosses (can include commissioning/maintenance times), including:
Attaching plots from zooming in on a few locklosses:
| Time | Tags | Zoom Plot | Wide Plot |
| 2024-10-31 12:12:26.993164 UTC (IMC) | OBSERVE IMC REFINED | annotated plot | plot |
| 2024-10-30 12:16:33.504883 UTC (IMC) | OBSERVE IMC REFINED | annotated plot | plot |
| 2024-10-30 16:09:21.548828 UTC (Normal) | OBSERVE REFINED OMC_DCPD | N/A | annotated plot |
The 10-31 zoom plot notes the framerates of the channels: ASC, REFL and NPRO_PWR are 2kHz and GS13 is 4kHz, the others are 16kHz.
Since September 18th we've had 21 locklosses from NLN tagged as FSS_OSCILLATION, of these 20 also had the IMC tag. Since September 12th, we've had 49 locklosses from NLN tagged IMC, so roughly 40% of these IMC locklosses have the FSS oscillation tag, since the NPRO was swapped we don't have any tagged with FSS_OSCILLATION.
(Reminder, the FSS_OSCILLATION tag is an old tag, intended for a different problem, but it tags times where the FSS fast mon goes above a treshold.)
Updated plot attached of NLN locklosses tagged with and without the IMC tag.
