02:28 UTC lockloss, IMC tag

00:44 UTC lockloss, ASC_AS_A and IMC lost lock ~10 ms apart.

TITLE: 11/02 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing at 157Mpc and have been Locked for over an hour. We relocked many times today and relocking tended to take a bit long due to many locklosses, but for the most part was hands-off. The only things that required help were 1) during both initial alignments I ran today, when we got to aligning PRC, the IMC unlocked and could not relock - every time it would relock the FSS would glitch and unlock. To solve this, I waited until the PSL_FSS guardian was in READY_FOR_MC_LOCK, and then I paused it. Then, I waited for the IMC to lock and it would lock fine that next try, and once it had been locked for a few seconds, I unpaused the PSL_FSS guardian (tagging OpsInfo) and 2) one of the times when we were in GREEN_ARMS, ALS_XARM was stuck in CHECK_CRYSTAL_FREQUENCY, but taking it to UNLOCKED and then ETM_TMS_WFS_OFFLOADED fixed it right away.
LOG:

14:30 Observing and locked for over 3 hours
14:47 Superevent S241102cy

15:17 Lockloss
    15:17 Started an initial alignment
        IMC lost lock when trying to align PRC
            - Every time it would relock, the FSS would oscillate and cause the IMC to lose lock again & FSS would lose lock
            - I paused the FSS guardian once the FSS was locked and then unpaused once the IMC was good and locked
    15:40 Initial alignment done, relocking
    16:12 Lockloss from MOVE_SPOTS
17:04 NOMINAL_LOW_NOISE    
17:05 Observing

17:13 Lockloss
18:05 NOMINAL_LOW_NOISE
18:08 Observing

18:38 Lockloss
    18:38 Started an initial alignment
        - Same issue as before - during PRC the mode cleaner unlocks and cannot relock due to the FSS glitching and unlocking.
            Steps that have worked for me: (tagging Ops)
            1) Wait until the PSL_FSS guardian is in READY_FOR_MC_LOCK
            2) Once it is, pause PSL_FSS
            3) Wait for IMC to lock
            4) Once it has, unpause PSL_FSS
        - For some reason, even once the FSS and IMC were locked and we were aligning PRC, the ISS diffracted power was jumping all over the place
    18:58 Initial alignment done, relocking
        - ALS_XARM CHECK_CRYSTAL_FREQUENCY issue - toggling Force/No Force did not work, so I changed ALS_XARM to AUTO and then took it to UNLOCKED. I selected ETM_TMS_WFS_OFFLOADED and it locked immediately before I could troubleshoot further
19:48 NOMINAL_LOW_NOISE
19:52 Observing

20:08 Lockloss
22:15 NOMINAL_LOW_NOISE
22:15 Observing                                                                                                                                                                                   

TITLE: 11/02 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 8mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.24 μm/s
QUICK SUMMARY:

• We've been locked for a little over an hour
• Secondary microseism is descreasing, wind is low

Using the darm_integral_compare.py script from the NoiseBudget repo (NoiseBudget/aligoNB/production_code/H1/darm_integral_compare.py) as a starting point, I made a version that is simplified and easy to run for when our range is low and we want to compare range vs frequency with a previous time.

It takes two starting times, supplied by the user, and for each time, it grabs the DARM data between the start time and an end time of starttime+5400 seconds (1.5 hours). Using this data it calculates the inspiral range integrand and returns two plots(pdf) - one showing the range integrand plotted against frequency for each set of data(png1), and then the second plot just shows DARM for each set of data, along with a trace showing the cumulative difference in range between the two sets as a function of frequency(png2). These are saved both as pngs and in a pdf in the script's folder.

This script can be found at gitcommon/ops_tools/rangeComparison/range_compare.py. To run it you just need to supply the gps times for the two sets of time that you want to compare, although there is also an optional argument you can if you want the length of data taken to be different than the default 5400 seconds. The command used to generate the PDF and PNGs attached to this alog was as follows: python3 range_compare.py --span 5000

Over the past 2 days I've been seeing lots of IMC locklosses right after a main lockloss while ISC_LOCK is in READY. This has been happening 10+ times between relocks. The IMC guardian seems to lock for a split second in its "AQUIRE" state then as soon as it moves on to "BOOST" it loses it. It looks the the MC2 transmision maybe isn't stable, then the BOOST state turns on MC2 lock filters and ramps gains. We also see the bottom stage of MC2 SUS saturate during this. Some are from FSS oscillations but not all.

Lockloss @ 11/02 20:08 UTC after 20 minutes locked

Lockloss @ 11/02 18:38 UTC after only 33 minutes locked

Sat Nov 02 10:10:55 2024 INFO: Fill completed in 10min 51secs

Janos, Dave:

VACSTAT version 1.10

Following several recent false positive VACSTAT alarms due to noise in a single gauge (usually BSC3 PT132) we have made a change to the code to mitigate this.

The new alarm logic is:

• If a single gauge trips, increase the sensitivity of the other gauges. Then if at a later time a second or more gauges trip, send the cell phone alarm to the vacuum team.
• If multiple gauges trip at the same time, send the cell phone alarms to the vacuum team.

The alarm system was modified to alert the vacuum team if the channel H1:CDS-VAC_STAT_GLITCH_NUM_GAUGES is 2 or greater.

My cell phone alarm will continue to sound on a single gauge trip, because tripped gauges are latched out from the sytem and we need to reset any false positives to restore the nominal functionality.

The new code was installed Friday afternoon (15:40 01nov2024 PDT).

Design reasoning:

In the event of an in-chamber gas release the local gauge will see a large signal. Gas migration to neighboring chambers will result in smaller signals there. Increasing their trip sensitivities will increase the chance of seeing a multi-chamber event.

Lockloss @ 11/02 17:13UTC after only 9 minutes locked

Lockloss @ 11/02 15:17 UTC after 4 hours locked

TITLE: 11/02 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 16mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.26 μm/s
QUICK SUMMARY:

In Observing and have been Locked for 3hours 15mins. Secondary microseism is coming down and wind is just over 20mph