TITLE: 12/30 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.27 μm/s
QUICK SUMMARY:

Currently relocking and at ENGAGE_SOFT_LOOPS. Secondary microseism has gone way down over the last half a day and wind is low.

(CoreyG, DanielS)

At 1925utc, H1 dropped out of Observing due to the Squeezer.  After a couple minutes, could see that the MANAGER & LO_LR nodes were cycling and not getting anywhere. 

Wiki First

At this point, I opened the Troubleshooting SQZ wiki's "Can't Get To FREQ_DEP_SQZ" section.  From this, I took MANAGER to NO SQUEEZING and then opened up all the SQZ Nodes. 

At this point I notice the OPO_LR node which had been green is actually also cycling (but not as fast as the MANAGER & LO_LR.  And OPO_LR was also  showing a User Message, BUT it displayed too quick that I had to take a screenshot really quick to catch it.  This node would hang out at the NOMINAL state (LOCKED CLF DUAL), drop out from this, display a message(s) for 1sec and then return to nominal state....all within a minute or so.  There were actually 2 messages, they were to the best I could see (also attached):

1. pump fiber rej power in ham7 high, nominal 35e-3, align fiber pol on sqzt0
2. Disabling pump iss after 10 lockloss counter.  Going to through LOCKED CLF_DUAL_NO_ISS to turn on agai

At this point, I started diving into medms for anything related to the shy messages above.  "pump fiber rej power" gave me the most info for where to look, but at this point, I also wanted to do an alog search.

Then Alog Search

I searched OPO_LR, where I focused on an alog from (1) RyanS which referenced a Tony alog and (2) RyanC.  These three alogs narrowed my focus and had me looking at:

1. HAM7 Fiber Rejected Power signal
2. SQZT0 waveplate

The Rejected power has been trending for the last month+ up to a max value around 3.0 (where it starts from about 0.03).  I wasn't totally sure I had the scenario right, so at this point I reached out to Daniel (main contact for SQZ on the Call List).

Phoning Daniel For Help

After giving a summary the best I could, Daniel logged in and started looking.  Some of the main points he mentioned:

• Fiber polarization
• Did not have enough green power
• Running out of range
• Crystal may be degrading
• Need more Green power!

With all that said,m he was able to get the LO_LR node to its nominal state, but he mentioned we could also fix the Fiber Rejected Power (atleast until the new year or about a week!).  So he started to work on this.  And this ended up being what was done by RyanS in his alog.  See attached for image showing which waveplate/picomotor he adjusted on SQZT0 along with the HAM7 Fiber Rejected Power signal he watched & reduced.  See attached for zoom in of the rejected power signal adjusment he made. 

Daniel mentioned there will probably be a nead to make adjustments out on the table, but this change can get us through the holiday break.

Return to OBSERVING at 2056utc

nds1:

The /var file system on h1daqnds1 (the default NDS) became 100% full yesterday. This is discovered by the daily central syslogs for this machine increasing from roughly 3GB per day to 19GB yesterday.

For a short term fix, I first removed the syslog.3.gz and syslog.4.gz files to make enough room so I could compress syslog.1 and daemon.1 This reduced the disk down to 50% usage.

alarms:

To "green up" the alarms system which had been permanently RED since Thursday due to HAM6's gauge going offline I temporarily removed the PT110 channnels from the alarms system and restarted at 10:33 Sun 29dec2024.

As a place holder to remind me to return these channels back to alarms when the gauge is fixed, FRS32957 will remain open.

Channels removed:

H0:VAC-LX_Y0_PT110_MOD1_PRESS_TORR

H0:VAC-LX_Y0_PT110_MOD1_PRESS_TORR_ERROR

Sun Dec 29 10:16:20 2024 INFO: Fill completed in 16min 16secs

TITLE: 12/29 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 13mph Gusts, 7mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

H1 has been locked for 10+hrs!  Secondary microseism is mostly under the 95th percentile line, and winds are only around 10mph (with light rain on the drive in).

TITLE: 12/29 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing and have been Locked for almost 2 hours. Just spent an hour working with Vicky to get sqz back up and running because it had unlocked and I couldn't get it to relock. SQZ_LO guardian had the message "Beam diverter is open but no 3MHz on OMC, could be an alignment problem". 

With Vicky, we first tried resetting the SQZ angle by manualing to the RESEST_SQZ_ASC_FDS guardian state in SQZ_Manager, but that didn't work. The problem was that the FC couldn't even lock because the LO couldn't lock. What had happened is that the "squeezer angle servo ran away, and because squeezer ASC alignment setpoints are coupled to the SQZ angle, so this time when SQZ ANGLE runs away it also drags the SQZ ASC away. This time it dragged the SQZ ASC away enough that the LO loop could not lock (because nearly no signal on SQZ-OMC_RF3 beatnote (RF3 was around -22, normally around -9)" (Vicky, 2024).

What we did that ended up working (and the ideal way to do this in the future):

     - Reset SQZ angle (CLF_REFL_RF6 to 185)

     - Go to NO_SQZ

     - Trend ZM4 and ZM6 driftmon and adjust sliders to bring SUS back to last time in FDS (last lock)

     - Go back to FDS

Vicky's notes on sqz todo's regarding this problem:

     - "Improve the notification given (currently "Beam diverter is open but no 3MHz on OMC, could be an alignment problem") when SQZ_LO can't lock to alert ops to check SQZ ASC or check ZM4/6 LOCK filters, etc, to make this problem more diagnosable"

     - "SQZ MANAGER needs a more robust way to clear SQZ ASC now that integrators are in the ZM4/6 LOCK filter banks (see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=80534). Oli tried to RESET_SQZ_ASC_FDS but that didn't clear the suspension lock filter banks."

     - "We should get at more robust guardian state to clear SQZ ASC in the lock filter integerators, and corresponding OPS instructions for when OMC RF3 LO locking fails + SQZ ANGLE is railed (I found it at 235 ish, it should be like 185-195). "

* All quotes by Vicky

On a different note, I figured out why the LOCKLOSS_SHUTTER_CHECK guardian had been stuck stalled and had been requiring us to change the states between high and low arm power - it's because it is NOT normally managed by ISC_LOCK like I had thought. Yesterday after I had MANUAL'd it, thinking that it had previously been managed by ISC_LOCK, I set it to be managed by ISC_LOCK. It actually is normally just on AUTO. I set it to AUTO and had no issues with it during the locking sequence.

LOG:

00:05 Started trying to relock
    - Sitting in OMC_WHITENING for an hour damping violins
02:45 NOMINAL_LOW_NOISE
02:45 Observing

03:51 Lockloss https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=82037
05:23 NOMINAL_LOW_NOISE
05:25 Observing

05:46 Out of Observing due to squeezer unlock
    - SQZ_LO gives message "Beam diverter is open but no 3MHz on OMC, could be an alignment problem"
    - With help from Vicky, we figured it out
07:02 Back to Observing

Lockloss @ 12/29 03:51UTC

(Lockloss tool still hasn't been restarted since the cluster issues on Thursday)

TITLE: 12/28 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Wind
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Winds put us over the tipping point for not making it possible to Lock H1 (with the continued horrendous µseism); so marking Observatory Mode at WIND state.  For the latter part of the shift, took H1 to IDLE (to help prevent ringing up violins).

With H1 down, attempt a full Site Walkthru to the End Stations (only made it to EY).
LOG:

• 1527 Started to look into why ISCLOCK was stuck at CHECK AS SHUTTERS
  • 1535 MANUAL to PREP TR CARM & then return to AUTO to NLN
  • 1538 LOCKLOSS (start of many locklosses assume due to very high winds and very high microseism)
• 1545 Sent Dave text regarding Guardian Log "leap second" issue
  • 1600 chatted with Dave and he started work
• 1906-2017 H1 to IDLE due to winds
  • tried locking at 2017, but still lots of ALS locklosses, back to IDLE at 2030
  • 2247 Tried locking again, but only for a few minutes due to locklosses
• 2126-2240 Site WalkThroughs VEA Edition: This was done since H1 was down due to high winds + high microseism (Corey)
  • 2126-2154 LVEA + FCES 
    • Hard hat caution sign was on the floor so I removed that.
  • 2155-2240 EY & attempt of EX (too many tumbleweeds after MX)
    • At EY the Wind Fence leftmost wall panel has come undun (see attached).
    • Also noticed some sort of plumbing outside near the entrance to EY which looks like something has been trying to dig a hole around it (see attached).
    • Could not drive beyond MX due to impassable "X2" road (see attached).
    • While driving around the site, also knocked some ice off the CP1 exhaust (but this required pitchforking some tumbleweeds out of the way) (see attached).
• 2248-2340 Site Walk Through Common Spaces Edition (Oli)

TITLE: 12/29 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Wind
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 20mph Gusts, 16mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.72 μm/s
QUICK SUMMARY:

Currently trying to relock and at FIND_IR. We were able to catch DRMI for a few seconds before losing it. It's still very windy and the microseism is still super high but I wanted to try relocking for a bit while the wind is slightly lower at 25 (still gusting up to 30+)

Closes FAMIS#26450, last checked 81585

BRS Driftmon

Auxiliary BRS Channels

At the beginning of December, Ryan noted (81585) that the BRSY temperature had been drifting up a bit, but it looks like it has now settled at a temperature just below 23°C. Everything else is looking normal and within range.

Closes FAMIS#26350, last checked 81932

Corner Station Fans (attachment1)
- All fans are looking normal and within range.

Outbuilding Fans (attachment2)
- All fans are looking normal and are within range.

Shift Update: 

Lots of tumbleweeds rolling by outside & Tons of ALS locklosses as the winds are back at 30+mph (this morning there were only two short lulls in the winds...where H1 was able to try MAX POWER on one of these windows).  Because of the double whammy of high microseism AND high winds (which can be heard in the Control Room, will take H1 to IDLE to help prevent ringing up violins.  Will check back hourly (time for lunch).

Laser Status:
    NPRO output power is 1.849W
    AMP1 output power is 69.87W
    AMP2 output power is 137.0W
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 11 days, 0 hr 28 minutes
    Reflected power = 24.6W
    Transmitted power = 104.2W
    PowerSum = 128.8W

FSS:
    It has been locked for 0 days 0 hr and 5 min
    TPD[V] = 0.7813V

ISS:
    The diffracted power is around 3.5%
    Last saturation event was 0 days 0 hours and 6 minutes ago
Possible Issues:
        PMC reflected power is hig

Sat Dec 28 10:15:41 2024 INFO: Fill completed in 15min 38secs

h1guardian1's /usr/share/zoneinfo/leap-seconds.list file expired 28 Dec 2024 resulting in guardlog errors.

I updated the tzdata package on h1guardian1 which resolved the problem. I verified the CDS workstations already have updated leap-second files (they expire 28 Jun 2025).

(Dave Barker, Fil Clara, Jenne Driggers, Corey Gray, Gerardo Moreno, plus those who signed work permit and whom I phoned last night)

Last Night Recap:
Last night at 959pmPT, there was an audible alarm for the HAM6 vacuum gauge (PT110).  Gerardo went out to check to see if it was dead.  He phoned Fil and they did a power cycle of the gauge to see if this would improve the situation, but it did not.  After this, H1 was taken to IDLE for the night (OWL shift cancelled).  (Should have taken Observatory Mode to CORRECTIVE MAINTENANCE at this point, but only remembered to do this this morning.)

Work Permit 12260 generated.  Fil arrived on-site around 9amPT (17utc).

This Morning's Summary of Fil's Work:

The HAM6 Interlock (for operations of the Fast Shutter + PZT) is now Using the BSC3 Interlock (or VAC gauge from BSC3 since it was decided to BYPASS the HAM6 VAC gauge which failed last night [alog 82005]).  This means that the BSC3 Interlock is no longer being used for the ITM ESDs (which sounds like it's fine because Jenne/Fil said we don't use the ESDs for the ITMs).

Fil noted that he confirmed that the Fast Shutter Chassis in the LVEA is OPERATIONAL.  Fil then left the site after giving me this summary.

As far as a test of this new HAM6 Interlock configuration, I intentionally ran a test of the AS Port Protection system (sitemap/SYS/ AS Port Protection....this was my first time doing this).  After clicking the RUN (test) button, after a few seconds received an "OK" for the (1) Test & (2) Fault.  After this test and the signed-off Work Permit, we are OK to GO for High Power locking.

Currently running an Initial Alignment (note:  ISC_LOCK initially went into ERROR [red box] when I selected Initial Alignment (screenshot of error message attached).  LOAD-ing ISC_LOCK (suggested in the Log) cleared this up.

ALSO: 

• Attaching a screenshot of the AS Port Protection medm
• CDS Alarm box on the CDS Overview remains (related to the lingering issue of the failed PT110 gauge).