FAMIS 31066

Starting about 4 days ago, the steady increase in PMC REFL power leveled out and appears to have turned around over the past day or so. I don't see anything here that immediately points to a reason for this. Otherwise, no major events on these trends.

Tue Dec 31 10:12:37 2024 ALERT: Fill done (errors) in 12min 33se

TC-A min was -65C which is exactly the trip temp, so the fill did not automatically complete. After about a minute at this level I manually cancelled the fill at 10:12.

TC-A started very high today, at +19C, so I anticipated there could be an issue with the fill. I have increased the trip temp to -60C for tomorrow's fill.

TITLE: 12/31 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 3mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.18 μm/s
QUICK SUMMARY: H1 lost lock at 14:43 UTC, ending this lock stretch at just over 29 hours. The online lockloss tool has not been updating since 12/26 at 20:14 UTC (5 days ago), so I don't know of a cause yet, but I plan to look into it. H1 is currently automatically relocking and up to RESONANCE.

• Wind low, secondary microseism trending down to around the 50th percentile
• LAB2 DM still in error
• All other systems nominal

TITLE: 12/31 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Currently Observing at 160 Mpc and have been Locked for over 20.5 hours. A few minutes ago we got another superevent candidate! Looking nice in H1! The evening has been quiet with nothing needing to be touched by me!

I did notice a notification on SQZ_OPO_LR that says "CMB EXC is on", and looking at the channel it checks, it looks like SQZ-OPO_SERVO_COMEXCEN went from 0 to 1 around 09:57 UTC on 12/30, which is a bit after we got locked last night and while we were having the SQZ issues. There are no excitations listed on the test point or CDS Overview screens, so I am wondering if this is some error? Tagging SQZ
LOG:

05:41 Superevent S241231bg

TITLE: 12/31 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY: Forgot to put in my start of shift alog, so here it is as a midshift update. Currently Observing and have been locked for 17.5 hours. Quiet evening with nothing to report.

TITLE: 12/30 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Locked entire shift (of a 15+hr lock) with (2) drops from Observing mostly for Squeezer (and also due to PI24).  H1 was also mostly with No Squeezing since last night, and then Sheila and Daniel were able to get squeezing back around lunchtime (alog)---just in time for a superevent in the afternoon!

Environmentally we are looking good with lower microseism and low winds!
LOG:

• 1556 H1 Knocked out of OBSERVING due to rung up PI24 (it would redamp its self within a minute)
  • While out of observing looked at Squeezer issues for about 20min, but then returned to Observing with NO Squeezing, to return to this when Sheila would be available.
• 1805 Tyler phoned about heads up of King Soft arriving today (est 1930utc) for taking a water sample at woodshop.  They have a card to get through gate and will be non-invasive for observing
• 1900 Noticed that I couldn't hear Verbal.  Restarted verbal to no avail and then noticed the speaker icon on the desktop was muted.  Not sure why/how that happened, so it was unmuted.
• 1955 Out of Observing for Sheila to look at the Squeezer.  She noted the issue with the ISS (as RyanC mentioned last night) and also mentioned the SQZT0 AOM.  She made adjustments with a SQZT0 half waveplate which is just downstream of that AOM (same one Daniel adjusted to address high rejected power issue from yesterday).  She could only look at this briefly and then messaged Daniel about what she saw.  She said if her change did not work for getting the MANAGER to FDS_READY_IFO, then we can return to Observing with NO SQUEEZING.  After a few minutes, Daniel hopped online after getting Sheila's message.  
  • After a while, Daniel was able to get the SQZ to a good spot
• 2043 Back to Observing
• 2336 Superevent S241230ev
• 2230--2350 Site WalkThrough:  Corner Area + Road-To-MX (corey---with a pitchfork for tumbleweeds!)

We seem to be running into repeated problems where the green power is not sufficient to get to the nominal OPO green transmitted power.

Today the SHG output was particular low, so I adjusted the input pointing as well as lowered the temperature to 35.0C (from 35.5C). This gave us about 10% more green power, but still not enough! I then further increased the green power by adjusting the waveplate in front of the cube.

The rejected power (H1:SQZ-SHG_REJECTED_DC_POWERMON) is currently showing 1.8mW, down from around 4.7mW when I started. There is a little bit further to go, but we may have to realigning the green fiber coupler on the table. And if this doesn't work, we probably need to coonsider moving the OPO crystal.

Link to Shift Summary 

• The duty cycle for this week was 30.4%
• H1 was out of observing for almost the whole day of 12/24 and 12/27. See aLOG(81960), aLOG(81964), aLOG(82012) and aLOG(82016)
• Some days, the lock losses were due to earthquakes, wind and ground motion, some due to HAM issues and some have unknown causes.
• Average range was between 155-165 Mpc for the days that were majorily in locked mode.
• The week had pretty high wind and ground motion for the majority part of multiple days. Some days seismic activity was also high contributing to mutiple locklosses. 
• The Corner BSC 10 has constant high amplitude low frequency noise multiple days.
• H1 EX and H1 EY Acoustic PEM somedays has long periods of low frequency noise in the spectrograms. 
• Some HVETO round winner channels were new while some repeated for multiple days. See the shift summary for more details. 
• FScan plots were not available for 3 days of the week. While, the lines remained under 600 for the rest of the days. 

Mon Dec 30 10:16:30 2024 INFO: Fill completed in 16min 27secs

TCmins [-91C, -45C] OAT [+6C, 42F]. TC-A started positive, hence multiple decades in log plot.

1556utc H1 was taken out of observing due to PI24 ringing up (it was damped out within a minute), BUT....

Since we were out of Observing (and I had already gotten a workspace ready to work on the Squeezer), I wanted to try a "simple" Squeezer check (basically what Oli did with taking the Manager to DOWN and going back to FDS.

I thought I was close to FDS, because all of the sub-nodes went to NOMINAL states and then I believe the Manager would get to FDS, but then go to NO SQUEEZING.....Then I realized all their NOMINAL states were set for NO SQUEEZING! 

So I worked on transitioning all 8-nodes Nominal states for SQUEEZING, to work on squeezing, but the issue I ran into was that the OPO_LR node had trouble getting to LOCKED_CLF_DUAL due to "USERMSG 0: No pump ISS means injecting bad squeezing. Going DOWN."

--->This sounds like yet another different SQZ issue.  I did not have issues with the SQZ_FC as far as I could tell (also did not get to see the "leap second" issue RyanC noted last night).

At this point, I was stumped, but got a mattermost from Sheila saying she could help in a bit. So I switched all SQZ node Nominal states for NO Squeezing to get more Observing in the mean time.

SIDE NOTE:  The buisiness for transitioning SQZ Nominal States is not quick or trivial (atleast for my slow self).  While  transitioning nodes for Observing WITH Squeezing, I ran into a LOADING error for the OPO_LR node.  Eventually found that it was a typo in the wiki (for editing all the nodes I copied the state from the wiki into the node python files.  I discovered the wiki had a typo [or the state had a slightly different name int he past].  For OPO_LR, the wiki said to make the Nominal -> "LOCK_CLF_DUAL".  Eventually figured out from the log, that this name was wrong it is "LOCKED_CLS_DUAL".  At any rate, I updated wiki.  And now have more Nominal node experience...I think this is only my 2nd or 3rd time going through the buisiness of "editing nominal states" for observing.

TITLE: 12/30 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

H1's been locked for 6.25hrs (with no squeezing); we have triple coincidence now, but debating whether to drop out of Observing for Squeezer investigations---It feels like we've had a variety of squeezer issues for the last 36 hours through multiple differnt locks.  Microseism continues it's downward trend with it down to touching the 50% percentile line.  Winds have been at or under 10mph for most of the last 24hrs.

(Although Oli spotted the male elk while they left the site, he was not spotted this morning.)

SQZer not locked, SQZ_FC gave a notification about Leap data being expired in its log. I tried bringing FC1 and FC2 back to where they were the last time we locked the SQZer to to avail, following the wiki I tried to walk through the locking process but the ISS keeps losing lock and couldn't stay locked, I keep seeing the 10 locklosses notification. I wasn't able to get the ISS to lock, I ended up taking us to obs without SQZing, I had to accept a few SDFS.

10:31 UTC Observing

TITLE: 12/30 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing at 155Mpc and have been Locked for 2.5 hours. One lockloss during my shift and relocking was entirely hands-off, but we had trouble getting into Observing because the squeezer wouldn't lock.

The FC would go to FC_ASC_ON, then less than a second later the OPO would drop from LOCKED_CLF_DUAL to LOCKED_CLF_DUAL_NO_ISS and then ENGAGE_PUMP_ISS, and either at the same time or immediately following this, the LO guardian would unlock from the OMC, and the filter cavity would go to DOWN a few seconds later. This happened over and over until I took SQZ_MANAGER to DOWN and then back to FDS, at which point we were able to get squeezing and go into Observing. I originally thought this issue was related to the problem Corey and Daniel worked on earlier, but the values for SQZ-SHG_FIBR_REJECTED_DC_POWERMON were still nice and low to where Daniel had adjusted them to, and I don't think there was an issue with the OPO temperature since we were able to lock with the ISS with the setpoint at 80uW?? Tagging SQZ
 

LOG:

00:30 Relocking and at MOVE_SPOTS
00:46 NOMINAL_LOW_NOISE
00:48 Observing

02:19 Lockloss
03:34 NOMINAL_LOW_NOISE
    - SQZ was having issues locking
    - Got it figured out by going to NO_SQUEEZING and then back to FDS
03:49 Observing

Lockloss @ 12/30 02:19 UTC

TITLE: 12/29 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

H1 was locked over 17hrs until it had a LockLoss (whose cause is unclear--LockLoss tool is down).  Before the lockloss there was a drop from Observing due to the Squeezer.

Microseism continues it's downward trend of the last 24hrs (and winds were not an issue today).
LOG:

• 1557 Seismic to CALM (from USEISM since 12/25 ~18utc)
• 1620-1715 Bull elk sighting on-site with LExC camera. 
• 1730--1808 tumbleweed clearing w/ pitchfork on X-arm for LN2 delivery on NYE (corey)
• 1820 Chatted w/ Dave about his work to free space for nds1 & to also make the CDS Alarm box on CDS Overview GREEN (it had been RED due to the HAM6 PT110 failure a few days ago).
• 1925-2056 Out of Observing Due to SQZ_OPO_LR / halfwave plate adjustment (phoned Daniel to help)
• 2107 Saw the PCal trace on DARM FOM have a big glitch!
• 2229 LOCKLOSS (after 17hrs!)
  • ALS Scan Alignment completed, but DRMI/PRMI didn't look good.
  • 2249-2305 Manual Initial Alignment (since green arms went through Scan Alignment
  • ALSx had 5 locklosses (looked as if ASC drove it off)
  • Eventhough ran a Manual Alignment, PRMI & DRMI were a bit tough to acquire (even with lower microseism and low winds), but currently on our way to NLN
• 2250-2300 EQ Mode activated  (M5.1 Ethiopia EQ)  L1 loses lock around this time.
• 2323 Site Walkthru: Corner Area (Oli)

TITLE: 12/30 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.27 μm/s
QUICK SUMMARY:

Currently relocking and at ENGAGE_SOFT_LOOPS. Secondary microseism has gone way down over the last half a day and wind is low.