OPS Day Shift Summary

TITLE: 12/09 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is DOWN due to Earthquake

H1 was only able to lock for 36 minutes today (lockloss alog) in between earthquakes. It's been a very quakey day with:

• 3 6+ mag EQs (all 3 in the aleutian islands in Alaska.
• 8 5+ mag EQs
• 6 4+ mag EQs

We tired locking after the first 6.3 magnitude EQ died down only to have another 6.3 magnitude EQ 3 hrs later (and then another 6.3 10 mins after that). I've attached a screenshot of our microseism. The secondary microseism is also over the 90% threshold, which makes us more succeptible to lower mag EQs. We've been in useism and earthquake mode all day essentially.
 

LOG:

None

Ops Eve Shift Start

TITLE: 12/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM_EARTHQUAKE
    Wind: 6mph Gusts, 3mph 3min avg
    Primary useism: 1.11 μm/s
    Secondary useism: 0.86 μm/s
QUICK SUMMARY:

Currently in DOWN due to an earthquake in Japan. Microsseism is still pretty high so we were having a hard time getting back up anyway.

Suspension watchdog trips during the large earthquake

Back in March (76269) Jeff and I had updated all the suspension watchdogs (besides OFIS, OPOS, and HXDS since those were already up to date) to use better blrms filtering and to be output into um. We set the suspension watchdog thresholds to values between 100 and 300 µm, but these values were set arbitrarily since there was no way to previously see how far the stages move during different scenarios. We had upped a few of the thresholds after having some suspensions trip when they probably shouldn't have, and this is a continuation of that.
During the large earthquake that hit us on December 5th, 2024 18:46 UTC, all ISI watchdogs tripped as well as some of the stages on several suspensions. After a cursory look, all suspensions that tripped only had either the bottom or bottom+penultimate stage trip, meaning that with the exception of the single suspensions, the others' M1 stage damping should have stayed on.

We wanted to go through and check whether the trips may have just been because of the movement from the ISIs tripping. If that is the case, we want to raise the suspension watchdog thresholds for those stages so that these suspensions don't trip every single time their ISI trips, especially if the amount that they are moving is still not very large.

Suspension stages that tripped:

Triples:
- MC3 M3
- PR3 M2, M3
- SRM M2, M3
- SR2 M2, M3

Singles:
- IM1 M1
- OFI M1
- TMSX M1

MC3 (M3) (ndscope1)

When the earthquake hit and we lost lock, all stages were moving due to the earthquake, but once HAM2 ISI tripped 5 seconds after the lockloss, the rate at which the OSEMs were moving quickly accelerated, so the excess motion looks to mainly be due to the ISI trip (ndscope2).

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	86	150 (unchanged)
M2	150	136	175
M3	150	159	200

 

PR3 (M2, M3) (ndscope3)

Looks to be the same issue as with MC3.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	72	150 (unchanged)
M2	150	162	200
M3	150	151	200

 

SRM (M2, M3) (ndscope4)

Once again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM5 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	84	150 (unchanged)
M2	150	165	200
M3	150	174	225

 

 SR2 (M2, M3) (ndscope5)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM4 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	102	150 (unchanged)
M2	150	182	225
M3	150	171	225

 

IM1 (M1) (ndscope6)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM2 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	175	225

 

OFI (M1) (ndscope7)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM5 saturated and the ISI watchdogs tripped.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	150	209	250

 

TMSX (M1) (ndscope8)

This one seems a bit questionable - it looks like some of the OSEMs were already moving quite a bit before the ISI tripped, and there isn't as much of a clear place where they started moving more once the ISI had tripped(ndscope9). I will still be raising the suspension trip threshold for this one just because it doesn't need to be raised very much and is within a reasonable range.

Stage	Original WD threshold	Max BLRMS reached after lockloss	New WD threshold
M1	100	185	225

OPS Day Midshift Update

While the microseism has gone down a small bit, locking has still been difficult due to EQs.

First, there were two back to back EQs (mag 4s and 5s) that killed a potential lock at near NLN (LASER_NOISE_SUPPRESSION)

Then, we got to NLN and were OBSERVING for 36 mins but another two back to back (mag 4s and 5s) caused a lockloss.

Finally, just now while relocking, another two higher 6.5 mag and 6.3 mag EQs from Japan are coming through.

All of this is made much worse due to the already over 1 um/s secondary microseism.

As such, I will bring H1 to DOWN and ENVIRONMENT and wait until the Earth rings down a bit.

PSL Weekly Report - Weekly FAMIS 26329

Closes FAMIS 26329, Last checked in alog 81539

Laser Status:
    NPRO output power is 1.849W
    AMP1 output power is 70.27W
    AMP2 output power is 137.2W
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 11 days, 16 hr 6 minutes
    Reflected power = 22.82W
    Transmitted power = 105.8W
    PowerSum = 128.7W

FSS:
    It has been locked for 0 days 0 hr and 21 min
    TPD[V] = 0.8091V

ISS:
    The diffracted power is around 3.4%
    Last saturation event was 0 days 0 hours and 21 minutes ago

Possible Issues: None reported

Lockloss 18:54 UTC

Lockloss 36 mins into NLN due to 2 back to back EQs (4.5 and 5.8) that happened during very high microseism. Relocking now.

Sun CP1 Fill

Sun Dec 08 10:12:22 2024 Fill completed in 12min 19secs

Note to VAC: Texts were sent for this fill, but no emails

alarms issue 01:00 Sunday 08dec2024

Jonathan, Dave:

The alarms service on cdslogin stopped reporting around 1am this morning. Symptoms are status file was not being updated (caused alarm block on CDS Overview MEDM to turn PURPLE) and the report file was not being updated. Presumably no alarms would have been sent from this time onwards.

At 08:10 I restarted the alarms.service on cdslogin. A new report file was created but not written to, the /tmp/alarm_status.txt file was not changed (still frozen at 01:00) but I did get a startup text. Then 14 minutes later the files started being written. I raised a test alarm and got a text, but no email.

At 09:38 after not getting a keepalive email at 09:00 or any SSH login emails I rebooted cdslogin. Same behavior as 08:10; report file created not written, tmp file not created, startup text sent successfully. After 14 minutes alarms starts running, writes to file system, test alarms are texted but no emails at all.

Jonathan is going to check on bepex.

OPS Day Shift Start

TITLE: 12/08 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 20mph Gusts, 16mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.71 μm/s
QUICK SUMMARY:

IFO is in ENVIRONMENT and LOCKING. IFO stayed in down all last night due to high winds and microseism.

Since last night, the microseiem has leveled off and even gone down a bit. The wind hasn't changed much. Attempting to lock to see where we get to.

Ops Eve Shift End

TITLE: 12/08 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: TJ/Oli
SHIFT SUMMARY: Currently unlocked. We've been sitting in DOWN for the past hour while the secondary microseism stays really high. We also currently have a smallish earthquake coming through. I will set the ifo to start trying to relock again.

Earlier while trying to relock, we were having issues with the ASLX crystal frequency. When this is a consistant issue we have to fix it by going out to the end station to adjust the crystal temperature. I trended the ALSX channels alongside the EX VEA temperatures and it looks like a couple of the temperatures went down, section D down by almost one degree F, right around when we started having crystal frequency issues. The wind also was blowing into the VEA, which we know since the dust counts were high then. I believe it's possible that the wind was cooling down the air in the part of the VEA near where we have the ALS box and changing the temperature of the crystal enough to affect the beatnote. I only have this one screenshot right now (ndscope), but I had trended back a few months and seen a possible correlation between when we get into the CHECK_CRYSTAL_FREQUENCY state for ALSX, the temperature inside the EX VEA, and the dust counts indicating wind entering the VEA. It's hard to know for sure especially because the air/wind outside is now much colder than it was a couple months ago, but it would be interesting to know the location of the D section and look for these correlations more closely. tagging ISC
LOG:

22:15 started an initial alignment
22:40 initial alignment done, relocking
    - ALSX beatnote issue - CHECK_CRYSTAL_FREQUENCY
        - toggled force/no force
        - finally caught with no force
    - ALSX beatnote issue again
        - toggled force/no force and enable/disable
    00:01 Put ifo in DOWN since we can't get past DRMI due to the high microseism
    00:29 tried relocking
    01:06 back to DOWN
02:38 Trying relocking again                                                                                                                              

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
23:03	PEM	Robert	LVEA	YES	Finish setting up for Monday	23:48

OPS Day Shift Summary

TITLE: 12/08 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is DOWN due to MICROSEISM/ENVIRONMENT since 22:09 UTC

First 6-7 hrs of shift were very calm and we were in OBSERVING for majority of the time.

• We lost lock during a particularly high gust of wind and as the microseism went beyond 2 um/s and haven't been able to lock since. LL alog 81667.
• Successfully ran through today's planned calibration sweep. alog 81666.

The plan is to stay in DOWN and intermittently try to lock but the last few attempts have resulted in 6 pre-DRMI LLs with 0 DRMI acquisitions. Overall, microseism is just very high.

LOG:                                                                                                                                                                       

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
23:03	PEM	Robert	LVEA	YES	Finish setting up for Monday	23:48
23:03	HAZ	LVEA IS LASER HAZARD	LVEA	YES	LVEA IS LASER HAZARD	06:09

Ops Eve Shift Start

TITLE: 12/08 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT_USEISM
    Wind: 15mph Gusts, 9mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.78 μm/s
QUICK SUMMARY:

Currently in DOWN and trying to wait out the microseism a bit. Thankfully wind has gone back down

Investigating SRM M3 WD Trips During Initial Alignment Part 2

Trying to gather more info about the nature of these M3 SRM WD trips in light of OWL Ops being called (at least twice in recent weeks) to press one button.

• Since 10/16/2024, there have been 10 instances where M3 WD trips during SRC align during initial alignment, which have caused a total loss of 1hr and 49 mins of locking time. While this isn't a lot, the trending WD trips for SRM shows that this is a more recent occurrence.
• Out of the 31 total M3 SRM WD trips since start of O4b (and not counting vent), 12 happened in the last 5 weeks, with 10 of these being during initial alignment (i.e. this type of trip).
• Since the rise of this issue, we have gone through a total of 112 initial alignments, meaning it's occurred in 9% of initial alignments. Doesn't sound like a lot but if guardian has an initial alignment during an OWL shift, this suggests a 1/10 chance of being woken up just to press "untrip".
• Sometimes, M3 is too rung up in this state that untripping takes 20 mins but other times it only takes 1-2 mins.
• FRS Ticket

Relevant Alogs:

Part 1 of this investigation: 81476

Tony OWL Call: alog 81661

TJ OWL Call: alog 81455

TJ OWL Call: alog 81325

It's mentioned in some more OPS alogs but no new info.

Next Steps:

• The possible cause per a last convo with TJ was that guardian thinks that SRC is aligned when actually it is locked to the wrong mode, causing the SRM controls to go awry when re-misaligning during initial alignment - so just rehash that conversation when possible.
• Find out what proportion of initial alignments this has happened to since 10/16.

Lockloss

Lockloss @ 12/07 22:09 UTC. Possibly due to a gust of  wind since at EY it had jumped up from lower 20s to almost 30mph the same minute of the lockloss? A possible contributer could also be the secondary microseism - it has been quickly rising over the last several hours and is now up to 2 um/s.

SRM Watchdog trip

TITLE: 12/07 Owl Shift: 0600-1530 UTC (2200-0730 PST), all times posted in UTC
STATE of H1: Aligning
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 5mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

IFO stuck in initial alignment because SRM watchdog H1:SUS-SRM_M3_WDMON_STATE trip.
Watch dog tripped while we were in Initial alignmnet not before, and was not due to ground motion.

I logged in discovered the trip. Reset the watchdog and reselected myself for Remote OWL notifications.