TITLE: 12/30 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
H1 Still locked at NOMINAL_LOW_NOISE and OBSERVING for 4 Hours and 45 min.
VPW Temps are falling back down.

LOG:
Doug the water Quality Vendor Went to the LSB, to test the water around 2
And Around that time I did a very through walkthough across the entire site including the End Stations.

TITLE: 12/30 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 8mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.64 μm/s 
QUICK SUMMARY:

• We've been been locked for 4:45, in Observing since 19:24UTC
• Microseism is still high but is trending down

TITLE: 12/29 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.59 μm/s
QUICK SUMMARY:

Lockloss 17:29 UTC
Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at ldas_admin_lho@ligo.caltech.edu to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.
Lockloss at FIND_IR 18:05 UTC
Lockloss at LOCKING_ALS
Lockloss at LOCKING_ALS
Lockloss at LOCKING_ALS

Started an Initial_Alignment 18:15 UTC
Locking restarted

Nominal Low noise reached at 19:20 UTC
A few Visitors came to the LEXC building and walked behind the LEXC building out of site of the cameras.
Observing Reached at 19:25 UTC
IFO Locked and OBSERVING for 3+ hours.

A Tyler's Water vendor Douglas stopped by the LSB to test the faucet Water.
Walkthrough:
I took that time to do a walk around. Everything at the LSB & LEXC was alright, but the Temps at the VPW were much higher than yesterday's.
The hottest location was 78 F and yesterday that location was 72F. Even the Coldest location had noticably higher temps. Tagging CDS.

Lockloss from Observing.
Lockloss near GPStime: 1387906209

Lockloss select tool cannot reach the lockloss page
Querying lockloss web...
Could not fetch events from web; searching guardian logs...

https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/
Leads me to the following error

"Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request."

There apears to a PI ring up that happened at that time. I'm not sure what the thresh hold for damping activation would be for this PI mode.





 

Fri Dec 29 10:07:24 2023 INFO: Fill completed in 7min 21secs

TCmins -138C,-129C

TITLE: 12/29 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 4mph 5min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.65 μm/s
QUICK SUMMARY:
H1 has been Locked and observing for 11 hours and 35 minutes!

New CDS_Overview indicator H1EDC error points to nuc26 having 12 channels with out a connection for some reason.

H1:CDS-MONITOR_NUC26_CPU_LOAD_PERCENT
H1:CDS-MONITOR_NUC26_CPU_COUNT
H1:CDS-MONITOR_NUC26_MEMORY_AVAIL_PERCENT
H1:CDS-MONITOR_NUC26_MEMORY_AVAIL_MB
H1:CDS-MONITOR_NUC26_PROCESSES
H1:CDS-MONITOR_NUC26_INET_CONNECTIONS
H1:CDS-MONITOR_NUC26_NET_TX_TOTAL_MBIT
H1:CDS-MONITOR_NUC26_NET_RX_TOTAL_MBIT
H1:CDS-MONITOR_NUC26_NET_TX_LO_MBIT
H1:CDS-MONITOR_NUC26_NET_RX_LO_MBIT
H1:CDS-MONITOR_NUC26_NET_RX_ENO1_MBIT
H1:CDS-MONITOR_NUC26_NET_TX_ENO1_MBIT

It seems like these are monitors for this particular computer.

TITLE: 12/29 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Currently Observing and have now been Locked for 3.5hours. Spent most of my shift waiting for the violins and DCPD values to go down to be able to lock. Like I said in my midshift update(75070), I don't know if I could have gone into NOMINAL_LOW_NOISE earlier than I did, since the DCPD_WINDOW value were so high, and going into NLN usually raises them further.
LOG:

00:00 Detector in MAX_POWER

01:40 I took us to LOWNOISE_ASC, and then continued stepping through states one at a time.
01:52 Reached OMC_WHITENING
    - Violins are still really high, and DCPD counts are more than twice as large(based on ndscope) as they typically are when in OMC_WHITENING.

02:48 Earthquake mode activated

02:57 Back to CALM

04:27 NOMINAL_LOW_NOISE
    - I was so sure that we would saturate or that the DCPD counts would at least jump way up but they didnt change at all?

04:28 Observing

Currently in OMC_WHITENING and have been here since 01:52UTC (so about 2hours). My main reasoning for not proceeding into NOMINAL_LOW_NOISE is that the OMC_DCPDs (according to {A, B}_WINDOW_{MIN, MAX} - I couldn't find the current ADC counts) are the largest they've been in at least the last ~230 days - see ndscope comparison between the DCPDs now versus four days ago when the DCPD counts were also very large - since the DCPDs jump when we get into NOMINAL_LOW_NOISE, I am worried about it causing us to lose lock. However, it looks like at its current rate, it won't get down to reasonable values until way past the end of my shift, so I'll wait a little longer before taking us into NOMINAL_LOW_NOISE and just hope that we don't saturate and lose lock!

TITLE: 12/29 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
I've been Damping Violins for a while now at max power.

Called Dave to help with the Leap second issue.Dave was able to determine that it was an issue with DEB 10 machines not getting the new LEAP file (see Daves alog.) NUC 5 is now giving us the MicroSeism information we want but DMT viewer is still in error.

Dave had me power cycle the Thermostats in H2 again and we confirmed that the air flow is now better.

LOG:
I went to H2 to restart Thermostats
And did a sweep of LEXC, LSB & VPW Temps ( VPW Temps range from 66 to 72 infront of the sever exhausts.)

TITLE: 12/28 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 5mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.82 μm/s
QUICK SUMMARY:

Detector is at MAX_POWER and Tony has been trying to damp the violin modes for a while because they're still really high. Microseism is very high but is maybe heading downwards, and wind is around 10mph.

Tony, Dave:

This afternoon Tony discovered a ndscope issue on nuc5, which we tracked down to its leap seconds file having expired today, 28th December 2023.

This is only an issue with debian10 machines, debian11/12 have been updated.

For now I have copied the /usr/share/zoneinfo/leapseconds file onto all the deb10 FOM machines, ndscope now runs with no errors. Next Tuesday we will schedule the package updates.

TITLE: 12/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Microseism
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 5min avg
    Primary useism: 0.13 μm/s
    Secondary useism: 0.99 μm/s
QUICK SUMMARY:
The Primary and Secondardy MicroSeism's are Elevated, and honestly not living up to their micro prefix much at all. Tagging SEI.

Locking attempts today:
16:05:11Z    LOCKING_ALS -> LOCKLOSS
16:07:17Z    LOCKING_ALS -> LOCKLOSS
started an Initial_Alignment

16:44:03Z    LOCKING_ALS -> LOCKLOSS
17:04:59Z    PREP_DC_READOUT_TRANSITION -> LOCKLOSS
17:07:10Z    LOCKING_ALS -> LOCKLOSS
17:09:15Z    LOCKING_ALS -> LOCKLOSS
17:13:04Z    LOCKING_ALS -> LOCKLOSS
17:15:47Z    LOCKING_ALS -> LOCKLOSS
17:22:16Z    LOCKING_ALS -> LOCKLOSS

Dave got onto TeamSpeak to ask me to restart the H2 Thermostats.
Paused the IFO in Check_Violins to damp at 2 Watts before trying to move up.

18:06:30Z    CHECK_VIOLINS_BEFORE_POWERUP -> LOCKLOSS

18:21:22Z    DRMI_LOCKED_CHECK_ASC -> LOCKLOSS
18:25:05Z    LOCKING_ALS -> LOCKLOSS

Paused the IFO in Check_Violins to damp at 2 Watts before trying to move up. (note: The 60W settings for ITMY modes 1-4 work well in 2W) 
19:53:48Z    TRANSITION_FROM_ETMX -> LOCKLOSS
19:57:36Z    LOCKING_ALS -> LOCKLOSS

Dave suggested that I restart NUC5 the Seismic Nuc, to see if DMT veiwer would work today.
And I got a new NUC5 error " Leap second data is expired. " and I can no longer run NDSCOPE on nuc5.
I have copied the file path for the ndscope that shows me ground motion, and I can run it on my workstation but ndscope nolonger works on NUC5 at all. Tagging CDS.

"Micro"Seism Screenshot
 

Thu Dec 28 10:06:42 2023 INFO: Fill completed in 6min 38secs

Gerardo fixed the TCs yesterday, their starting temps today were -16C,-10C. I reduced the trip temps for today's fill from -60C to -70C.

TCmins after the overfill were -110C,-103C.

TITLE: 12/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 0mph 5min avg
    Primary useism: 0.16 μm/s
    Secondary useism: 0.96 μm/s
QUICK SUMMARY:
H1 has been unlocked for 6 hours due to a large earthquake and an elevated uSeism.
Initial attempts to lock are failing at locking ALS.
The microSeism isn't as high as it was yesterday so perhaps we can get H1 locked today, But the Primary microSeism looks higher than it was yesteday.



.

H1 lost lock at around 9:30 UTC from a M6.5 earthquake from Japan and has been trying to relock since. Due to the continued very high microseism, it has not been very successful (the highest ISC_LOCK state reached being DRMI_TO_POP). Since the continued locking attempts seem to be ringing up violin modes as seen yesterday, I'm going to leave H1 at READY for now until the ground motion calms or the daytime operator wants to make an attempt at lock acquisition.

TITLE: 12/28 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:  Currently Observing and have been Locked for 4 hours. Relocking took more than half of my shift today just because the violins were so high and then due to needing to step through each ISC_LOCK state one at a time, and the BS camera issue (75060). Once we were in Observing everything has been calm. Secondary microseism is going down a bit, and wind was below 15mph throughout my entire shift.

While damping the violins at 2W, the ITMY Mode 5/6 setting of FM6 and FM10 gain = -0.1 did not work, but I was able to find a new setting that did work - FM6 and FM10 with gain = +0.1. This new setting was updated in the Violin Damping google sheet in the RF Damping (2W) column.

I also spoke with RRT Shifters Maurice van Putten and Maryam Aghaei-Abchouyeh about the violins and their relationship to ground motion and ability to lock at higher powers. This is what was said in the RRT MM that sums up the discussion:

"Discussion with Oli Patane (LHO Control Room) re H1 prevented from locking by anomalous microseismic activity and (width and strength) of violin modes. Detailed (tuning of) damping of violin modes does not seem to ameliorate this issue. Conceivably, the violin modes themselves do not directly prevent locking, rather, they are (by their high Q) diagnostics of the HF tail of seismic activity.