Alerted by H1 manager that there was an issue - relocking had just taken a while and then we just had to wait in OMC_WHITENING for a long time because the violins were pretty rung up. We got into Observing fine eventually.
TITLE: 12/23 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Quiet shift with H1 locked and observing throughout. Microseism has remained high but wind is still fortunately low. Current lock stretch up to 5.5 hours.
FAMIS 31065
On-table FSS work last Tuesday (alog81865) is clearly seen on several trends. Since then, PMC REFL looks to be a bit noisier. I also bumped up the ISS diffracted power 2 days ago to get back to the 4% we like to have it these days.
TITLE: 12/22 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 154 Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: 2ndary microseism increased a little more throughout the day, 2 locklosses and relocks. We've been locked for a little over 2 hours.
LOG: No log.
TITLE: 12/23 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 5mph Gusts, 2mph 3min avg
Primary useism: 0.10 μm/s
Secondary useism: 0.96 μm/s
QUICK SUMMARY: H1 currently relocking and up to PREP_DC_READOUT.
20:56 UTC lockloss, ASC CHARD ringup?
22:29 UTC Observing
Sun Dec 22 10:09:57 2024 INFO: Fill completed in 9min 54secs
TCmins = [-110C, -50C]
TITLE: 12/22 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 6mph Gusts, 4mph 3min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.76 μm/s
QUICK SUMMARY:
I was called to intervene with the detector and found that we were in NLN but couldn't get into Observing because of an SDF diff on the ramp time for the ETMY Lock ESD bias. When TJ and I were testing out the code for the function that generates the INCREASE/DECREASE BIAS states for SUS_PI, we had decided on leaving the line in that sets the bias ramp time to 30 for going from a bias offset of -9 back to the nominal bias of -4.9. We thought it would be fine because the ramp time would get reverted when we lost lock, but we forgot about the part where we have to go back into Observing after the PI gets damped at the higher bias (oops). At least it was me who had to intervene so I can suffer the consequences of my mistakes! I changed that line in the code(attachment) so that when we are going from the higher bias to the nominal lower bias, the ramp time is the nominal 120 seconds. So it'll take longer to ramp back to nominal bias, but at least we'll be able to go back to Observing!
TITLE: 12/22 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Quiet shift until an EQ caused a lockloss. Relocking has been challenging due to high microseism, but H1 has been able to make progress and is currently relocking up to ENGAGE_SOFT_LOOPS. Fortunately, it looks like the microseism has turned around and started to trend down.
Lockloss @ 04:08 UTC - link to lockloss tool
Cause likely due to M5.6 EQ out of Alaska and aftershocks; H1 had been locked for just over 7 hours.
[OpsInfo] I noticed that verbal alarms didn't announce the transition to earthquake mode in this instance. Found out that this is because EQ transitions were only announced for the regular EARTHQUAKE state in SEI_ENV, but we were in USEISM_EARTHQUAKE due to the high secondary microseism. I've updated the verbal alarms code to now also announce the transition to earthquake mode when going to the USEISM_EARTHQUAKE state.
TITLE: 12/21 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 34mph Gusts, 26mph 3min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.66 μm/s
QUICK SUMMARY: H1 has been locked for 3 hours.
TITLE: 12/21 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: One lockloss from an earthquake, continuously increasing microseism made relocking more difficult. The wind started to pick up around 22:00UTC. We've been locked for just 3 hours, I was not able to get the calibration measurement in.
LOG: No log
Sat Dec 21 10:05:49 2024 INFO: Fill completed in 5min 46secs
Running on TC-A only. TC-B (min=-52C) has been removed from plot to reduce y-axis log decade span.
Ryan S, Jason, Dave:
In the past few days cds_report was flagging a large log file for h1daqnds1. On investigation I found it was due to the DIAG_MAIN guardian node asking for 10 second trends of H1:PSL-ISS_DIFFRACTION_AVG eight times every second. This was generating +630k logs daily over the past few days.
Working with Ryan S, the problem was found to be due to the ISS diffraction avg signal trending downwards and going below DIAG_MAIN's 2.5 low limit, which was then requesting the 10 second averages of this channel from the DAQ.
In consultation with Jason, Ryan made two changes:
1. Boosted the ISS diffraction avg back up above 2.5 [see attached trend]
2. Removed the code which requests the 10 second average of the average channel, the 1 second average this channel represents is sufficient lookback [DIAG_MAIN.py code change shown below]
665 @SYSDIAG.register_test
666 def PSL_ISS():
667 """ISS diffracted power
668
669 """
670 diff_high = 5.5
671 diff_low = 2.5
672 if PSL_on():
673 ¦ diff_now = ezca['PSL-ISS_DIFFRACTION_AVG']
674 ¦ # if diff_now > diff_high:
675 ¦ # diff_pwr = try_avg(-10, 'PSL-ISS_DIFFRACTION_AVG')
676 ¦ # if diff_pwr >= diff_high:
677 ¦ # yield "Diffracted power is high"
678 ¦ # elif diff_now < diff_low:
679 ¦ # diff_pwr = try_avg(-10, 'PSL-ISS_DIFFRACTION_AVG')
680 ¦ # if diff_pwr <= diff_high:
681 ¦ # yield "Diffracted power is low"
682 ¦ if diff_now > diff_high:
683 ¦ ¦ yield "Diffracted power high"
684 ¦ elif diff_now < diff_low:
685 ¦ ¦ yield "Diffracted power low"
686
We were getting hit by a 6.1 from Vanuatu at the time, https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1418832746.
SEI went to earthquake mode 1 minute before the LL.
21:00 UTC Observing
TITLE: 12/21 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 14mph Gusts, 10mph 3min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.49 μm/s
QUICK SUMMARY:
We've been locked for 6 hours
2ndary microseism is increasing, wind is low/moderate
TITLE: 12/21 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: H1 has been locked and observing for about 4.5 hours. Been a quiet evening since H1 relocked early this shift.
