FINALLY made it! After lots of struggles to get DRMI to lock, 3 lock losses from Transition_From_ETMX, we are now observing.
We've now had two lock losses from the Transition from ETMX state or immediately after while trying to reacquire. Unlike back on Nov 22 (alog81430), SRCL seems fine. For the first one, the locklost tool 1417133960 shows the IMC-SERVO_SPLITMON channel is saturating ~5sec before lock loss, then there is some odd LSC signals 40msecs before the tool tagged the lock loss (attachment 1). This might just be the lock loss itself though. The second one (locklost tool 1417136668) hasn't tagged anything yet, but ETMY has a glitch 2.5 sec before lock loss and ETMX seems to move more from that point (attachment 2).
Another one while I was looking at the code for the Transition_From_ETMX state. We were there for a few minutes before I noticed CHARD & DHARD inputs ringing up. Unsure how to save it, I just requested it to move on but it lead to a lock loss.
I ended up changing the SUS-ITMX_L3_LOCK_BIAS_TRAMP to 25 from its 30 to hopefully move to a safer place sooner. Since it was already changed from 60 a week ago, I didn't want to go too short. Worked this one time.
Sheila, Camilla
We had another lockloss with an ETMX_L2 glitch beforehand this morning, plot, and it seems that even a successful transition this morning had a glitch too but it was smaller plot. We looked at the ISC_LOCK.py code and it's not yet obvious what's causing this glitch. The successful transition also had a DARM wobble up to 2e6 plot but when we have the locklosses, DARM goes to ~10e6.
While looking at all the filters, we found that ETMX_L2_LOCK_L ramp time in 20s screenshot although we only wait 10s in ISC_LOCK. We will edit this tomorrow when we are not observing. We don't think this will effect the glitch as there is no input/output to this filter at the time of the glitch.
The only thing that seems like it could cause the glitch is the DARM1 FM1 being turned off, we don't yet understand how and had similar issues we thought we solved in 77640
This morning I edited ETMX_L2_LOCK_L FM1 ramp time down to 10s, reloaded coefficients.
TITLE: 12/01 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Just had a ~19.5hr lock! (was hoping we'd make it past 22.5hrs for a new record since the new NPRO swap)
Other than that a fairly quiet shift. (I'm under the weather and having TJ tag in.)
LOG:
TITLE: 12/01 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 3mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.46 μm/s
QUICK SUMMARY:
Just lost lock before I showed up. Currently acquiring DRMI. The useism is increasing, peaking above the 95%. Wind is low.
Smoot sailing GW Detector wise with H1 at 17hrs8min (but on nuc28, the LockClock died, so I ran the launch startup script, and this restarts the LockClock. So you have to add about 16hr13min to whatever time the Lock Clock has for this current lock.)
Dave phoned in to address issues with the cdslogin computer.
What Else: Francisco and Neil are onsite. In the last few hours the microseism continues increase and now peaking above the 95th percentile line.
Sun Dec 01 10:08:08 2024 Fill completed in 8min 5secs
No texts/emails for this fill due to cdslogin issues.
cdslogin is in a strange state, not quite down but not sending alarms or alerts. It is pingable, I am still using it as a ssh-tunnel for my no-machine connection to opslogin0, but it does not accept any new ssh logins and my shell on it cannot find any commands.
File system error starting at 03:42:20 this morning
I power cycled cdslogin remotely via IPMI (10:50 power down, 10:52 power up) to force a fsck. The system came back up in operational mode, the systemd services alarms and locklossalerts started normally.
These services write to the local file system, which is presumably why they were down when the local FS switched to read-only mode.
Two improvements spring to mind:
Make alarms and alerts memory resident only, no reliance on any file system.
Make these services portable to cdsssh if cdslogin became unusable.
There is no indication of any mains power issues at 03:42 this morning. No UPS reports, and the three phases of the corner station mains-mon look good throughout.
TITLE: 12/01 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 1mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.33 μm/s
QUICK SUMMARY:
Happy December 1st! H1's been locked 13.25hrs with a range hovering around 160Mpc. It's a balmy 35ºF outside, µseism is starting to touch the 95th percentile line again, minimal breezes.
CDS SDF has a yellow box on the CDS Overview again.
TITLE: 12/01 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: One lockloss this shift, we've been locked for 3:20. Low frequency noise is slightly elevated.
LOG: No log.
00:53 UTC lockloss
02:41 UTC observing
TITLE: 11/30 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
H1 was locked entire shift except for a lockloss right before the shift. Also squeezed in the Saturday Calibration (so just under 90min of downtime for the locking + calibration).
LOG:
TITLE: 11/30 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 2mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.23 μm/s
QUICK SUMMARY:
H1 will be thermalized right around the Sat Calibration time of 1930utc (1130am PDT). L1 has been down due to a frontend crash/chassis swap this morning, so will jump on this down time to get the calibration run for H1. Louis Dartez wanted operators to continue running the new/non-standard simulines measurement (noted below).
Measurement NOTES:
