Ops Eve Shift Summary

TITLE: 07/31 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: While last night's shift went on without any observing time, tonight was the exact opposite with H1 locked and observing for the duration with steady range. Many aftershocks of yesterday's large earthquake with magnitudes over 5.0 rolled through continuously this evening, but H1 rode through all of these, occasionally entering earthquake mode during some of the larger motion. H1 has now been locked for over 15 hours.

On an operations note, based on TJ's suggestion from his time on last night's owl shift, I've edited the H1_MANAGER node to start an initial alignment instead of immediately calling for assistance if IR is not found during the 'FIND_IR' state. It will call for assistance, however, if an alignment has already been run and IR is still not found. Changes have been loaded and committed to svn.

OPS Wednesday DAY shift summary

TITLE: 07/30 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: We've been in and out of Earthquake mode all day from the constant aftershocks from the 8.8 yesterday off the coast of eastern Russia. We've been locked for almost 9.5 hours.
LOG:                                                                                                                               

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
13:45?	FAC	CBEX survey crew	MidY	N	Survey work	20:45
15:20	FAC	Nellie	Optics lab	N	Tech clean	15:34
16:32	FAC	Chris	Vac prep lab	N	Painting	18:32
18:55	ISC	Jennie	Optics lab	LOCAL	ISS array	19:52
20:13	FAC	Chris	VAC prep lab	N	Painting	21:30
21:13	ISC	Keita	Optics lab	N	Quick checks	21:30
21:15	EE	Marc	MidY	N	Grab some parts	21:56
21:20	FAC	Eric	EndY	N	Chiller yard alarm reset	21:39
21:51	AOS	Mitch	MidX	N	Look for a pelican case	22:10

I increased the gain on ITMY5/6 from +0.01 to +0.015 and it damps mode5 60-70% faster and mode6 100-110% faster based on the short and long monitors.

• 18:10 - 18:22 UTC We dropped observing as the SQZ FC lost lock, ran an SCAN_OPOTEMP
• 20:33 - 20:37 UTC we dropped OBS from the SQZ PMC losing lock, it was notifying about low volts before hand
  • 22:43 UTC Yellow dust alarm Optics lab, ~2000cts

BBSS BRD Mounts Recieved and Fit-Checked

Ibrahim

Got the newly cleane BRD mounts (D1900570) for the BBSS and fit-checked them on the suspension. They fit just fine with the only note being that the bottom loop wires (the small ends after the clamp) had to be moved out of the way. Of the 2 I fit-checked, 3 of the 4 holes went in very easily and the other fit mostly easily (no extra force applied going in just was not totally smooth turning in). Pictures attached.

The plan now is to get with Rahul and install the actual BRDs, then re-take the Bounce and Roll mode measurements to confirm that they are damping as expected. 

ETMX glitch locklosses vs ETMX bias - O3b vs O4a

Camilla, Oli

We were wondering what the bias had been during O3, as compared to the amount of ETMX glitch locklosses that had occurred during O3. We referenced Iain Morton's presentation, slide 17. Ignoring the 'SAME' lockloss tags since we know those were caused by the PSL glitching, we found that only 5% of the locklosses from O3b had an ETM_GLITCH tag, while 15% of the locklosses from O4a (plus a bit of O4b) had an ETM_GLITCH tag. Since the method in use at this time for finding ETMX glitches wasn't very good at catching the smaller glitches, we can guess that both percentages may actually be a bit higher, but either way, the percentage of locklosses from ETMX glitches during O3b were much less. We decided to look at what the ETMX bias was during O3b and how that compares to the bias we've been using throughout O4 (until it was changed on Monday 86027).

For all of O3 and up until February 28, 2023 (67698), we had been at a bias of around -450, and then after that date we changed to a bias of around +130, which is what we've had for all of O4 up until a couple of days ago.

 

Ops Eve Shift Start

TITLE: 07/30 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 9mph 3min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY: H1 has been locked for almost 9 hours.

• Ground motion has been elevated all day, bouncing up and down as aftershocks from the large EQ yesterday roll through
• Wind low
• All other systems nominal

OPS Wednesday DAY midshift update

STATE of H1: Observing at 145Mpc

We've been locked for 4:21.

• 18:10 - 18:22 UTC We dropped observing as the SQZ FC lost lock, I decided to run the SCAN_OPOTEMP state during this. SQZing doesn't look all that much different afterwords, the BLRMs look about the same.
• A full 24 hours of earthquakes.

Wed CP1 Fill

Wed Jul 30 10:19:53 2025 INFO: Fill completed in 19min 49secs

Gerardo confirmed a good fill curbside.

Unbeelievable Summer at LHO

The site has been abuzz as of late. For the past few months, western honeybees have turned LIGO from interferometer to apiary. The first swarm, near the LSB lift station was reported in early May. Since then, Facilities group and others have been combing the site, and some 15 colonies have been captured and extracted with the help of Phillip Johnson from The Bee Team (seen in buzzworthy photos below). 

The bees have been largely indiscriminate about hive habitat selection. To date we have removed them from spools, BTE interiors, irrigation boxes and connexes. It's not clear where the bees came from, or how they got here. The nearest location to site that I could find which would utilize bees, Brainstorm Cellars, is some 6 miles away. This is further than a swarm will migrate. Stranger yet, the habitat between us and the nearest possible location is, to my understanding, not favorable for hive building which would rule out leap frogging to us. Nevertheless, the bees are here and thriving. The queens are prolific egg layers; the forager bees are caked in pollen and the extractions of more established colonies are chalked full with 10's of pounds of delicious, capped honey. Bee temperament is, across the board, very mild. Even during our most invasive extractions they seem largely unbothered. 

All that to say, I hope to see the increase of bees across site taper off strongly, especially as we inch closer to Fall. In the meantime, M. Landry has recently reached out to the WSU Honeybees and Pollinators Program to help us understand how we came to find this abnormal explosion of bees. That meeting has yet to take place. 


C. Soike, M. Robinson, T. Guidry

This work demands we the dust off the ol' LHO Paper Plate Award!!

"This buzz-worthy award goes to Tyler Guidry, Phillip Johnson, Mitch Robison, Chris Soike, and Kim Stewart for LHO's Un-bee-lievble Summer (LHO aLOG 86102)."

Congratulations!

OPS Wednesday DAY shift start

TITLE: 07/30 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 154 Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

• We've been locked for about 20 minutes but we're not observing due to a PCALX issue that I don't think I've seen before
  • "PCAL OSC switches are missconfiguired Check OAF CAL-INF_MASTER and CAL-SWEPT_SINE channels"
  • I went to PCALX then EXCITATIONS then clicked ENABLE on H1:CAL-PCALX_SWEPT_SINE_ON and cleared this
• 14:28 UTC Observing
• Lots of aftershocks overnight as reported by TJ

Ops Owl Update

Looks like the first attempt at locking after the ground motion settled enough failed at find IR. I was notified, but we should change this to start an initial aligment first.

• Started initial alignment, couldn't find IR for input alignment.
  • Restored IMs back to osem positions before the earthquake
• SRY wouldn't lock and even tripped SRM. Moved SRM in P & Y to get better flashes.
• Then another earthquake rolled through, though not huge, it delayed locking for a bit.
  • Reset H1_MANAGER and IFO_NOTIFY, we'll see if it can bring us back on its own.

DQ Shift report for the week from 2025-07-21 to 2025-07-27

Below is the summary of the DQ shift for the week from 2025-07-21 to 2025-07-27

• During this week, the average BNS range was around 150 Mpc, and there were no large variations in the range.
• Except for a few features, the strain spectrogram looked reasonable.
  • There was excess noise at the first and second harmonics of the violin modes at the beginning of the locks.
  • In addition to persistent lines, there were also lines whose amplitude and frequency varied over time. Most of these features were below 300 Hz. The details of these features are available in the a-log 85984. These lines were identified using the STRAIN_NOLINES channel, and hence some of these lines may be removed/cleaned in the STRAIN_CLEAN channel.
  • At the beginning of the locks, sometimes the calibration lines were seen in the STRAIN_NOLINES for a few minutes.
  • On a couple of instances, the IFO was locked during earthquakes during which there was excess noise in the 10 -20 Hz band.
• The glitch rates at different Omicron SNRs were mostly consistent throughout the week.
  • Most of these glitches were centered around specific frequencies such as 23 Hz, 34 Hz, 46 Hz, etc.
• There were no new and significant features in PEM channels
  • Many of the accelerometers and some microphones in the corner station also see similar non-stationary lines in h(t) described above (link to plots in a-log 86010).
• The fscan usually identified 29.969* Hz and 99.9986* Hz comb features.
• The dominant channels in Hveto were usually ASC TRANS, HPI, and ISI from the corner station. However, most days they had a poor use percentage.

The full DQ shift report with day-by-day details is available at https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20250721

Ops Eve Shift Summary

TITLE: 07/30 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY: No observing time this shift due to the M8.8 (revised) earthquake off the Russian coast this afternoon and about a dozen M5.0+ aftershocks since. I waited until around 04:50 UTC to start untripping suspensions and seismic platforms to give the Earth time to calm down so they wouldn't just immediately trip again; everything came back up without issue, although IFO alignment looks poor. Since the ground is still shaking and the seismic environment is still in earthquake mode, H1 won't be locking for a while. However, I did set up the auto owl shift so that H1_MANAGER will start the process (and likely also need to run an alignment) once we're out of earthquake mode, even if that won't be for another hour or two.

Daily DRMI Locking attempts since the Vent

I have attatched a PDF that contains 54 plots with minute trend data of the daily DRMI Locking process since the vent.
The DRMI Locking process being claimed here is channel H1:GRD-ISC_LOCK_STATE_N  being between states 18 and 101.
All times are in UTC.

These plots were made by  sc_iterator.py which relies on statecounter 3.0. 

 

 

EX Pcal cabling...

FranciscoL, TonyS

Summary: We connected a BNC from the "Out Mon" of D1300599 to "In 2" of D1400423. The channel H1:CAL-PCALX_OFS_AOM_DRIVE_MON_OUT16 now plots what the AOM is doing. Problem solved.

---

Earlier today I went to track where in the chain of connections the InLoopOut signal was "lost" (I made that claim in 85753). All the signals of InLoopOut were working all the way out to the TCS rack. However, after a closer look on all the schematics for the Pcal and the TCS boards, I noticed that I was tracking the wrong channel. In other words, the InLoopOut channel was not the problematic channel -- this would otherwise be a super urgent matter, as the InLoopOut is the signal coming straight from the OFS PD -- but it was the AOM Mon channel (for the interested reader, look at D1300226V13 --> page 3 --> X1-TCS-C1-4 --> second (top to bottom) 9-pi D-Sub Male output, to find the channel). At the time of realizing my mistake, the maintenance period was long over, so I left it as a pending task for our next visit to EX.

The world gifted me with very high seismic activity and Tony and I finally finished the problem by connecting the BNC that was missing according to D1300226 schematic and stated in the summary. Attached are screenshots of the ndscope before (earlier today, as I was looking for the problem) and after plugging the BNC in the chassis, in their respective order. The change is seen in the green trace on the second (top to bottom) plot.

Tagging CDS so we are all aware that the problem was solved and it is concluded now.

CDS Maintenance Summary: Tuesday 29th July 2025

WP12709 Replace EX Dolphin IX S600 Switch

Jonathan, Erik, EJ, Dave:

The Dolphin IX switch at EX was damaged by the 06apr2025 power glitch, it continued to function as a switch but its nework interface stopped working. This meant we couldn't fence a particular EX front end from the Dolphin fabric by disabling its switch port via the network interface. Instead we were using the IPC_PAUSE function of the IOP models. Also because the RCG needs to talk over the network to a swich on startup, Erik configured EX frontends to control an unused port on the EY switch.

This morning Erik replaced the broken h1rfmfecex0 with a good spare. The temporary control-EY-switch-because-EX-is-broken change was removed.

Before this work the EX SWWD was bypassed on h1iopseiex and h1cdsrfm was powered down.

During the startup of the new switch, the IOP models for SUS and ISC were time glitched putting them into a DACKILL state. All models on h1susex and h1iscex were restarted to recover from this.

Several minutes later h1susex spontaneously crashed, requiring a reboot. Everything as been stable from this point onwards.

WP12687 Add STANDDOWN EPICS channels to DAQ

Dave:

I added a new H1EPICS_STANDDOWN.ini to the DAQ, it was installed as part of today's DAQ restart.

WP12719 Add two FCES Ion Pumps and One Gauge To Vacuum Controls

Gerardo, Janos, Patrick, Dave

Patrick modifed h0vacly Beckhoff to read out two new FCES Ion Pumps and a new Gauge. 

The new H0EPICS_VACLY.ini was added to the DAQ, requiring a EDC+DAQ retstart.

WP12689 Add SUS SR3/PR3 Fast Channels To DAQ

Jeff, Oli, Brian, Edgard, Dave:

New h1sussr3 and h1suspr3 models (HLTS suspensions) were installed this morning. Each model added two 512Hz fast channels to the DAQ. Renaming of subsystem parts resulted in the renaming of many fast and slow DAQ channels. A summary of the changes:

In sus/common/models three files were changed (svn version numbers shown):

HLTS_MASTER_W_EST.mdl production=r31259 new=32426

SIXOSEM_T_STAGE_MASTER_W_EST.mdl  production=r31287 new=32426

ESTIMATOR_PARTS.mdl production=r31241 new=32426

HLTS_MASTER_W_EST.mdl:

only change is to the DAQ_Channels list, added two chans M1_ADD_[P,Y]_TOTAL

SIXOSEM_T_STAGE_MASTER_W_EST.mdl:

At top level, change the names of the two ESTIMATOR_HXTS_M1_ONLY blocks:

PIT -> EST_P

YAW -> EST_Y

Inside the ADD block:

Add two testpoints P_TOTAL, Y_TOTAL (referenced by HLTS mdl)

ESTIMATOR_PARTS.mdl:

Rename block EST -> FUSION

Rename filtermodule DAMP_EST -> DAMP_FUSION

Rename epicspart DAMP_SIGMON -> OUT_DRIVEMON

Rename testpoint DAMP_SIG -> OUT_DRIVE

DAQ_Channels list changed according to the above renames.

DAQ Changes:

This results in a large number of DAQ changes for SR3 and PR3. For each model:

+496 slow chans, -496 slow chans (rename of 496 channels).

+64 fast chans, -62 fast chans (add 2 chans, rename 62 chans).

DAQ Restart

Jonathan, Dave:

The DAQ was restarted for several changes:

New SR3 and PR3 INI, fast and slow channel renames, addition of 512Hz fast channels.

New H0EPICS_VACLY.ini, adding Ion Pumps and Gauge to EDC.

New H1EPICS_STANDDOWN.ini, adding ifo standdown channels to EDC.

This was a full EDC DAQ restart. Procedure was:

stop TW0 and TW1, then restart EDC

restart DAQ 0-leg

restart DAQ 1-leg

As usual GDS1 needed a second restart, but unusual FW1 spontaneously restarted itself after have ran for 55 minutes, an uncommon late restart.

Jonathan tested new FW2 code which sets the run number in one place and propagates it to the various frame types.