Ops Monday Morning report

TITLE: 04/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

Iluminators for HAM3 BSC1 and 2 are now on.
HEPI and ISI for BSC2 is locked out. tagging PEM. and there is now a Red H1:SYS-ETHERCAT_DEVICE_1_SLAVECOUNTACTUAL due to the iluminator Chassis device that Fil turned on between BSC1 and BSC1 in the beergarden to allow the illuminators to turn on. tagging CDS. 

PCAL troubleshooting: 
Dripta and I were trying to take FFT's of PCAL Lines for more Power budget work and we noticed that the PCAL Y laser is off with no current on H1:CAL-PCALY_LASERDIODECURRENT. I trended the channel back and compared it with some channels that I thought may link it to the IOP restarts. 
Dripta went to EY to toggle the Laser power rocker switch just in case it was just a simple issue of a turn it off and back on again.  Tagging CDS again.

I noticed that MC1 & 3 were saturating. 
IMC H1:IMC-WFS_GAIN was set to 0 and the history was cleared to solve the problem of MC1 & MC3 were being saturated.
Trending some channels that were suspected to perhaps  be the cause of the junk signal and thus the cause of the saturations.
The IMC ASC has it's own block to do power normalization, different from the LSC power normalization. The output of this block is "INNORM" used to normalize MC2 NSUM for the IMC WFS triggering.
Relavent FRS ticket: 
https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=9387

More IOP model restarts happening again for HAMS 7,8 then HAM 5,6, & BSCs 1,2, & 3.
Please see the attached SDF Diffs for HAMs 5, 6 and the BS: 
SUSSR3
SUSRM
SUSIFOOUT
SUSSQZOUT
SUSBS (my bad I had sort on substring selected with an old Optical Align string in the search field.)
All SDFs were accepted before the IOP restart. 
ITMS  IOPs have not been rebooted yet. Tagging SUS, SQZ & ISC

MC2_trans Beam Dump twas Twissled today, then put back into original place. 

The ITMs had their IOP's restarted after a time cable swap today as well.
All SUS were turned back to Aligned.
SEI for HAM 2,3,4  and BSC 1 & 2 still have their HEPI's and or their ISI's WD tripped or other wise in a ISI Damped Hepi offline state.

As you can see from this ITMY trend, Jim went inside to lock the ISI and tripped the ISI, then Jason went in and tripped it again, then there was an IOP restart. 

A slightly less exciting trend of ITMX.


LOG:
 

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
14:47	FAC	Randy	LVEA	n	Prep for dome removal and other moves	16:06
15:09	FAC	Kim, Nelly	LVEA	n	Tech clean - HAM3, Biergarten	16:07
15:47	FAC	Randy	LVEA BSC2	N	Working on removing BSC2 Dome	16:06
16:07	VAC	Randy	MID Y	N	Picking up matieral	17:53
16:09	FAC	Nellie	Mid & End Y	N	Technical cleaning & Resupply	18:09
16:13	SEI	Jim & Ibrahim	LVEA BeerGarden	N	Locking Hepi , ISI on BSC2	18:05
16:26	FAC	Kim	LVEA Output Arm	N	Technical Cleaning & Resupply	17:40
16:33	CBEX	Tyler's C&E Trentching	Mid Y	N	trenching for CBEX Building	02:33
16:41	VAC	Gerardo	LVEA BSC2	N	Removing Vac Equipment from BSC2 dome	18:10
16:43	IAS	Jason	LVEA	N	Warming up the FARO system	17:13
17:25	CHETA	Camilla	Optics Lab	n	Grabbing optics.	18:26
17:36	VAC	Jordan	LVEA BSC2	N	Helping Gerado	18:10
17:38	EE	Fil & Corey	LVEA	N	Setting up cameras and turning on Illuminators	19:38
18:11	PCAL	Dripta & Disha	EY	N	Power Cycling PCAL laser	19:02
18:29	IAS	Jason & Crouch	LVEA HAM3 to BSC1&2	N	FARO surveying HAM3 & BSCs	19:57
19:37	IAS	Ibrahim	LVEA BeerGarden	N	DeBaffling	19:53
19:41	VAC	Randy	LVEA BeerGarden	N	BSC2 Dome removal prep	20:02
19:51	EE	Marc & Erik	HAM Shaq & HAM7	N	Swapping power supply Power cables inside chassis	20:23
19:58	PEM	TJ	EX	N	Checking Dust Mon Pump	20:29
20:30	TCS	Camilla & Maddi , TJ	EX	Yes	Hartman troubleshooting &work	22:05
21:01	IAS	Jason & Ryan, Jim, Ibrahim	LVEA HAM3, BSC1&2	N	Faro work inside the Vac chambers	00:05
21:43	VAC	Gerardo & Jordan	LVEA HAM1	N	taking parts off of HAM1	22:38
22:12	IAS	Sheila	LVEA Ham3	N	Helping with Faro work	22:38
23:19	IAS	TJ	Optics Lab & LVEA	N	looking for a Class B Ruler in mm.	23:49

CDS Maintenance Summary: Monday 13th April 2026

WP13110 Upgrade IO Chassis Timing Card Firmware and Sync Ribbon

Marc, Erik, Jeff, Tony, Dave:

The following IO Chassis were upgraded today: h1cdsh8, h1susb13, h1sush[12,56,6,7], h1seib[1,2,3], h1seih[23,45].

Models were rebuilt against RCG5.5.2 for: h1cdsh8, h1seib[1,2,3], h1seih[23,45], h1sush[56,7]

This completes the IO Chassis TC upgrades. T2600143 has been updated.

Restore h1hwsey Hartmann Wavefront Sensor Computer

TJ, Jonathan, Dave:

Last Friday I got the h1hwsey computer running again. This afternoon TJ started the HWS ETMY software on it. Just prior to this Jonathan and I stopped and disabled the dummy IOC which had been running on cdsioc0 since dec2025.

comparison of FC1, SRM, SR3 changes when mega cleanroom turned on and off

This is a follow up on the FC1 alingment confusion described in 89751 (also FRS ticket 37555).  

I compared what happened to FC1 and FC2 during the mega cleanroom temperature swings.  

The vertical shift is seen just about equally from the three top mass osesms on FC1, consistent with the blade springs getting warmer, not an individual osem problem. At this time both HAM5 + HAM7 were under vacuum.  

	FC1	SRM	SR3
M1 V	-34 um	-48 um	-23 um
M1 P	-160 urad	+114 urad	-8 urad
M3 P	+48 urad	+76 urad	less than 2 urad

 

BSC1&2 locked this morning

Ibrahim and I went to ITMY this morning, I locked HEPI on ITMY while he was setting up the radios and charger. After HEPI was locked, I garbed up and went in with a stool, isopropyl and an allen and locked ITMY and BS ISIs. Mostly went smoothly, some of the lockers need convincing with the allen key. All done now, and ready for IAS.

Current locked status is: HEPI HAM1,2&3 and BSC 1,2&3 are all locked. HAM3 and BSC1&2 ISI are locked. Everything else is unlocked, though we should probably leave all of the SEI guardians in ISI_DAMPED_HEPI_OFFLINE, in the corner.

Mon CP1 Fill, note later time of 10:20

Mon Apr 13 10:30:08 2026 INFO: Fill completed in 10min 5secs

The 10:00 run did not proceed due to urgent security patching. Today's run was rescheduled at 10:20.

Corner station chiller sensor replacement

Evaporator refrigerant pool temperature sensor was replaced this morning on Chiller 1. The old sensor was failing to communicate the temperature of the evaporator during circuit 1 operation, which limited the cooling capacity. 

Ops Monday Morning report

TITLE: 04/13 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 22mph Gusts, 17mph 3min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.16 μm/s 
QUICK SUMMARY:
SUS JM3 is still saturating 
SUS BS is also seeing some saturation
BSC2 Dome Not removed yet. 
What work are we thinking will happen today?: 
Faroing Inside the vacuum chamber shooting through HAM3 to BSC2.  
Vac parts removed from BSC2 dome.  

Sun CP1 Fill

Sun Apr 12 10:11:32 2026 INFO: Fill completed in 11min 28secs

 

Sat CP1 Fill

Sat Apr 11 10:09:36 2026 INFO: Fill completed in 9min 32secs

 

BSC1 Door Removed

(Travis S., Randy T., Gerardo M.)
Filter cavity tube work
BSC1 +X door was removed today, to be able to accomplish that first we had close the big gap left on by the cleanroom curtains, and do some work with the filter cavity tube.  At the filter cavity tube next to BSC3 connection we removed one side of the bellows, studs were removed, we opted to do that so as not to stretch the door cover, we used aluminum foil to cover both of the holes on the bellow and gate valve before pulling the C3 cover thru.  No issues to report all worked out good.

Door Removal
We attached the engine hoist to the door and loaded the engine hoist with the door.  After some struggle the door was convinced to detach, the outer O-ring left its groove on two spots, top and bottom about 10 inches for each side, we could not get to the top O-ring very hard to reach, no space, so we just managed to unstick it from the door flange and left it out of the groove.  Door was removed without a problem, O-ring protectors were installed, and door covers were set on chamber and door.

h1hwsey powered up, not running code at this moment

TJ, Dave:

TJ would like to get h1hwsey running again. If memory serves, this did not come back after the 06dec2025 site power outage and at the time it was not needed so we did not investigate it further and instead ran the h1hwsetmy_dummy_ioc.service on cdsioc0 (alog88435) to green the EDC.

At 16:00 this afternoon I went to EY and found that h1hwsey's boot disk caddy was not quite seated fully. This computer is not labeled, but can be verified by: tracing the KVM chan3 cable, checking PCIe camera receiver is installed.

I was able to monitor its reboot using the KVM console. I logged in as user controls and was able to ping the EX HWS machine to verify network operation.

Before I left I labeled the machine.

We are currently not running the IOC on h1hwsey and continue to run the dummy IOC on cdsioc0. We will keep this configuration for the weekend, and on Monday we'll stop the dummy IOC and run the real one.

OPS Day Shift Summary

TITLE: 04/10 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

IFO is in IDLE for PLANNED MAINTENANCE

Main tasks today:

• BSC1 Door Off - Done
• CDS Work - all done
  • Sei H16 restarted
  • ISI Ham6 Updated
  • Timing Card upgraded
  • Susaux H56 restarted
  • Susaux B13 restarted
  • DAC restart to add more STS gnd channels
• CER Chassis Work - Done
• IAS FARO - HAM3 - Done

LOG:

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
14:36	FAC	Kim, Nellie	LVEA	N	Technical cleaning	15:21
14:56	FAC	Randy	MY, LVEA	N	Picking up lift	16:41
15:04	IAS	Jason	LVEA	N	Turning on FARO	15:17
15:49	FAC	Kim, Nellie	LVEA	N	Technical cleaning	16:25
16:08	OPS	TJ	LVEA	N	Helping Jim with HAM3 Platforms, looking around	16:58
16:14	SEI	Jim	LVEA	N	HAM3 Platforms	16:31
16:28	EE	Marc	CER	N	PSL Chassis Work	16:58
17:13		Camilla, Gerardo	Optics Lab	N	Inspecting optics	18:32
17:14	IAS	Ryan C, Jason	LVEA	N	FARO	19:40
17:17	FAC	Kim	LVEA	N	Technical Cleaning	17:46
18:33	VAC	Travis, Gerardo	LVEA	N	Checking out FC Tube	19:37
20:06	FAC	Randy	LVEA	N	BSC1 Doors	22:06
20:14	SUS	TJ, Elenna	Optics Lab	N	Manta Ray Cabling (BHSS)	21:08
20:31	IAS	Jason, Ryan C	LVEA	N	FARO	22:31
20:34	VAC	Gerardo, Travis	LVEA	N	Biergarten work	22:45
20:43	EE	Marc, Erik	CER	N	Chassis work	21:08
21:09	TCS	Madi, TJ	EY	Local	Hartmann	22:45
22:48	CDS	Dave	EY	N	Labeling	23:20
23:01	PEM	Robert	EX	N	Grounding studies	00:01

 

Input and BS Pointing Over Last Two Weeks

In an effort to capture the input and BS alignment in different configurations, I trended back some suspensions and their OpLev and Raw OSEM values for P and Y. Table is below and plot of total timespan is attached.

Date	03/23	04/06	04/07	04/10 (before BSC1 work (1500UTC))	04/10
Configuration	HAM 1,2,3 and BSC 1,2 HEPI Unlocked/Nominal, HAM3 ISI Unlocked/Nominal, GV5 Hard Closed	BSC 1,2 HEPI Locked, HAM2 HEPI Unlocked, HAM1/3 HEPI Locked, HAM 3 ISI Locked. GV2 Hard Closed, GV2, GV5 Hard Closed, GV 7 Soft Closed. HAM1 +y Door Still Off	Pre-Vent HAM1,2,3 HEPI Locked, BSC 1/2 HEPI Locked, HAM3 ISI Locked HAM1 +y Door ON	Corner Vented, HAM1,2,3 HEPI Locked, BSC 1/2 HEPI Locked, HAM3 ISI Locked HAM1 +y Door ON	Corner Vented, HAM1,2,3 HEPI Locked, BSC 1/2 HEPI Locked, HAM3 ISI Locked HAM1 +y Door ON, HAM3 Doors Off, BSC1 Door Off
Counts
BS P Oplev	5.59	-2.59	-5.44	-1.10	-1.1
BS Y Oplev	-21.43	-18.84	-18.28	3.52	6.20
MC 1 P	-701.17	-693.12	-645	-165.00	-1894.65
MC 1 Y	-1386.13	-1352.15	-1359.53	-1381.29	-1224.55
MC 2 P	862.00	732.19	712.30	852.70	1200.76
MC 2 Y	-629.41	-405.45	-406.43	-376.60	-998.15
MC 3 P	401.57	372.37	358.45	332.01	-1268.06
MC 3 Y	-154.20	-192.72	-202.2	-270.48	-470.61
IM 4 P	-2442.81	-2364.65	-2257.6	-2332.12	-2329.7
IM 4 Y	-95.51	-73.04	-80.57	-103.24	-105.1
PR 2 P	-435.90	-425.54	-395.4	-418.5	-417.83
PR 2 Y	-1174.28	-1134.31	-1141.43	-1246.12	-1245.86

CDS Maintenance Summary: Friday 10th April

WP13110 IO Chassis Timing Card Upgrade

Marc, Erik, Dave:

Today we upgraded the IO Chassis Timing Card firmware, sync ribbon cable and RCG (where needed) on:

Front End	Timing Card Firmware	Sync Ribbon Cable	RGC5.5.2 update
h1psl0	yes	yes	yes
h1susauxh13	no	yes	no
h1susauxh2	no	yes	yes
h1seih16	yes	yes	yes
h1susauxh56	yes	yes	yes

There were no issues. DCC doc T2600142 was updated.

WP13163 h1isiham6 build against latest blend_switch_library

Jim, Ibrahim, Erik, Dave:

h1isiham6 was updated to use the latest blend switch library, it was the last ISI HAM model to do so.

We combined this with the upgrade of its IO Chassis, and also rebuilt all of h1seih16's models against RCG5.5.2.

A DAQ restart was required.

Move EDC back to h1susauxb13

Erik, Dave:

Back in Dec 2025 at the start of the upgrades we moved the EDC from what was then susauxb123 so it would not be impacted by the BBSS preparation work.

Now that we have completed the BBSS prep, EDC has come home to what is now called h1susauxb13 (sans BS).

This didn't quite go as smoothly as planned and required several restarts of both DAQ legs before it was correct.

The basic squence was:

1. Erik made the puppet changes such that h1edc would not start on h1susauxh56 but would do so on h1susauxb13.

2. Following the upgrade of h1susauxh56's IO Chassis, we powered this up to verify no edc on it.

3. We then hand restarted edc on susauxb13, then started the DAQ 0leg. We had DAQ errors at this point.

4. After a reboot of susauxb13 we found the issue, the daq receiver was not being restarted on DC0, just the daqd service. One DC0 was restarted this way, it had a good edc status. Unfortunately it started not quite in sequence and a clean 0-leg restart later got all the delta-times positive.

5. Restarting the 1leg went a bit better, but NDS1 did not restart cleanly and needed a manual start.

h1susauxh56 is now ready for its BHD upgrade with no impact on EDC trending.

DAQ Restart

Erik, Dave:

As mentioned above this was quite a messy start due to the move of EDC to susauxb13. The EDC INI was also changed to add the non-RGC FEC/SDF channels for the new models started in the past week (spih23, suslo12, susauxh6).

The h1isiham6 ini was also part of this DAQ restart to add the +12 STS GND channels.

 

ITM only CHETA likely to exacerbate PI issues, particularly for LHO

It dawned on me recently that with the scratched view ports for CHETA, the delay of the CHETA roll out could have a subtle saving grace.
The intended use for IR1 is to heat the ITM with the same power as the absorbed power, so that the ITMs would remain in more or less the same thermal state at all times, making DRMI locking fast and reliable, and removing some of the impact on SRC detuning changing over time as the ITM lenses would evolve less than in O4.
It appears that in the CHETA Design Document, the effects on mode spacing and therefore Parametric Instabilities (PIs) was completely overlooked. Shame, no PI expert in LIGO was consulted, I'm even an author on the cited document in the design document and how we used CHETA like actuators at UWA to mitigate PI.

Very simplistically, rolling out CHETA only for the ITMs (as was the plan for the upcoming IR1) would have two impacts on how the optical mode spacing evolves:

• Mode spacings would start closer to the final, thermalised, frequencies.
• They would evolve with the same time constant (only ETM heating) as in O4, to the thermalised state.
  • But! Because they start closer, the mode spacing would evolve slower.

In O4 LHO's QUAD Radii of curvature's operating point has them stuck between two lock-losing PIs, and they haven't managed to demonstrate reliable damping of either. This lead to several periods in O4 where their lock stretches would be repeatedly cut short at around 4 hours.
The implementation of CHETA on ITMs only would have put them closer to their 10.4 kHz PIs as a starting condition, and because the mode spacing would be changing more slowly over time, they would experience the frequency matched condition of PI for longer periods of time. Thus:

• LHO would lose locks more reliably.
• LHO would have shorted locks.

I could spend time modelling this but its blindingly obvious once you notice.
CHETA on both ETMs and ITMs would theoretically allow you to start with essentially thermalised mode spacing, allowing you to more reliably side-step PIs.

BSC1 +Y Door Bolts Removed + BSC2 Dome Bolts Removed, BUT Had One Galled Dome Bolt!

(Jordan V, Corey G)

In prep for upcoming BBSS Install, "broke" bolts (which is slang for loosened them) on:

• BSC1's +Y Door in the Beer Garden
• BSC2's Dome (to see if giving the Dome's O-Ring some time to "decompress" helps us out at all with removal next week).  

BSC1's +Y Door Bolts (mostly) Removed & Accelerometer Cable Disconnected

BSC1's door had all bolts but (4) removed.  Sadly, I removed the bottom (easy ones) bolts first and then when I got to the top bolts they were very hard to loosen because of some sag.  Additionally, their access is horrible with:  a stiffening plate for the chamber right above the rear access to these bolts, then there are cable trays also in the way, and the C3 fabric cover is draped in your face the whole time.  Jordan helped out and we were able to get the last top bolt removed.  We also loosened the (4) bolts 1-turn which is the typical procedure.

There was an accelerometer still cabled up to this door.  I disconnected the cable (labeled "PEM:BSC1_ACC_X") and stuffed the cable into the cable tray above this door.

Time lapse of me removing half the bolts in this LINK (since we only files allowed in alog have to be smaller than 15Mb.)  :(

BSC2's Dome Bolts (mostly) Removed, But Have A Galled Bolt!

BSC2's dome bolts were removed.  4-bolts were left in place (they were loosened and then tightened up a little).  One notable, is there is 1-galled dome bolt.  Jordan and I used the long wrenches to see if we could use more force to help it break loose, but no luck.  We did not try pouring some alcohol on the bolt to help as a lubricant.  The bolt is loose and has about 1-2mm of a gap on it.  So, the pros could try use even longer cheater bars, a saw cutter, or they were also chatting about a tool to cut the nut off.  At any rate, 

IO Chassis Firmware and Sync Cable Upgrade

Per WP13110 we have upgraded the IO chassis listed below.
This link contains the asbuilt for all of the H1 IO Chassis: https://dcc.ligo.org/LIGO-D1301004
This link contains the ECR for the IO Chassis sync cable upgrade: https://dcc.ligo.org/LIGO-E2600069

This link is to the work in progress spreadsheet LHO Front Ends Upgrade Status

Corner:
SUSB2H34 has upgraded firmware and sync
SUSAUXB2H34 has upgraded firmware and sync
SEIH7 has upgraded firmware and sync
OMC0 has upgraded firmware and sync
OAF0 has upgraded firmware and sync
ASC0 has upgraded firmware and sync
LSC0 has upgraded firmware and sync