Belated alog from Friday.

Summary:

After realigning IMC and IM2 on Thursday, we continued aligning IM3 to put the beam in front of the PRM to the nominal position, IM4 to set the beam position in front of PR2, and PRM so the beam retroreflects. IM1 was left untouched, so the only change in IM1-IM2 line is from the change in the IMC alignment.

Clipping on IM4 baffle (HA12) disappeared completely. Beam position on IFI output baffle looked OK. Beam position on IFI input baffle didn't change (as expected) and still looked OK. There are some mystery 

We saw flashes on ASC REFL sensors but they were weak. We tried to center them using RMs but RM2 seems to rail in YAW.

What to do:

Scan PRM to make sure that we're NOT looking at some kind of ghost beam on ASC REFL sensors.

Assuming that we're looking at the real beam on REFL sensors, we can do either (or some of) these:

• Relieve RM2 by rotating RH9 (the fixed IFO refl pick off mirror in HAM2) by positive 500urad or so in YAW. This is the equivalent of the combination of negative 0.53 mrad for RM1 and negative 2mrad for RM2 in YAW. After doing this we can add 530urad to RM1 and 2000urad and continue centering from there. See attached cartoon (doesn't scale but the sign of things is correct) and the script.
• IM1 is almost degenerate as RH9 as far as relieving RM1 and RM2 is concerned, so we can rotate IM1 by negative 500urad or so in YAW. This requires that we tweak IM3, IM4 and PRM again but has an added benefit of moving the beam spot on the IFI input baffle closer to the center.
• Make the OSEM coil driver output impedance smaller for RMs to gain more range. 1.2k to 500 Ohm should be safe. 

Thankfully no more power outages following the second (14:27) outage Sat 30may2026. Here are the systems we left offline which will be restarted tomorrow, Mon 01jun2026.

PSL Diode Room Beckhoff Controls

Long range dolphin end station Adnaco expansion chassis (h1cdsrfm)

NCAL EX

BRS EX, EY

HWS EX, EY

MAINSMON EX

CHILLERMON EX, EY

The second outage took the end station chillers down again. Jordan V restored power to the CWPs, and I cleared all alarms remotely. Temps rose a couple degrees at each end to just under 70f. Those should correct quickly, if not already.

R. McCarthy J. Vanosky T. Guidry

After the failure of the compressors HMI last week, see alog 90318. A tech from Rogers Machinery (Brandon) came on site today to install a new HMI unit. The compressor had to be powered off to make the swap, so I isolated the purge air from the LVEA at the ball valve just past the drying towers. 

The HMI swap was pretty quick only took about 10 minutes, and we can now see the temperatures and pressures in the compressor. I let the purge air skid run for ~10 minutes to make sure the dewpoint was stable, at -75C, then opened back to the LVEA. 

The replacement oil pump should be delivered next week, at which time a tech will have to come out again and make the swap.

In the two power issues today Erik had to manually start fmcs-epics-ioc.  It is a mac mini in a custom rack container that makes it hard to see if it is on or to power cycle it.  This machine causes issues at each power outage.

After the second outage I put a VM together to host a containerized version of the code.  It is staged and ready to go, but not tested.  If it goes down again we go onto the new system and start the container.  It should allows to get the fmcs going without sending someone to fiddle with the machine.

To do this:

1. log into fmcs-epics-ioc.

2. systemctl start deb10ioc

To make it perminant we need to modify the systemd file to have an install section, do a daemon-reload and a systemctl enable.

Ultimately we need to rebuild this ioc on a new OS release.

This morning we tried a power cycle of both the computer and the IO Chassis, with the chassis down for about 40 seconds. This did not bring the 3rd Adnaco BP back so we replaced it.

This afternoon we tried a 4 minute power down of the IO Chassis, and when the computer came up it saw all the cards. So perhaps the backplane removed from the IO Chassis this morning is good.

As before all front end computers powered down with the exception of cdsh8 and psl0. When we powered them back up all recovered except h1sush12, which has again lost its 3rd Adnaco backplane in its IO Chassis, even though this is a new BP installed this morning.

HWS at end have gone down, check. 9th port on timing master (dts) is down, check. FMCS EPICS is down, check.

this time h1cdrfm's adnaco expansion chassis at the end stations have not gone down though.

Lost power again.

Beginning at 9:57a I began receiving alarms for a fire pump run and a few minutes later alarms for end and corner station chilled water pumps. Initially, FAC systems looked ok. After combing FMCS remotely following persistent alarms, end station pumps were alarmed out, and chillers were naturally in run inhibit. Because I could not reset remotely, I went to site to investigate. I observed that both VFD's on the EX CWP were not receiving power. The electrical service at EX in the AHU was not tripped, but the power light was not illuminated. I reset both and they stayed on without issue. Revisiting the VFD's confirmed they were back online. These issues were mirrored at EY, and the solution was the same.

Despite the same alarms at the corner, there were no observable issues with the corner chilled water system. Nor were the fire pumps running. Mid stations seem to be insulated from any issues or alarms. As of 12:45p FAC systems have returned to their nominal state.

R. McCarthy T. Guidry

Here is what we know is not running (in no particular order)

h1cdsrfm (end station adnaco boxes not responding)

ncalex

brs_ex, brs_ey

hws_ex, hws_ey

mains_ex

chiller_ex, chiller_cs

MSR timing master 9th port has no link, missing DTS fanout?

Following the power issue I came out to the site to check on the purge air, pump carts, RGAs, etc.

The Kobelco had an audible alarm, but was still running. Since the interface screen is dead I could not see what the exact alarm was. Drying towers were also not running, though the dewpoint monitor was still reading -75C. I isolated the purge air at the ball valve past the drying towers, then reset both the compressor and the drying towers. I let the system run isolated for ~30 minutes to make sure dewpoint was stable, then opened the line back to the LVEA.

The CP1 turbo was still running, but the solenoid valves on the foreline had closed. I closed the gate valve to the CP volume, and closed the manual o-ring valve on the turbo foreline. Then reset the cart to open the solenoid valves. Once foreline pressure was stable I then opened the foreline o-ring valve and finally opened the CP gate valve. The pressure internal to the cryopump quickly stabilized, and the pressure never got above 1E-7 Torr since the turbo was still spinning. See attached striptool plot (Thanks Gerardo).

Next up was the XBM turbostation which had tripped completely, the turbo/forepumps/and solenoid valves all tripped, so again I closed the gate valve to the XBM and restarted the turbo station. Once the turbo was back to full speed and pressures on either side of the gate valve looked ok, I opened the gate valve.

The three aux carts in use (GV5 annulus/corner rga/HAM6 rga) all had the pumps still running but the solenoid valve had tripped and closed. Pressures were all still ok (<1e-4 Torr) so I reopened the solenoid valves, waited for pressure to drop and stabilize, then set the carts back on the interlocks. RGA trees were isolated from aux carts and were pumped by small ion pump, so no potential backstreaming into RGA volumes.

Finally, I went to the VPW to check on the vacuum ovens. VBOB had no issues, the turbopumps, forepump, and heaters were all running normally. VBOD is vented and was not baking, though the RGA scroll pump had tripped off, turbo was still running. I closed the foreline valve, and restarted the scroll pump and then opened the foreline valve once the foreline pressure was ok. VBOC had a couple issues, the RGA scroll and turbo both tripped off, so I isolated the turbo and restarted the scroll pump first then spun up the turbo and opened back to the RGA tree. The oven heaters had also tripped off. The oven was baking at 150C and the temp had dropped to 130C by the time I arrived, so I reset the heaters and made sure temperature was ramping back up to 150C.

I also verified all of the chamber cleanrooms were still running ok (HAM1/2, HAM3, biergarten, BBSS test stand cleanroom, BSC2 fan bank, and HAM6 cleanroom).

Leaving site now.

We are investigating a power issue on site which took down all the front ends. We are starting the recovery process.

Initially we thought it was contained to the MSR, but the DTS, Mech Room and end stations are showing problems and reboots.

Erik is on his way to the site. Jordan is on site looking at the vacuum system.

WP13772 TWO Offload

Dave

This morning I started the copy of TW0's raw minute files which have accumulated since 09dec2025 from local SSD-RAID to archive spinning media (h1ldasgw0).

As was done for the last offload in December 2025, following the DC0/FW0 consolidation done 18nov2025 by Jonathan, I am again using h1daqtw0 itself to do the copy. Normally the offload is done by the framewriter, which is still the case on the 1-leg.

The old files were frozen at GPS 1464188058 (07:54:00 Sat 30 May 2026 PDT) prior to offload.

NDS0 was restarted at 08:03 with a temporary configuration to serve the past 6 months of data from its temporary location on TW0 while the copy proceeds.

The copy was started on TW0 at 08:54. It usually takes just over a day to complete.