I received a call from Oli at 11:46 this morning informing me that water pressure at the building had fallen. I logged into the FMCS remotely and found that the potable holding tank had been completely drained and only one RO was making water. Before leaving home to come out to site, I contacted Randy and had him manually disable the domestic pressure skid and force the second RO skid on. At this point, because of the known leak, Richard and I have decided that the RO system will replenish the potable holding tank but that the water supply will remain valved out so as not to continue to tax the entire water treatment plant. 

C&E Trenching who had already been slated to come make the repairs next week has been contacted. After explaining the current situation, they have graciously agreed to move us up in their schedule and will be on site Monday Morning to being the work. Once the repair is made, water should be restored.

T. Guidry

Today's activities:

- All the turbos have been valved out and switched off
- The corner was vented through the Y-manifold vent valve, with 3-4 Torr/min speed - it took 3.5 hrs
- As a side effect, the current of GV5's AIP went up, so now this Annulus volume is being pumped with an aux cart
- Another aux cart is pumping the RGA in the corner (between HAM5 and HAM4)

TITLE: 04/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: Done early for today! The corner is vented! It's possible that not everything got finished today due to a lack of water pressure causing people to leave early, but at least all of the most important tasks have been completed, if not all of them.

• DONE VAC - Checklist for corner vent
• DONE VAC - Vent corner
• ? VAC - 1000 AMU RGA work
• DONE FAC - Wipe down and crane 1st gantry over IMC tube and fit check at HAM1
• DONE FAC - Wipe and install platforms around HAM1
• ? FAC - Cleanup furniture in South Bay to make room

LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

Fri Apr 04 10:06:13 2025 INFO: Fill completed in 6min 10secs

TITLE: 04/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 6mph Gusts, 4mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

Today's a pretty light day, mostly just venting the corner and some work around HAM1 to prepare for the ISI installation. Here's what's planned:

• VAC - Checklist for corner vent
• VAC - Vent corner
• VAC - 1000 AMU RGA work
• FAC - Wipe down and crane 1st gantry over IMC tube and fit check at HAM1
• FAC - Wipe and install platforms around HAM1
• FAC - Cleanup furniture in South Bay to make room

Today's activities:

- HAM1's Y+ and Y- doors have been taken off (thanks to Tyler, Randy for the help in the door removal, and to Tony and Corey for breaking the bolts earlier)
- All 5 HAM1/HAM2 viewport covers have been installed
- The 16.5" flange was taken off from the top of HAM1 - this will be the basis of the new vacuum interlock gauge
- A high number of leak checking was done, in detail:
 - Y-manifold: Y+ X+ side flanges (3 pcs.); Turbo flange (1 pc.)
 - BSC8: Y+ X+ corner (3 pcs.); Y- X+ corner (3 pcs.); X+ HEPI bellows (2 pcs.); X+ collar 16.5" flanges (2 pcs.)
 - BSC8-BSC1 spool: Y+ X+ flanges (3 pcs.)
 - BSC1: Y+ X+ side flanges (3 pcs.); X+ collar flanges (2 pcs.); X+ HEPI bellows (2 pcs.); X+ door flanges (5 pcs.); Y- X+ side flanges (3 pcs.)
 - BSC2: Y+ X+ side flanges (3 pcs.)
 - BSC3: Y+ HEPI bellows (2 pcs.); Y+ X- side flanges (3 pcs.); Y+ collar flanges (2 pcs.); Y+ door flanges (5 pcs.); Y+ X+ side flanges (3 pcs.)
 - HAM2-HAM3 spool: vent port (1 pc.); X- flanges (11 pcs.); Y- side flanges (2 pcs.); X+ Y- flanges (5 pcs.)
 - HAM3: Y- door flanges (5 pcs.)

Beginning early last week water usage, specifically what is delivered to the buildings following treatment, spiked. Not only was our consumption far outside of our normal usage during operating hours, but it was significantly higher than we would expect in off hours/weekends. After tracking burn rates, I calculated an average continuous flow of ~1.25gpm. 

With the help of the operator core and Richard, Facilities began the process of trying to determine the source of the lost water. This usually means site walks both inside and out looking for running water/stuck valves/wet areas. Persistent rain was no help in determining a potential source outside and there was no apparent continuous/suspect flow inside. Given that, we began the process of elimination by isolating water supply to individual buildings. We began with the staging and VPW which share a shutoff valve but to no avail. The same was done at the LSB and LExC yielding the same result. Head scratching continued and I eventually stumbled upon a small wet area of asphalt just outside the Carpenter Shop near the potable water holding tank. It was not immediately clear to me that this would be the source of the leak since the rain was still intermittent and the presence of water was pretty unsuspecting given what we have likely put in the ground in that area. So, I snapped a photo and kept an eye on this area in the coming days. Each day that went by the spot did not dry and actually grew larger. At this point it is clear that the water leak is beneath the asphalt and judging by the as-built drawings is probably a failed 45 deg joint. 

Beginning early next week the asphalt will be removed and the surrounding area will be vacuumed out as opposed to dug so as not to have to interrupt the supply of high volts to the chillers etc. Once the failed area is identified potable supply will have to be cut off to the entire site while repairs take place.

E.  Otterman R. McCarthy C. Soike T. Sanchez T. Guidry 

WP 12408
ECR E2200043

Beckhoff electronics for the HEPI Controller Upgrade are now installed in the MER. Power and network cables were pulled from the Mechanical Room vacuum rack. Power to the electronics is on the vacuum UPS. Field cables to BSC2 are now terminated. Modifications to the motor control panel are still required. They will be done concurrently with the software updates as HEPI needs to be taken offline. Date for upgrade TBD.

I noticed that the 500nm counts were stuck so I restarted the dust monitor (didn't fix the stuck value), and restarted the IOC but it didn't come back after the restart, the dust monitor isn't writing 2 values correctly it seems? I might have to swap this to another dust monitor, I don't have any more spare pumped dust monitors so I think I would want to take one of the 2 currently in stagings' ground floor if possible. I also double checked that all the settings on the actual device were the same as the others. The 0.3 RAW counts seem to be still working confusingly.

The reported error is as follows

2025/04/03 15:16:59.169508 gt521s1 H1:PEM-CS_DUST_LVEA10_300NM_RAW_IN: Input "    ,   ,010,060,000..." does not match format "%*[-0-9]*[:0-9]*ff*f*f*f"

WP 12438
Rack Layout D1001425

RF cables for ASC REFL WFS were pulled to HAM1 flange D6. Cables need to be dressed into the ISC-R2 rack. Some electronics in ISC-R2 were moved to make room for a delay line and 2-Ch Demodulator. Delay line chassis is installed. Demodulator chassis is being assembled at CIT. Existing coaxial cables need to be tested for delay phase before being pulled out and re-terminated. At this time some patch panel work in ISC-R2 will also be done.

F. Clara, M. Pirello, T. Sanchez, D. Sigg

FInished updating the sensors for HAM1 ISI in the staging building. New CPS seem to be alive and noise is within spec, considering the state the ISI is in (on storage container base and locked). TMDs and vertical L4Cs will be added in chamber. We will swap to the narrower transport base next week in prep for moving to OSB. I will post asds for cps soon, screenshot isn't working on this laptop.

TITLE: 04/03 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

IFO is in PLANNED ENGINEERING for VENT

Extremely productive day in which the following tasks were completed:

• KIM - Stock garb room by HAM3 - DONE
• KIM/NELLIE - 2nd HAM1 Cleaning - DONE
• RANDY - Move forklift to HAM1 South bay area - DONE
• RANDY - Ameristat HAM1 HEPI grout areas - DONE
• 5 PEOPLE - Move HAM1 cleanroom into final place - DONE
• VAC/FAC - HAM1 Doors off at +Y tp HAM3 CR, -Y crane to High Bay Area - DONE
• GERARDO/TRAVIS - Install Septum VP covers in HAM1
• EE - POP Cabling - DONE
• VAC - HAM1 Remove 16" Top Flange forklift down for work
• VAC - HAM1 Top flange Guage work table side - GM?
• VAC - Checklist for corner vent
• VAC - Leak check Corner under vac

Attached is a photo of the most recent SUS/SEI guardian configurations.

Oli is taking over the shift for the remaining 2 hours.

LOG:

M. Todd, C. Compton

We started setting up the Access CO2 laser in the optics lab, and after confirming that the beam coming from the aperature has not changed since previous tests, we have begun installing optics to test a setup we are imagining for CHETA.

Figure 1 is the proposed setup to test the QPD path. The ITM in this setup will really just be a screen or block, and we may do some razor-blade profiles of it. The QPD should be at the same Gouy phase as the "ITM", and this model adheres to that. So far, we have installed three of the steering mirrors and encorporated their measured distances (aligned to the optics grid) to this model, as seen in Figure 2.

Here are the beam parameters at the relevant points in the model, which may be confirmed with profiles.

Closes FAMIS 26500. Last checked in alog 83228

It is worth noting that while all ISIs were in ISI_DAMPED_HEPI_OFFLINE and I did this task during lunch with minimal LVEA activity and untripped WDs, we are in a vent so values may not be nominal. Despite this, the outputs are still comparable to last month's with the same +-1 number of out-of-range STSs/T240s.

STS Centering

Averaging Mass Centering channels for 10 [sec] ...
2025-04-03 12:42:01.264160
There are 1 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -2.34 [V]
All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.623 [V]
STS A DOF Y/V = -0.678 [V]
STS A DOF Z/W = -0.626 [V]
STS B DOF X/U = 0.241 [V]
STS B DOF Y/V = 0.946 [V]
STS B DOF Z/W = -0.394 [V]
STS C DOF X/U = -0.801 [V]
STS C DOF Y/V = 0.799 [V]
STS C DOF Z/W = 0.612 [V]
STS EX DOF X/U = -0.052 [V]
STS EX DOF Y/V = -0.052 [V]
STS EX DOF Z/W = 0.067 [V]
STS EY DOF Y/V = -0.077 [V]
STS EY DOF Z/W = 1.291 [V]
STS FC DOF X/U = 0.19 [V]
STS FC DOF Y/V = -1.11 [V]
STS FC DOF Z/W = 0.653 [V]
Assessment complete.

T240 Centering

Averaging Mass Centering channels for 10 [sec] ...
2025-04-03 12:44:13.270291
There are 14 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -1.086 [V]
ETMX T240 2 DOF Y/V = -1.059 [V]
ETMX T240 2 DOF Z/W = -0.718 [V]
ITMX T240 1 DOF X/U = -1.658 [V]
ITMX T240 1 DOF Y/V = 0.416 [V]
ITMX T240 2 DOF Z/W = 0.411 [V]
ITMX T240 3 DOF X/U = -2.126 [V]
ITMY T240 3 DOF X/U = -0.835 [V]
ITMY T240 3 DOF Z/W = -2.379 [V]
BS T240 3 DOF X/U = -0.504 [V]
BS T240 3 DOF Z/W = -0.355 [V]
HAM8 1 DOF X/U = -0.316 [V]
HAM8 1 DOF Y/V = -0.438 [V]
HAM8 1 DOF Z/W = -0.723 [V]
All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.079 [V]
ETMX T240 1 DOF Y/V = -0.051 [V]
ETMX T240 1 DOF Z/W = -0.05 [V]
ETMX T240 3 DOF X/U = -0.021 [V]
ETMX T240 3 DOF Y/V = -0.154 [V]
ETMX T240 3 DOF Z/W = -0.047 [V]
ETMY T240 1 DOF X/U = 0.052 [V]
ETMY T240 1 DOF Y/V = 0.13 [V]
ETMY T240 1 DOF Z/W = 0.194 [V]
ETMY T240 2 DOF X/U = -0.098 [V]
ETMY T240 2 DOF Y/V = 0.183 [V]
ETMY T240 2 DOF Z/W = 0.062 [V]
ETMY T240 3 DOF X/U = 0.21 [V]
ETMY T240 3 DOF Y/V = 0.079 [V]
ETMY T240 3 DOF Z/W = 0.105 [V]
ITMX T240 1 DOF Z/W = 0.258 [V]
ITMX T240 2 DOF X/U = 0.204 [V]
ITMX T240 2 DOF Y/V = 0.07 [V]
ITMX T240 3 DOF Y/V = 0.293 [V]
ITMX T240 3 DOF Z/W = 0.16 [V]
ITMY T240 1 DOF X/U = 0.043 [V]
ITMY T240 1 DOF Y/V = 0.088 [V]
ITMY T240 1 DOF Z/W = 0.027 [V]
ITMY T240 2 DOF X/U = 0.021 [V]
ITMY T240 2 DOF Y/V = 0.264 [V]
ITMY T240 2 DOF Z/W = 0.087 [V]
ITMY T240 3 DOF Y/V = 0.046 [V]
BS T240 1 DOF X/U = 0.107 [V]
BS T240 1 DOF Y/V = -0.274 [V]
BS T240 1 DOF Z/W = -0.208 [V]
BS T240 2 DOF X/U = 0.046 [V]
BS T240 2 DOF Y/V = -0.294 [V]
BS T240 2 DOF Z/W = 0.174 [V]
BS T240 3 DOF Y/V = -0.044 [V]
Assessment complete.

Closes FAMIS 27812. Last checked in alog 83486

CO2Y:

Before: 10.4ml

After: 10.4ml

Water Added: 0ml

CO2X:

Before: 30.1ml

After: 30.1ml

Water Added: 0ml

No water leaking in our leak detection system (aka dixie cup)

TL;DR - Don't worry about the new models and screens in /userapps/trunk/sus - Brian is developing these for the OSEM estimator project and none of these are (currently) linked to any observatory model.

--

FYI - As part of the project to see if we can improve the noise caused by the OSEMs, we are developing new models and medm screens for the HLTS. Tracking for this ECR is in FRS ticket 32526.  

We are running these at Stanford for testing, and putting them into userapps because we will be trying them on a triple at LHO soon (detail will be in a different log, they are still in flux a bit).

As such, I've created several new things in userapps. These are not linked to anything at either site - I'm putting this note here to reduce potential worry. These are still in development, so I suggest waiting a bit before you look at these.

New Files:

/userapps/trunk/sus/common/models/
HLTS_MASTER_W_EST.mdl   -  library part for M1 with the estimator calculation
SIXOSEM_T_STAGE_MASTER_W_EST.mdl - library part for upgraded HLTS model with the estimator included.

userapps/trunk/sus/common/medm/hxts/
SUS_CUST_HLTS_OVERVIEW_W_EST.adl - new overview screen with estimator

These files are copies of the existing parts as of March 2025, and I added _W_EST to the name of the copy.

userapps/trunk/sus/common/medm/estim/ - directory of new sub-screeens for the estimator

Updated files:

userapps/trunk/isi/common/models/
blend_switch_library.mdl - I added a new model to this library to allow smooth switching between the osem damping, the estimator damping, and OFF.

I'm running this from the new top-level model in

/userapps/trunk/isi/s1/s1suspr3.mdl

Yesteerday I noticed that the CS pump could only pull 18inHg (nominal 19) even with the bleed valve closed. There was lots of carbon dust around the exhaust and the filter tumb screws were tight, all signs that it was time for a rebuild. Today, Ryan C and I swapped it with the spare, which fired right up and pulled to well above 20inHg before I adjusted the bleed valve and brought it back to 19.

Two safety notes:

1. The out plastic on the spade connectors look like they have been damaged from recent swaps (attachment 1). Since we have to feed these through a 90deg elbow and then fish them out of the housing before pluging them in, I'm not surprised that it could have been damaged. We will talk with Fil or similar to find a better solution.
2. These pumps get very hot, >130F at times, and the foam it sits on to damp it has started to melt and burn (attachment 2). I'll find a new material for it to sit on and update.