Marc and I turned on the HAM7 Hi Voltage Bypass in the Mech Rm Mezzanine. We first turned the PSAMS down to zero, they hadn't been zero'ed before Hi-voltage was turned off. WP13282
Belated alog from Friday.
After realigning IMC and IM2 on Thursday, we continued aligning IM3 to put the beam in front of the PRM to the nominal position, IM4 to set the beam position in front of PR2, and PRM so the beam retroreflects. IM1 was left untouched, so the only change in IM1-IM2 line is from the change in the IMC alignment.
Clipping on IM4 baffle (HA12) disappeared completely. Beam position on IFI output baffle looked OK. Beam position on IFI input baffle didn't change (as expected) and still looked OK. There are some mystery
We saw flashes on ASC REFL sensors but they were weak. We tried to center them using RMs but RM2 seems to rail in YAW.
| Before P, Y [urad] | After P, Y | difference P,Y | |
| RM1 | -196, 281.8 | -242, -19.3 | -46, -301.1 |
| RM2 | 910, 490 | 912.1, -2125.15 (railing) | +2.1, -2615.15 |
Scan PRM to make sure that we're NOT looking at some kind of ghost beam on ASC REFL sensors.
Assuming that we're looking at the real beam on REFL sensors, we can do either (or some of) these:
See screenshot for before/after slider values for RM, IM and other optics.
Baffle pictures. Whenever appropriate, there are two pictures (one to access PIT and the other to access YAW) to avoid the parallax problem as the thickness of the baffles cannot be ignored.
IFO REFL looks good on the IFO REFL baffle (HA13) in front of the HAM2-HAM1 septum window (IFO_REFL_HA13.jpg).
IM4 baffle (HA12) is good in PIT (IM4_HA12_PIT_good.jpg), not great in YAW but is OK (IM4_HA12_YAW_OFF.jpg). Next time I'll take a picture through IM4 to show you it's quite acceptable than it looks from these pictures.
IR picture of the IM4 baffle (HA12) shot from IM3 side shows a ghost beam it catches when there's a flash (IM4_HA12_ghostbeam.jpg). Circled in red is the ghost beam (in this case it was 10 mode flash). Circled in green are things that are always visible (IM4_HA12_noflash.jpg) and aren't the ghost beam.
IFI input beam position is great in PIT, not great in YAW (IFI_INPUT_PIT_GREAT_YAW_OFF.jpg, IFI_INPUT_PIT_GREAT.jpg).
However, I still see some kind of clipping of bacdkward-going beam once in a while. See the IR photo shot from IM3 direction IFI_INPUT_clipping.jpg. You're looking at the baffle aperture through calcite wedge from the direction of IM1. Compare with IFI_INPUT_noflash.jpg where there was no flashing.
Not every flash shows this clipping-like thing, it seems to happen less frequently than the flashing.
To make things more confusing, there's a reflection of something shiny in IFI and the edge of that shiny thing is almost on top of the edge of the IFI input baffle aperture so it's hard to say what's going on. Nothing like this was visible on the input side facing IM2.
We temporarily moved IM1 to move the beam spot on IM2 more centered (IFI_INPUT_IM1movedtocenter.jpg) and I can see something closer to the center (probably scattering from the CWP?) but no clipping. So probably it's a good thing to move IM1 counter-clockwise as I indicated in the alog above. We cannot do it too much as we rail RM2 in the opposite direction, but we can probably move the beam by a mm or so on IM2.
Both CO2X and CO2Y Chillers turned back on with no issue. Interestingly as the laser is off they are currently heating the water.
All CDS systems have been recovered with the exception of EY dust monitor. The EY Comtrol serial-to-ethernet is not working, Erik is replacing its power supply.
Erik started the HWS code. At EX he powered up the h1cdsrfm expansion chassis and power cycled the BRS_EX. At EY he powered the expansion chassis.
We started h1cdsrfm which has cleared all the long-range-dolphin IPC errors. EY WAP just needed powering up via the Pulizzi.
Patrick started the code on the Beckhoff systems for CHILLER_CS, CHILLER_EX, BRS_EX, BRS_EY, NCALX and MAINS_EX.
EDC is currently down to 16 disconnected channels, all EY_DUST.
J. Kissel" As is all over the aLOG this morning, LHO had back-to-back site-wide power-outages this past Saturday 2025-05-30 -- LHO:90395. Here's the quick, pain-free record of recovering the SPI pathfinder's function to the levels we left it on Friday 2026-05-29 (LHO:90382). LASER State of laser as I found it (entering the lab with laser safety googles ON): - Yellow chain and signage across the entry warning of "laser running unattended" still blocking entry. - Magnetic lock of "outer" curtain released, manually hook-locked "inner" curtain still closed. - Laser hazard sign OFF - Laser STOP Button still "OUT" (in the ON position) - Laser Hazard sign still keyed "ON" - NPRO shuttered, still keyed ON, but not powered. In short -- everything worked as expected in the event of a power outage if it happened while a laser system was running unattended. RECOVERY (all with laser safety glasses ON; obviously but just saying it "out loud"): - Unchained and stored "unattended laser" sign. - Pushed in ("OFF") laser stop button. - Keyed OFF laser hazard sign. - Pulled out ("ON") laser stop button to allow for re-energizing of interlock. - Keyed ON laser hazard sign. - Closed magnetic "outer" curtain and swiped in, laser hazard sign lit up as expected. - Confirmed NPRO laser output is shuttered. - Looked at NPRO controller, red light ON under the OFF button (as expected because it was already/still keyed ON). - Plugged in and turned on PD100D interface for on-board S304C power monitor PD (installed within the laser conditioning/fiber collimating bread-board), confirming 1064 [nm] wavelength readout settings. - Waited ~2 minutes for the diodes' controllers to settle, then hit the green ON button - No issues with diode temp transients, laser worked just fine. - Unshuttered laser, and on-board S304C power monitor PD happily confirmed the expect ~198 [mW]. ELECTRONICS State of electronics as I found it: - Electronics +/-18 V and +/-24.0 V power supplies were ON, with their outputs producing voltage at their nominal settings. - Laser prep chassis and PD TIA chassis all showed happy green lights. - Keysight 33600A RF source had reset, with outputs off and all the wrong settings. RECOVERY (all with laser safety glasses ON; obviously but just saying it "out loud"): - Ran through T2600039 to restore all frequency settings - Exported March 2026 notes to LHO:90409 and restored CH1 (REF) and CH2 (MEAS) amplitude settings to -1.7 and -1.4 [dBm], respectively - Turned CH1 and CH2 output ON. Laser Prep Chassis SM05PD01A monitor PD ADC voltage: 7.83 [V]. Laser Prep Chassis RF monitors M2 (REF) and M3 (MEAS) ADC voltage: ~750 and ~750 [mV] FBR_PWRIN_REF On-board power monitor PD ADC voltage: 4.68 [V]. In short :: All these monitor ADC voltages are at normal values. SPI IFOs No actions required after turning on LASER and ELECTRONICS. REF IFO Efficiency -- 4.96 V / (2 * 2.69V) = 0.922 MEAS IFO Efficiency -- 3.00 V / (2*3.95 V) = 0.38 Recovered! Hopping back into The Mystery Machine...
Gerardo, Jordan, Camilla.
On Thursday we assembled VAC sealing part of the CHETA VPs, D1700340-type02. The thinnest part of the optics are at 12o'clock if the VP is oriented with the P/N of D1700338 writing also ~12o'clock.
D1700338-v2-001 with D1700339-v2-001 and D1700338-v2-003 with D1700339-v2-003. Both have optics D1100439-type02, which are DAR coated for 1064nm and 10.6um.
Next steps is VAC testing, then first contact the window surfaces and add secondary window.
TITLE: 06/01 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 7mph Gusts, 5mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY: Will continue recovery of various sytems following power outages over the weekend and will work on discovering what still needs attention.
Current status:
Thankfully no more power outages following the second (14:27) outage Sat 30may2026. Here are the systems we left offline which will be restarted tomorrow, Mon 01jun2026.
PSL Diode Room Beckhoff Controls
Long range dolphin end station Adnaco expansion chassis (h1cdsrfm)
NCAL EX
BRS EX, EY
HWS EX, EY
MAINSMON EX
CHILLERMON EX, EY
Here are all the MSR UPS Emails we received on Saturday. The UPS clock appears to be incorrect and about 14mins fast. No GC UPS emails were received on Saturday.
Sat Morning Outage 09:39
Date: 05/30/2026
Time: 10:12:59
Code: 0x0002
Critical - System: Warmstart. (2 copies received)
Date: 05/30/2026
Time: 10:13:06
Code: 0x0101
Informational - UPS: Restored the local network management interface-to-UPS communication.
Sat Afternoon Outage 14:27
Date: 05/30/2026
Time: 14:45:38
Code: 0x020F
Critical - UPS: An input voltage or frequency problem prevents switching to bypass mode. (4 copies received)
Date: 05/30/2026
Time: 14:45:40
Code: 0x0109
Warning - UPS: On battery power in response to an input power problem. (2 copies received)
Date: 05/30/2026
Time: 14:45:45
Code: 0x010A
Informational - UPS: No longer on battery power. (2 copies received)
Date: 05/30/2026
Time: 14:45:57
Code: 0x0210
Informational - UPS: An input voltage or frequency problem no longer prevents switching to bypass mode. (2 copies received)
h1hwsex and h1hwsey are running in a tmux session. To get to the process, ssh as controls to the server and run "tmux a".
The second outage took the end station chillers down again. Jordan V restored power to the CWPs, and I cleared all alarms remotely. Temps rose a couple degrees at each end to just under 70f. Those should correct quickly, if not already. R. McCarthy J. Vanosky T. Guidry
After the failure of the compressors HMI last week, see alog 90318. A tech from Rogers Machinery (Brandon) came on site today to install a new HMI unit. The compressor had to be powered off to make the swap, so I isolated the purge air from the LVEA at the ball valve just past the drying towers.
The HMI swap was pretty quick only took about 10 minutes, and we can now see the temperatures and pressures in the compressor. I let the purge air skid run for ~10 minutes to make sure the dewpoint was stable, at -75C, then opened back to the LVEA.
The replacement oil pump should be delivered next week, at which time a tech will have to come out again and make the swap.
In the two power issues today Erik had to manually start fmcs-epics-ioc. It is a mac mini in a custom rack container that makes it hard to see if it is on or to power cycle it. This machine causes issues at each power outage.
After the second outage I put a VM together to host a containerized version of the code. It is staged and ready to go, but not tested. If it goes down again we go onto the new system and start the container. It should allows to get the fmcs going without sending someone to fiddle with the machine.
To do this:
1. log into fmcs-epics-ioc.
2. systemctl start deb10ioc
To make it perminant we need to modify the systemd file to have an install section, do a daemon-reload and a systemctl enable.
Ultimately we need to rebuild this ioc on a new OS release.
This morning we tried a power cycle of both the computer and the IO Chassis, with the chassis down for about 40 seconds. This did not bring the 3rd Adnaco BP back so we replaced it.
This afternoon we tried a 4 minute power down of the IO Chassis, and when the computer came up it saw all the cards. So perhaps the backplane removed from the IO Chassis this morning is good.
As before all front end computers powered down with the exception of cdsh8 and psl0. When we powered them back up all recovered except h1sush12, which has again lost its 3rd Adnaco backplane in its IO Chassis, even though this is a new BP installed this morning.
HWS at end have gone down, check. 9th port on timing master (dts) is down, check. FMCS EPICS is down, check.
this time h1cdrfm's adnaco expansion chassis at the end stations have not gone down though.
Looks like the PSL Diode Room Beckhoff has gone down this time.
Following the power issue I came out to the site to check on the purge air, pump carts, RGAs, etc.
The Kobelco had an audible alarm, but was still running. Since the interface screen is dead I could not see what the exact alarm was. Drying towers were also not running, though the dewpoint monitor was still reading -75C. I isolated the purge air at the ball valve past the drying towers, then reset both the compressor and the drying towers. I let the system run isolated for ~30 minutes to make sure dewpoint was stable, then opened the line back to the LVEA.
The CP1 turbo was still running, but the solenoid valves on the foreline had closed. I closed the gate valve to the CP volume, and closed the manual o-ring valve on the turbo foreline. Then reset the cart to open the solenoid valves. Once foreline pressure was stable I then opened the foreline o-ring valve and finally opened the CP gate valve. The pressure internal to the cryopump quickly stabilized, and the pressure never got above 1E-7 Torr since the turbo was still spinning. See attached striptool plot (Thanks Gerardo).
Next up was the XBM turbostation which had tripped completely, the turbo/forepumps/and solenoid valves all tripped, so again I closed the gate valve to the XBM and restarted the turbo station. Once the turbo was back to full speed and pressures on either side of the gate valve looked ok, I opened the gate valve.
The three aux carts in use (GV5 annulus/corner rga/HAM6 rga) all had the pumps still running but the solenoid valve had tripped and closed. Pressures were all still ok (<1e-4 Torr) so I reopened the solenoid valves, waited for pressure to drop and stabilize, then set the carts back on the interlocks. RGA trees were isolated from aux carts and were pumped by small ion pump, so no potential backstreaming into RGA volumes.
Finally, I went to the VPW to check on the vacuum ovens. VBOB had no issues, the turbopumps, forepump, and heaters were all running normally. VBOD is vented and was not baking, though the RGA scroll pump had tripped off, turbo was still running. I closed the foreline valve, and restarted the scroll pump and then opened the foreline valve once the foreline pressure was ok. VBOC had a couple issues, the RGA scroll and turbo both tripped off, so I isolated the turbo and restarted the scroll pump first then spun up the turbo and opened back to the RGA tree. The oven heaters had also tripped off. The oven was baking at 150C and the temp had dropped to 130C by the time I arrived, so I reset the heaters and made sure temperature was ramping back up to 150C.
I also verified all of the chamber cleanrooms were still running ok (HAM1/2, HAM3, biergarten, BBSS test stand cleanroom, BSC2 fan bank, and HAM6 cleanroom).
Leaving site now.
Well, it's Groundhog Day...again.
Came out to the site after the 2nd power glitch. The same pumps had the same issues found for the morning power glitch. I have reset the Kobelco, CP1 pump cart, aux carts, XBM turbopump, and the VPW vacuum ovens. Same procedure as above. Attached is a picture of the CP1 pump cart status and the XBM turbostation as found after the power glitch.
J. Freed
For the build of SPI, I measured the expected power output from a dual source Keysight 33600A through SPI's 2W amp D2500004. These measurements were taken with a E4418A EPM power meter using the HP 8484A 30dB attenuator attachment. After analyzing the data it is hard to say that the power meter power readings were accurate but the power the AOMs are receiving should be below the damage threshold of 33dBm.
I measured the power at 2 points in the chain from Keysight -> 2W -> SPI prep, for both channels. (SPI_Build_RF_Diagram.png) The first point is just before the 2W amp, and the second point is just before the SPI prep for both channels. Then I compared that with what the output monitor voltage was showing on SPI prep for both channels.
Power_Before_2W_Amp.png Is the data collected from power measurements just before the 2W amplifier. The limit of 22dBm is the limit of the keysight. From this it can be seen that there is not a 1-1 measurement between what the keysight says it is outputting and what the HP 8484A sensor is measuring. This difference is also changing based on the input power itself. With lower power overestimating what the keysight says its outputting, and higher power underestimating what the keysight says its outputting. This is a problem with the probe as switching out the probe to an E-series gives a relatively consistent error of -0.2dB between what the keysight says its outputting and what is being measured. Unfortunately the HP 8484A sensor is the only probe that can measure up to the 32-32.5dBm needed for SPI. Interpolating the results of the HP 8484A sensor up to 32 dBm gives approximately a 0.4dB underestimation of the power.
Then I took a measurement of the power after the 2W amp at the end of the cable that would plug into SPI prep. Namely I adjusted the power from Keysight, such that the output measured on the probe was 32dBm which is what SPI prep is expecting accounting for any possible underestimation of the power. Note that power past the 2W amp decreases over time (by about 0.5dB total) as the device warms up so I had to wait a couple of hours after start-up to take this measurement. The results were:
AMP 1 (Which goes into the ref path) measured 32dBm when the keysight displayed: -1.7dBm
AMP 2 (Which goes into the Meas path) measured 32dBm when the keysight displayed: -0.3dBm
Be aware that that 32dBm has an error of +/- 0.5dB due to the factors listed above.
Then using the fact above, I found that the whole AMP 1 chain has a gain of 33.7 dBm from what the keysight displays. And AMP 2 chain has a total gain of 32.3dBm from what the keysight displays. Using this fact I could find the voltage associated with each of the monitor channels of SPI prep.
Monitor_read_AMP1_REF.png and Monitor_read_AMP2_MEAS.png show the monitor reading voltage vs the approximate input RF power into SPI prep. for both channels. Interpolating the results gives a calibration equation between RF power and monitor voltage of:
AMP 1 (REF): [V] = -0.0249*[dBm] + 1.5480
AMP 2 (MEAS): [V] = -0.0246*[dBm] + 1.5387
With both channels reaching 32dBm around 0.75V on the monitor channel this gives an easy target for controling the power.
J. Kissel
Easy look up table after setting up Keysight 33600A per T2600039, and setting the amplitudes as we have had for most of the build ::
CH1 = AMP 1 = REF = M2 MON
f = 79.995904 MHz (79,995,904 Hz)
amp = -1.7 dBm
M2 MON = 740 mV
CH2 = AMP 2 = MEAS = M3 MON
f = 80.000000 MHz (80,000,000 Hz)
amp = -1.4 dBm
M3 MON = 755 mV
Remember to turn the output ON for each channel.