Following the power issue I came out to the site to check on the purge air, pump carts, RGAs, etc.

The Kobelco had an audible alarm, but was still running. Since the interface screen is dead I could not see what the exact alarm was. Drying towers were also not running, though the dewpoint monitor was still reading -75C. I isolated the purge air at the ball valve past the drying towers, then reset both the compressor and the drying towers. I let the system run isolated for ~30 minutes to make sure dewpoint was stable, then opened the line back to the LVEA.

The CP1 turbo was still running, but the solenoid valves on the foreline had closed. I closed the gate valve to the CP volume, and closed the manual o-ring valve on the turbo foreline. Then reset the cart to open the solenoid valves. Once foreline pressure was stable I then opened the foreline o-ring valve and finally opened the CP gate valve. The pressure internal to the cryopump quickly stabilized, and the pressure never got above 1E-7 Torr since the turbo was still spinning. See attached striptool plot (Thanks Gerardo).

Next up was the XBM turbostation which had tripped completely, the turbo/forepumps/and solenoid valves all tripped, so again I closed the gate valve to the XBM and restarted the turbo station. Once the turbo was back to full speed and pressures on either side of the gate valve looked ok, I opened the gate valve.

The three aux carts in use (GV5 annulus/corner rga/HAM6 rga) all had the pumps still running but the solenoid valve had tripped and closed. Pressures were all still ok (<1e-4 Torr) so I reopened the solenoid valves, waited for pressure to drop and stabilize, then set the carts back on the interlocks. RGA trees were isolated from aux carts and were pumped by small ion pump, so no potential backstreaming into RGA volumes.

Finally, I went to the VPW to check on the vacuum ovens. VBOB had no issues, the turbopumps, forepump, and heaters were all running normally. VBOD is vented and was not baking, though the RGA scroll pump had tripped off, turbo was still running. I closed the foreline valve, and restarted the scroll pump and then opened the foreline valve once the foreline pressure was ok. VBOC had a couple issues, the RGA scroll and turbo both tripped off, so I isolated the turbo and restarted the scroll pump first then spun up the turbo and opened back to the RGA tree. The oven heaters had also tripped off. The oven was baking at 150C and the temp had dropped to 130C by the time I arrived, so I reset the heaters and made sure temperature was ramping back up to 150C.

I also verified all of the chamber cleanrooms were still running ok (HAM1/2, HAM3, biergarten, BBSS test stand cleanroom, BSC2 fan bank, and HAM6 cleanroom).

Leaving site now.