We are investigating a power issue on site which took down all the front ends. We are starting the recovery process.
Initially we thought it was contained to the MSR, but the DTS, Mech Room and end stations are showing problems and reboots.
Erik is on his way to the site. Jordan is on site looking at the vacuum system.
FMCS has been mainly restored.
All the frontends have been power cycled, they all came back except h1sush12.
EDC required a power cycled of h1susauxb13 to get going.
h1sush12 has losts its 3rd Adnaco backplane in its IO Chassis (has the 4th ADC on it). Erik is in the CER replacing this.
Jordan reports this was a strange outage, with a seemingly random scattering of pumps impacted across the site.
We same something similar, most but not all FOMs down, but most workstations rode through.
Erik's swap of the 3rd Adnaco backplane on h1sush12 fixed the issue, we can now see the 4th ADC and the models are running.
I have untripped all the SWWDs with the exception of BSC2.
DAQ is good.