We think the worst of the storm may be over. I'll let TJ fill in the details, I know of at least three glitches which took down front end models. The first just took out the ISC models at the end stations (suspect timing card power supplies), the second (at 8:40pm) was the biggest and it rebooted every front end computer. After that most front ends came back up after the reboots, h1ioppsl0 had a timing excursion which took about 15 minutes to clear. At that point the only problem was bad DAQ data from both PEM systems at the midstations. A further glitch caused h1ioplsc0 to DAC error, I restarted all the LSC models to clear this.

The Beckhoff slow control machines also rebooted. All code restarted except for the three PLCs at EX. These have just been started.

At time of writing, the only computer issues I see is the bad data from the midstation PEM front end computers to the DAQ. If memory serves, these send their DAQ data back to the end stations via ethernet-fibre_optics converters, and then piggy-back on the main switch-to-switch link to the MSR. Maybe the converters need a power cycle.