Jim, Cyrus, Dave

The large wind event was preceded by several power glithches which impacted on the DAQ and killed the front end computers.

After waiting to ensure that the power was stable again, we remotely (via management port) power cycled the FE computers at the end and mid stations and reset the MSR computers. Generally, computers not on the dolphin networks started themselves, some needed a power cycle. Once all the computers were all booted, they all started their models. At that point we discovered the Dolphin IPC in the MSR was non-operational. We suspect the glitchy nature of the outage put the Dolphin switches in a bad state. We stopped all models running on MSR computers attached to the Dolphin network (all but the PSL and SUSAUX). Power cycled the Dolphin switches. Rebooted the FE computers via front panel RESET button. Some models did not autostart and needed their "BURT_RESTORE" button pressed, which we did.

The DAQ was showing bogus data for slow channels (e.g. wind speed below 10mph when it was 50mph outside), so a clean restart of the DAQ was done. The NDS machines took many minutes before they got started, not sure why at the momemt.

Two systems started with a IRIG-B timing signal about 400 (should be 15), which then drifted down to nomimal over 20-30minutes. These were h1sush34 and h1iscey. We allowed these to become good rather than power cycle their IO Chassis.

Once the models were running, stable and had communication, I enabled the SWWD systems to drive the DACs.

Shiela is handling the recovery of the Beckhoff and PSL.