Today at 10:40 we shut down the MSR UPS (Work Permit 4366). On power up its commication error did clear for a while, but then returned.

The main impact on CDS uses was:

• loss of the timing master
• operator workstation and alarm handler machine lost power
• name servers were shutdown, preventing operator station reboot
• the GC virual server for alog, svn, bugzilla was glitched and stopped running

The biggest impact was the loss of the timing system, which required all front ends and their IO Chassis to be power cycled. The sequence followed was:

• power down front end computer (hard shutdown, no need to worry about Dolphin glitching)
• flip switch OFF on front of IO Chassis
• remove 24V DC power cord from rear of IO Chassis, then reattach
• flip switch ON on front of IO Chassis
• verify timing slave is operating correctly (1Hz led flash)
• power on all front end computers

All systems burt restored to their safe.snap settings. I manually restored the PSL models to the settings as of Noon Saturday 4th January.

With Jonathan on phone support, we rebooted the GC machine and Jonathan restored the services (see Jonathan's alog)

We power cycled all the front ends in the DTS. The staging building test stands were already powered down for the holidays and stay off. The LVEA test stand systems have been powered down for some time while those systems are moth-balled until needed again.