Dave, Thomas, Danny, Jamie, Dan Brown
While attempting to re-lock the IFO, the dolphin network crashed and all the codes needed a restart. we called Dave and he gave us instructions on how to log in and restart:
ssh controls@{computer}
then we run /etc/startWorld.sh
Everything is green on the cds overview screen now.
I've opened FRS11750 for this crash. We will fill in the lost-time after the IFO has been recovered.
Looking at the CDS overview posted in the alog (taken soon after the crash while h1susb123 was being recovered), there are several things to note:
h1sush34 may have been OK, only timing errors not FEC errors.
h1seib3 looks like it was OK, only IPC errors.
h1seib2 has both IPC and IOP-DAC errors, which eventually required a restart to clear.
h1oaf1 looks perfectly OK, which is surprising since one would expect at least IPC errors.
During the recovery h1ioplsc0 was restarted, and Thomas reported that h1oaf1 did not show any errors when this was done (this is a slave to the h1lsc0 timing master). This sounds suspicious and I would be wary of h1oaf1's state.