At 10:08 UTC (03:08 PDT) it looks like h1lsc0 kernel panic'ed. Strangely we are getting bad DAQ data status from a strange collections of machines (all end stations which are on dolphin, corner ISC and susauxb123!) see attached image.
h1lsc0 console is showing a kernel panic error (I'll post it later).
I'll recover all systems execpt LSC before we start today's upgrade.
Opened corresponding to FRS Ticket 11528. Likely we can quickly close the ticket, since the system has been recovered, but I leave it open for Dave to comment as he sees fit.
Here is the list of things I did until I was able to reset the DAQ status of these front end machines:
h1susauxb123: start streamers
h1lsc0: power computer down (after disabling its ix-dolphin switch port)
DAQ: restart
h1susauxb123: computer reboot. This system came back with no errors
h1asc0: computer reboot: This system came back with no errors
At this point, I could fix the DAQ mx stream on all the other models just by issuing a /etc/start_streamers.sh command.