Looks like h1seih16 has had another kernel-panic event. This time it has caused the DAQ data for related front ends to become marked as bad (CDS overview attached).
Similarly to earlier crashes like this, nearly but not all of the front end computers sharing the same 10GE port on the DAQ data concentrator has bad DAQ data. In this case h1seih16 shares its h1dc0 port with psl, all corner station sus and seismic, and corner station sus-aux (except susauxb123). So it is not known why sush2b and susauxh56 are OK.
At this point there is no reason to think that the real-time models are not running correctly, we have just lost EPICS controls of h1seih16 and the DAQ data from those affected systems.
Would the first person to arrive on site please call me and we can recover the systems.
opened ticket FRS11749
Thomas, Dave:
we recovered the system from the crash at 11:30 PST. Thomas took a photograph of h1seih16's console (attached), I disabled the dolphin switch port remotely (testing my new scripts in the process) and then Thomas pressed the front panel RESET switch. When h1seih16's mx_stream started, the bad-DAQ status on all the other front ends automatically cleared (no restart_streamers were needed).
We were able to relock to nominal low noise as well so the situation seems to be ok for now.