At 09:46:04 PDT h1iopsush34 suffered a large timing glitch. All models on this front end computer stopped running.
I've captured the logs are have restarted the models with /etc/statWorld.sh
Jeff K, Nico, Dave:
h1iopsush34 is not recovering. On first startWorld try it saw all the 18bit DACs but the model did not get going. I started the IOP by hand a second time and it only autocal'ed 5 of the expected 6 18bit DACs. This was confirmed by lspci. Before calling EE out, we will try a power cycle of the IO Chassis, I powered down the front end and Jeff K is power cycling the IO Chassis.
h1sush34 ~ # lspci -v |grep 3101
Subsystem: PLX Technology, Inc. Device 3101
Subsystem: PLX Technology, Inc. Device 3101
h1sush34 ~ # lspci -v |grep 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357