All five FE cores crashed at 11 mins past midnight Sun 1st April, this is not an april fools joke.
here is the dmesg capture before we reboot the machine.
[1426964.148086] h2pslfss: ADC TIMEOUT 1 31229 61 31293
[1426964.148088] h2psliss: ADC TIMEOUT 0 31229 61 31293
[1426964.148090] h2pslpmc: ADC TIMEOUT 2 31229 61 31293
[1426964.148092] h2psldbb: ADC TIMEOUT 3 31229 61 31293
[1426965.146334] h2ioppsl0: timeout 0 1000000
[1426965.146339] BUG: unable to handle kernel NULL pointer dereference at (null)
[1426965.146341] IP: [] fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146349] PGD 0
[1426965.146350] Oops: 0000 [#1] SMP
[1426965.146351] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:2b:01.0/class
[1426965.146353] CPU 1
[1426965.146353] Modules linked in: h2psldbb h2pslpmc h2pslfss h2psliss h2ioppsl0 open_mx mbuf
[1426965.146356]
[1426965.146358] Pid: 0, comm: swapper Not tainted 2.6.34.1 #6 X8DTU/X8DTU
[1426965.146359] RIP: 0010:[] [] fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146364] RSP: 0018:ffff8801beca7698 EFLAGS: 00010092
[1426965.146365] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000011e
[1426965.146366] RDX: 0000000000023c56 RSI: ffffffff81789ffd RDI: ffffffffa00311f2
[1426965.146367] RBP: ffff8801beca7ee8 R08: 000000007ffffff0 R09: 000000000000000a
[1426965.146368] R10: 0000000000000006 R11: 00000000ffffffff R12: 0000000000000000
[1426965.146370] R13: 0000000000000000 R14: ffffc900118d6000 R15: 0000000000000040
[1426965.146371] FS: 0000000000000000(0000) GS:ffff880001e20000(0000) knlGS:0000000000000000
[1426965.146372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1426965.146373] CR2: 0000000000000000 CR3: 0000000001a09000 CR4: 00000000000006e0
[1426965.146375] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1426965.146376] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1426965.146377] Process swapper (pid: 0, threadinfo ffff8801beca6000, task ffff8801bec7aac0)
[1426965.146378] Stack:
[1426965.146379] 0000271700002711 0000271700000006 000000010000290b 0000000200000003
[1426965.146380] <0> 0000003500000003 ffffc900118d7034 ffffffffa0033a80 0000000000000000
[1426965.146382] <0> 0000000000000000 ffffc900118bc000 0000000400000000 00000000000007f8
[1426965.146384] Call Trace:
[1426965.146390] [] ? cpumask_next_and+0x2c/0x39
[1426965.146394] [] ? cpumask_weight+0xc/0xe
[1426965.146396] [] ? find_busiest_group+0x36f/0x784
[1426965.146400] [] ? timekeeping_get_ns+0x16/0x38
[1426965.146402] [] ? apic_write+0x11/0x13
[1426965.146404] [] ? lapic_next_event+0x10/0x14
[1426965.146406] [] ? clockevents_program_event+0x75/0x7e
[1426965.146408] [] ? tick_dev_program_event+0x37/0xf7
[1426965.146411] [] ? enqueue_hrtimer+0x65/0x72
[1426965.146413] [] play_dead_common+0x6e/0x70
[1426965.146415] [] native_play_dead+0x9/0x20
[1426965.146417] [] cpu_idle+0x46/0x8d
[1426965.146422] [] start_secondary+0x192/0x196
[1426965.146423] Code: 00 8b 42 24 83 c8 01 89 42 24 8b 95 44 f8 ff ff 31 c0 8b b5 8c f8 ff ff e8 2a 13 00 00 48 8b 05 8f a7 00 00 48 c7 c7 f2 11 03 a0 <8b> 30 31 c0 e8 13 13 00 00 48 8b 05 78 a7 00 00 8b 00 66 85 c0
[1426965.146433] RIP [] fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146437] RSP
[1426965.146437] CR2: 0000000000000000
[1426965.146756] ---[ end trace d3abcc5123271f5d ]---
[1426965.146757] Kernel panic - not syncing: Attempted to kill the idle task!
[1426965.146758] Pid: 0, comm: swapper Tainted: G D 2.6.34.1 #6
[1426965.146759] Call Trace:
[1426965.146761] [] panic+0x73/0xe8
[1426965.146764] [] do_exit+0x6d/0x712
[1426965.146765] [] ? spin_unlock_irqrestore+0x9/0xb
[1426965.146767] [] ? kmsg_dump+0x115/0x12f
[1426965.146769] [] oops_end+0xb1/0xb9
[1426965.146772] [] no_context+0x1f7/0x206
[1426965.146774] [] __bad_area_nosemaphore+0x179/0x19c
[1426965.146776] [] bad_area_nosemaphore+0xe/0x10
[1426965.146778] [] do_page_fault+0xff/0x210
[1426965.146780] [] page_fault+0x1f/0x30
[1426965.146784] [] ? fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146788] [] ? fe_start+0xe24/0x214e [h2ioppsl0]
[1426965.146791] [] ? cpumask_next_and+0x2c/0x39
[1426965.146793] [] ? cpumask_weight+0xc/0xe
[1426965.146795] [] ? find_busiest_group+0x36f/0x784
[1426965.146797] [] ? timekeeping_get_ns+0x16/0x38
[1426965.146799] [] ? apic_write+0x11/0x13
[1426965.146800] [] ? lapic_next_event+0x10/0x14
[1426965.146802] [] ? clockevents_program_event+0x75/0x7e
[1426965.146804] [] ? tick_dev_program_event+0x37/0xf7
[1426965.146806] [] ? enqueue_hrtimer+0x65/0x72
[1426965.146808] [] play_dead_common+0x6e/0x70
[1426965.146810] [] native_play_dead+0x9/0x20
[1426965.146811] [] cpu_idle+0x46/0x8d
[1426965.146813] [] start_secondary+0x192/0x196
controls@h2psl0 ~ 0$