All five FE cores crashed at 11 mins past midnight Sun 1st April, this is not an april fools joke.

here is the dmesg capture before we reboot the machine.


[1426964.148086] h2pslfss: ADC TIMEOUT 1 31229 61 31293
[1426964.148088] h2psliss: ADC TIMEOUT 0 31229 61 31293
[1426964.148090] h2pslpmc: ADC TIMEOUT 2 31229 61 31293
[1426964.148092] h2psldbb: ADC TIMEOUT 3 31229 61 31293
[1426965.146334] h2ioppsl0: timeout 0 1000000 
[1426965.146339] BUG: unable to handle kernel NULL pointer dereference at (null)
[1426965.146341] IP: [] fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146349] PGD 0 
[1426965.146350] Oops: 0000 [#1] SMP 
[1426965.146351] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:2b:01.0/class
[1426965.146353] CPU 1 
[1426965.146353] Modules linked in: h2psldbb h2pslpmc h2pslfss h2psliss h2ioppsl0 open_mx mbuf
[1426965.146356] 
[1426965.146358] Pid: 0, comm: swapper Not tainted 2.6.34.1 #6 X8DTU/X8DTU
[1426965.146359] RIP: 0010:[]  [] fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146364] RSP: 0018:ffff8801beca7698  EFLAGS: 00010092
[1426965.146365] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000011e
[1426965.146366] RDX: 0000000000023c56 RSI: ffffffff81789ffd RDI: ffffffffa00311f2
[1426965.146367] RBP: ffff8801beca7ee8 R08: 000000007ffffff0 R09: 000000000000000a
[1426965.146368] R10: 0000000000000006 R11: 00000000ffffffff R12: 0000000000000000
[1426965.146370] R13: 0000000000000000 R14: ffffc900118d6000 R15: 0000000000000040
[1426965.146371] FS:  0000000000000000(0000) GS:ffff880001e20000(0000) knlGS:0000000000000000
[1426965.146372] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1426965.146373] CR2: 0000000000000000 CR3: 0000000001a09000 CR4: 00000000000006e0
[1426965.146375] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1426965.146376] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1426965.146377] Process swapper (pid: 0, threadinfo ffff8801beca6000, task ffff8801bec7aac0)
[1426965.146378] Stack:
[1426965.146379]  0000271700002711 0000271700000006 000000010000290b 0000000200000003
[1426965.146380] <0> 0000003500000003 ffffc900118d7034 ffffffffa0033a80 0000000000000000
[1426965.146382] <0> 0000000000000000 ffffc900118bc000 0000000400000000 00000000000007f8
[1426965.146384] Call Trace:
[1426965.146390]  [] ? cpumask_next_and+0x2c/0x39
[1426965.146394]  [] ? cpumask_weight+0xc/0xe
[1426965.146396]  [] ? find_busiest_group+0x36f/0x784
[1426965.146400]  [] ? timekeeping_get_ns+0x16/0x38
[1426965.146402]  [] ? apic_write+0x11/0x13
[1426965.146404]  [] ? lapic_next_event+0x10/0x14
[1426965.146406]  [] ? clockevents_program_event+0x75/0x7e
[1426965.146408]  [] ? tick_dev_program_event+0x37/0xf7
[1426965.146411]  [] ? enqueue_hrtimer+0x65/0x72
[1426965.146413]  [] play_dead_common+0x6e/0x70
[1426965.146415]  [] native_play_dead+0x9/0x20
[1426965.146417]  [] cpu_idle+0x46/0x8d
[1426965.146422]  [] start_secondary+0x192/0x196
[1426965.146423] Code: 00 8b 42 24 83 c8 01 89 42 24 8b 95 44 f8 ff ff 31 c0 8b b5 8c f8 ff ff e8 2a 13 00 00 48 8b 05 8f a7 00 00 48 c7 c7 f2 11 03 a0 <8b> 30 31 c0 e8 13 13 00 00 48 8b 05 78 a7 00 00 8b 00 66 85 c0 
[1426965.146433] RIP  [] fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146437]  RSP 
[1426965.146437] CR2: 0000000000000000
[1426965.146756] ---[ end trace d3abcc5123271f5d ]---
[1426965.146757] Kernel panic - not syncing: Attempted to kill the idle task!
[1426965.146758] Pid: 0, comm: swapper Tainted: G      D    2.6.34.1 #6
[1426965.146759] Call Trace:
[1426965.146761]  [] panic+0x73/0xe8
[1426965.146764]  [] do_exit+0x6d/0x712
[1426965.146765]  [] ? spin_unlock_irqrestore+0x9/0xb
[1426965.146767]  [] ? kmsg_dump+0x115/0x12f
[1426965.146769]  [] oops_end+0xb1/0xb9
[1426965.146772]  [] no_context+0x1f7/0x206
[1426965.146774]  [] __bad_area_nosemaphore+0x179/0x19c
[1426965.146776]  [] bad_area_nosemaphore+0xe/0x10
[1426965.146778]  [] do_page_fault+0xff/0x210
[1426965.146780]  [] page_fault+0x1f/0x30
[1426965.146784]  [] ? fe_start+0xe32/0x214e [h2ioppsl0]
[1426965.146788]  [] ? fe_start+0xe24/0x214e [h2ioppsl0]
[1426965.146791]  [] ? cpumask_next_and+0x2c/0x39
[1426965.146793]  [] ? cpumask_weight+0xc/0xe
[1426965.146795]  [] ? find_busiest_group+0x36f/0x784
[1426965.146797]  [] ? timekeeping_get_ns+0x16/0x38
[1426965.146799]  [] ? apic_write+0x11/0x13
[1426965.146800]  [] ? lapic_next_event+0x10/0x14
[1426965.146802]  [] ? clockevents_program_event+0x75/0x7e
[1426965.146804]  [] ? tick_dev_program_event+0x37/0xf7
[1426965.146806]  [] ? enqueue_hrtimer+0x65/0x72
[1426965.146808]  [] play_dead_common+0x6e/0x70
[1426965.146810]  [] native_play_dead+0x9/0x20
[1426965.146811]  [] cpu_idle+0x46/0x8d
[1426965.146813]  [] start_secondary+0x192/0x196
controls@h2psl0 ~ 0$