After card replacement and power up I was able to see the full compliment of cards. However the models did not start correctly and the IOP had a negative IRIGB time. After this cleared the IOP still had a DAC error. I killed all models and started the IOP. This time it failed with the same dmesg as was seen Tuesday morning.

I am now power cycling h1susex (leaving the IO Chassis alone).

After the second power cycle of h1susex (this time leaving the IO Chassis alone) I am again able to see the full compliment of cards, and again the IOP went into a negative IRIGB excursion, but the user models this time were not in a failed state. After 10 minutes the timing came good again, and everything greened up.

We are uncertain if the particular 18bit DAC we swapped was the cause of this failure, given the repeat of the dmesg when the IOP startup failed. Note that the dmesg error references an 18bit dac card in the line

RIP  [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]

Here is the IOP failure dmesg for comparison with the one yesterday

  638.430574] h1iopsusex: DAC I/O address=0xffffffff  0x0
[  638.430578] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  638.445131] IP: [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]
[  638.445425] PGD 81118b067 PUD 83fa9a067 PMD 0
[  638.445715] Oops: 0000 [#1] SMP
[  638.445998] CPU 4
[  638.446004] Modules linked in: h1iopsusex(+) dis_irm(P) dis_ix_ntb(P) dis_ix_dma(P) dis_intel_dma(P) dis_kosif open_mx gpstime mbuf [last unloaded: h1iopsusex]
[  638.447142]
[  638.447417] Pid: 7840, comm: insmod Tainted: P            3.0.8 #2 Supermicro Super Server/X10SRi-F
[  638.447982] RIP: 0010:[<ffffffffa440f772>]  [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]
[  638.448543] RSP: 0018:ffff88085cf5ddc8  EFLAGS: 00010282
[  638.448825] RAX: 000000000000003e RBX: 0000000000000000 RCX: 0000000000000b43
[  638.449111] RDX: 0000000000026b49 RSI: ffffffff81797e8c RDI: 000000000000000a
[  638.449397] RBP: ffff88085cf5de08 R08: 0000000000000002 R09: 000000000000000a
[  638.449683] R10: 0000000000000005 R11: 0000000000000000 R12: ffff88087bbfc000
[  638.449970] R13: 0000000000000001 R14: ffffffffa4428320 R15: 0000000000000001
[  638.450255] FS:  00007fa3198d8700(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000
[  638.450810] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  638.451093] CR2: 0000000000000000 CR3: 000000083f965000 CR4: 00000000001406e0
[  638.451379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  638.451665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  638.451951] Process insmod (pid: 7840, threadinfo ffff88085cf5c000, task ffff88087bdaef00)
[  638.452505] Stack:
[  638.452779]  00000000a4424640 00000000a4428320 0000000000000000 ffffffffa4428320
[  638.453344]  0000000000000000 0000000000000001 0000000000000002 0000000000000000
[  638.453909]  ffff88085cf5de58 ffffffffa440fee3 ffff88085cf5de28 00000001a0000d41
[  638.454474] Call Trace:
[  638.454753]  [<ffffffffa440fee3>] mapPciModules+0x5be/0x7f8 [h1iopsusex]
[  638.455041]  [<ffffffffa441011d>] ? mapPciModules+0x7f8/0x7f8 [h1iopsusex]
[  638.455329]  [<ffffffffa4410663>] init_module+0x546/0xd09 [h1iopsusex]
[  638.455617]  [<ffffffff8106363f>] ? load_module+0xc45/0xda8
[  638.455902]  [<ffffffffa441011d>] ? mapPciModules+0x7f8/0x7f8 [h1iopsusex]
[  638.456190]  [<ffffffff8100020a>] do_one_initcall+0x7a/0x129
[  638.456473]  [<ffffffff81063820>] sys_init_module+0x7e/0x1ca
[  638.456757]  [<ffffffff8154763b>] system_call_fastpath+0x16/0x1b
[  638.457041] Code: 00 e8 8c 51 c1 dc 48 89 c3 48 89 c2 8b 35 27 2a 6b 02 48 c7 c7 1e a7 41 a4 b8 00 00 00 00 e8 1e ad ff ff 4a 89 1c ed 20 1f a3 a6 <8b> 33 48 c7 c7 3b a7 41 a4 b8 00 00 00 00 e8 03 ad ff ff 8b 03
[  638.458086] RIP  [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]
[  638.458376]  RSP <ffff88085cf5ddc8>
[  638.458653] CR2: 0000000000000000
[  638.459394] ---[ end trace e1932dd1eb06f959 ]---