At 16:55 the IPC-ETH link on h1sush12 went offline for about 5 seconds, stopping all IPC traffic into and out of this front end. Since that time we have had a low rate of send errors from models running on h1sush12, but no receive errors.
Two models on h1sush12 send IPCs to remote models:
| chan | send model | rate | receive model |
| H1:IOP-SUS_H12_RM1RM2PM1_WDIPC | h1iopsush12 | 2k | h1iopseih16 |
| H1:IOP-SUS_H12_MC1MC3PRMPR3_WDIPC | h1iopsush12 | 2k | h1iopseih23 |
| H1:SUS-PRM_M1_LOCK_L | h1susprm | 16k | h1seiproc |
Note that the SWWD IPC send at 2k even though the models run at 64k.
The receivers on h1iopseih16, h1iopseih23 and h1seiproc reported a low level of receive errors overnight. Each time these were one error in the second. Most of the time the IOP error times were not the same as the PRM times, except for the last one (see trend)
In the past 14 hours, IOP had 5 errors and PRM 9 errors.
EJ recommends as a first step to replace h1sush12's ethernet fiber and the switch SFP (not replacing h12 Mellanox SFP at this time).
h1sush12 is connected to port 3 of sw-msr-ipc0. Looking at the port stats shows errors on port3 which are not seen on any other port (port2 shown for comparison).
h1susb2h34 Port2:
sw-msr-ipc0#show interfaces ethernet2
Ethernet2 is up, line protocol is up (connected)
Hardware is Ethernet, address is 4c09.9737.14bb (bia 4c09.9737.14bb)
Description: "h1susb2h34 | FMM-02M-1122"
Ethernet MTU 9214 bytes, BW 25000000 kbit
Full-duplex, 25Gb/s, auto negotiation: off, uni-link: n/a
Up 2 days, 16 hours, 31 minutes, 54 seconds
Loopback Mode : None
6 link status changes since last clear
Last clearing of "show interface" counters 2 days, 19:10:45 ago
5 minutes input rate 44.4 Mbps (0.2% with framing overhead), 34811 packets/sec
5 minutes output rate 937 Mbps (4.1% with framing overhead), 534449 packets/sec
6036140558 packets input, 962213865558 bytes
Received 84764 broadcasts, 6036055766 multicast
0 runts, 0 giants
0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
0 PAUSE input
90864581964 packets output, 19842295883609 bytes
Sent 1569974 broadcasts, 90863012607 multicast
0 output errors, 0 collisions
0 late collision, 0 deferred, 0 output discards
0 PAUSE output
h1sush12 port3:
sw-msr-ipc0#show interfaces ethernet3
Ethernet3 is up, line protocol is up (connected)
Hardware is Ethernet, address is 4c09.9737.14bc (bia 4c09.9737.14bc)
Description: "h1sush12 | FMM-02M-1120"
Ethernet MTU 9214 bytes, BW 25000000 kbit
Full-duplex, 25Gb/s, auto negotiation: off, uni-link: n/a
Up 15 hours, 49 minutes, 3 seconds
Loopback Mode : None
6 link status changes since last clear
Last clearing of "show interface" counters 2 days, 19:10:47 ago
5 minutes input rate 19.4 Mbps (0.1% with framing overhead), 18429 packets/sec
5 minutes output rate 962 Mbps (4.2% with framing overhead), 550830 packets/sec
3187368940 packets input, 418602501888 bytes
Received 96260 broadcasts, 3187272674 multicast
50 runts, 0 giants
57 input errors, 7 CRC, 0 alignment, 38 symbol, 0 input discards
0 PAUSE input
93703371906 packets output, 20383690243959 bytes
Sent 1558417 broadcasts, 93701815262 multicast
0 output errors, 0 collisions
0 late collision, 0 deferred, 0 output discards
0 PAUSE output