[Fil, Jonathan, Ej, Erik]
Two previously unused pairs of optical fiber between EY and the Corner station were extended from patch panels to the computer racks in EY and the MSR.
Two computers were setup on either end to test Ethernet throughput and reliability, lr-eth-c in the MSR and lr-eth-e and EY. Each computer had a Intel e810 with 100g SFP and an NVidia Mellanox with 25g/10g SFP.
The first pair tested did not produce a link light.
The second pair was able to push 80g on the Intel card and 25g on the NVidia card, with no lost data but with a low rate of correctable errors on the EY card.
In an attempt to lower that error rate, we swapped the extension fiber at EY from the first (failed) fiber pair to the second, and cleaned it multiple times. This in fact raised the correctable error rate, but it's still operating with no data losses.
With Jumbo frames configured these are the iper3 results on the 100Gig (Intel E810):
- - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-60.00 sec 168 GBytes 24.0 Gbits/sec 5 sender[ 5] 0.00-60.00 sec 168 GBytes 24.0 Gbits/sec receiver[ 7] 0.00-60.00 sec 206 GBytes 29.5 Gbits/sec 9 sender[ 7] 0.00-60.00 sec 206 GBytes 29.5 Gbits/sec receiver[ 9] 0.00-60.00 sec 188 GBytes 26.8 Gbits/sec 4 sender[ 9] 0.00-60.00 sec 188 GBytes 26.8 Gbits/sec receiver[SUM] 0.00-60.00 sec 561 GBytes 80.3 Gbits/sec 18 sender[SUM] 0.00-60.00 sec 561 GBytes 80.3 Gbits/sec receiverFor the 25 Gig real time testing we are getting a round trip time (mean) of 50.1 us. In the state we left it at the end of today there are some error counters ticking, but no lost packets.
controls@lr-eth-e:$ sudo ethtool -S ens4f0np0 | grep errrx_crc_errors_phy: 5
rx_in_range_len_errors_phy: 0
rx_symbol_err_phy: 4
tx_errors_phy: 0
rx_pcs_symbol_err_phy: 4
rx_err_lane_0_phy: 474801