Jeff, Ryan S, Oli
During last week's set of issues, something that we saw happen a few times during our lock reacquisition attempts were lots of EY saturations while going through LOWNOISE_COIL_DRIVERS/TRANSITION_FROM_ETMX. The saturations would stop once L2 to R0 damping was turned off (87713), so it seemed like the issue was with the ETMY L2 satamp, so we swapped it out with a different one (87722). We didn't see any of these repeating saturations after that, but also we were changing a lot of things at the time trying to figure out the problem, plus we hadn't been seeing these saturations during every single relock attempt.
The DAC channels that were showing saturations were from DAC1 channels 1 and 2. Checking the model, those channels line up with R0 F2 and F3, which are the channels that control Length on R0. We plotted R0's MASTER OUTs during times where we saw lots of the EY saturations, and at times where the saturations heard on verbals were normal, including a time before we swapped the ETMY L2 satamp on October 14th.
Here's a breakdown of the different examples we looked at:
We saw that for the times where the amount of saturations were what we consider 'normal', the LOWNOISE_COIL_DRIVERS/TRANSITION_FROM_ETMX states behave similarly in the R0 MASTER OUT channels, including after swapping the satamp. The OSEMs will see some movement, but it's not too far outside of where they usually sit. However, for the two times that we checked where we had the excessive EY saturations, we saw that right before they started, there was a high frequency glitch seen in the ETMY L2 Length witness channel. This glitch only moved L2 a small amount, about 0.5 um, but it was causing R0 to move a lot in Length, saturating or nearly saturating for a long time.
Plotting the impulse response of the SUS-ETMY_L2_R0DAMP_L filter bank, we see that these filters have an impulse response time of ~16 seconds, and breaking down the impulse response by each filter's contribution, we see that the FM8 (module 7) filter module, invPsmoo, has a wild impulse response. Because of this filter module, the impulse response of the entire R0_L2DAMP_L filter bank is extremely long, and the signal is very large. The frequency response plot for module 7 shows us that it approaches the 10^15 gain at higher frequencies. Additionally, these new satamps have about double the gain at high frequencies as compared to the old satamps, so that would also be exacerbating any issues at higher frequencies.
With all that said, it looks like the conclusion is that the EY saturation issues from last week were not caused by a faulty satamp, but instead by something else that caused L2 to glitch, and the long impulse response and high gain causing R0 to take forever to calm down.
A temporary solution would be to keep the L2 to R0 damping off during locking until after LOWNOISE_LENGTH_CONTROL has finished, to make sure that we are avoiding having it on during all the sudden movements that could upset R0.
With Jumbo frames configured these are the iper3 results on the 100Gig (Intel E810):
- - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-60.00 sec 168 GBytes 24.0 Gbits/sec 5 sender[ 5] 0.00-60.00 sec 168 GBytes 24.0 Gbits/sec receiver[ 7] 0.00-60.00 sec 206 GBytes 29.5 Gbits/sec 9 sender[ 7] 0.00-60.00 sec 206 GBytes 29.5 Gbits/sec receiver[ 9] 0.00-60.00 sec 188 GBytes 26.8 Gbits/sec 4 sender[ 9] 0.00-60.00 sec 188 GBytes 26.8 Gbits/sec receiver[SUM] 0.00-60.00 sec 561 GBytes 80.3 Gbits/sec 18 sender[SUM] 0.00-60.00 sec 561 GBytes 80.3 Gbits/sec receiverFor the 25 Gig real time testing we are getting a round trip time (mean) of 50.1 us. In the state we left it at the end of today there are some error counters ticking, but no lost packets.
controls@lr-eth-e:$ sudo ethtool -S ens4f0np0 | grep errrx_crc_errors_phy: 5
rx_in_range_len_errors_phy: 0
rx_symbol_err_phy: 4
tx_errors_phy: 0
rx_pcs_symbol_err_phy: 4
rx_err_lane_0_phy: 474801