bonding (active-passive) - TCP packets received out of order

This document (000020345) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 SP5
Intel® Ethernet 700 Series Network Adapter

Situation

On a system with Bonding Mode "active-passive" the packet capture (network trace) shows a lot [ TCP Out-of-Order ] packets.
It is also observed that the system generates TCP SACK to the sender which in return throttles the speed.
This is causing low throughput leading to performance problems.

The problem is not observed when a single network interface (NIC) instead of bonding is used.

The ethtool output for the single network interface shows:

# ethtool -i eth0
driver: i40e
version: 2.8.20-k
....

Resolution

Disable the Network Adapters LLDP hardware engine with ethtool:

# ethtool -set-priv-flags <interface name> disable-fw-lldp on

Cause

The Intel Ethernet 700 Series Network Adapter has a built-in hardware LLDP engine, which is enabled by default.
The firmware on the NIC cards is performing LLDP itself and interfering with what packets the kernel/driver sees.
The failure to forward the packets on to the network stack causes problems for the bond driver.

Per note from the hardware vendor:

"LACP may not function correctly in certain environments that require LLDP frames containing LCAP information to be forwarded to the network stack.".

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.