Bandwidth rate shaping with tc stops working after upgrade to SLES 10 SP2
This document (7002555) is provided subject to the disclaimer at the end of this document.
SUSE Linux Enterprise Server 10
For example a policy to limit the outgoing bandwidth for eth0 to approximately 430 KB/s (kilobytes) could be implemented like this :
# tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 1000Mbit rate 3482kbit prio 1 avpkt 1000 bounded isolated # tc filter add dev eth0 parent 1:0 protocol ip u32 match ip src 192.168.3.4 flowid 1:1The reason for this that TCP Segmentation Offload (TSO) has been enabled by default in the newer network cards. And while SLES 9 and earlier SLES 10 kernels did disable this feature by default (since it did not work on most hardware), this setting is now left enabled in the newer kernels.
TCP segmentation offload is used to reduce the CPU overhead of TCP/IP on fast networks. TSO breaks down large groups of data sent over a network into smaller segments that pass through all the network elements between the source and destination. This type of offload relies on the network interface controller (NIC) to segment the data and then add the TCP, IP and data link layer protocol headers to each segment. The NIC must support TSO, which is also called large segment offload (LSO).
Since TSO enables transmission of data "outside" the linux kernel to optimize the throughput without bugging the CPU, it then breaks the traffic control shaping because the kernel only sees the "tips of the icebergs" going through the NIC.
# ethtool -k eth0This should produce output like the following :
Offload parameters for eth0:
tcp segmentation offload: on
The solution to the problem is to disable TSO for the interface, which can be done by following these steps as a test :
- Remove any active tc settings from the interface with a command like
# tc qdisc del root dev eth0
- Change the TSO setting to off for the interface. If it is eth0, the command would be
# ethtool -K eth0 tso off(note the capital K)
- Verify that it has been disabled:
# ethtool -k eth0
- Reapply the tc settings - if using the example above :
# tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8 # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 1000Mbit rate 3482kbit prio 1 avpkt 1000 bounded isolated # tc filter add dev eth0 parent 1:0 protocol ip u32 match ip src 192.168.3.4 flowid 1:1
If it works, then the next step is to implement a permanent solution which disables TSO for the network card(s) in question during system boot :
- Open the /etc/sysconfig/network/ifcfg- file that coresponds with the card in an editor
- Add the following setting to the ETHTOOL_OPTIONS parameter :
-K iface tso off
so that the line e.g. reads :
ETHTOOL_OPTIONS='-K iface tso off'
- Save the file
- Repeat steps 1-3 for each card that should have TSO disabled
- Reboot the system to implement the new settings
- Verify that TSO has been disabled for each interface with
# ethtool -k ethX
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:7002555
- Creation Date: 05-Feb-2009
- Modified Date:11-Mar-2021
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com