SAP applications are using ephemeral port 40403 or 40404 excessively

This document (000020801) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12

Situation

On a Linux system with SAP applications installed, when applications ask for temporary port usage, also known as ephemeral ports, ports 40403 and/or 40404 are being assigned far more often than any other port.

Resolution

SAP installations often set a very large system wide "ip_local_port_range" for use, but then set a tremendously detailed list for "ip_local_reserved_ports" which is a list of ports to be avoided.

The practice of overlapping these two ranges is allowed, but it impacts the algorithm which selects ephemeral ports when applications request one.  The ports directly after the largest reserved range will be thousands of times more likely to be selected than other ports.

To avoid this behavior, instead of defining a very wide range for "ip_local_port_range" and defining exceptions within that with "ip_local_reserved_ports", define a smaller range for "ip_local_port_range" and then do not define any reserved ports within that range.

For example, after most SAP installations, the following ranges are the largest available without interruption:

10516 - 19199
or
21214 - 29999

Select ONE of those ranges, i.e. within /etc/sysctl.conf:

net.ipv4.ip_local_port_range = 21214 29999

And then make sure nothing within that range is specified within
net.ipv4.ip_local_reserved_ports

Then the algorithm will have nearly 9000 ports to chose from and should distribute the choices fairly evenly.  This range of ports is smaller than the default range, but in some cases a mid-sized range that is evenly used may be better than a larger range where only a few ports are likely to get selected.

NOTE:  If SAP is enforcing its port ranges somewhere other than /etc/sysctl.conf, their location may need to be tracked down and changed.

Regardless of what is present in /etc/sysctl.conf, you can check the settings actually in effect with:

sysctl -a | grep ip_local


SUSE has also released a couple of changes which can slightly mitigate this behavior, but for the most part it is still a necessary part of the port selection algorithm.  Those changes (with minor impact) are found beginning in:

SLES 12 SP4 kernel-default 4.12.14-95.96
SLES 12 SP5 kernel-default 4.12.14-122.116
SLES 15 SP1 kernel-default 4.12.14-150100.197.123
SLES 15 SP2 kernel-default 5.3.18-150200.24.129
SLES 15 SP3 kernel-default  5.3.18-150300.59.63
SLES 15 SP4 present in all kernels.

Another way to identify if your SUSE kernel contains the changes is the following command, substituting your kernel package name, for example, if you have kernel-azure:

rpm -q --changelog kernel-azure | grep -A1 -B1 1180153

Which (if the changes are present) should return entries such as:

- tcp: add some entropy in __inet_hash_connect() (bsc#1180153).
- tcp: change source port randomizarion at connect() time
  (bsc#1180153).
 

Cause

The port selection algorithm is strongly impacted by having ip_local_port_range and ip_local_reserved_ports settings which overlap each other.  The larger a range that is included in ip_local_reserved_ports, the higher he probability that the ports immediately after that range will be selected for use.  For example, if a range of 10000 ports is reserved, the ports just after that range will be 5000 times more likely to be selected than other ports.

Additional Information

Linux community discusses adding a caution to the doc which discusses these settings:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=a7a80b17c750

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020801
  • Creation Date: 05-Oct-2022
  • Modified Date:05-Oct-2022
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center