Resolving conntrack table full error messages: 'nf_conntrack: table full, dropping packets'

This document (000020149) is provided subject to the disclaimer at the end of this document.

Situation

Issue

When investigating a connectivity issue, you may experience errors like the below in the system logs:

nf_conntrack: table full, dropping packets

This error indicates the connection tracking table size has been exhausted. This can manifest with different symptoms, such as intermittent or consistent network timeouts.

The conntrack table keeps state on open connections that the kernel is translating. This occurs often in a Kubernetes cluster when pods access an external endpoint, or another service within the cluster. These scenarios use NAT and stateful firewall rules which are maintained as entries in the conntrack table.

Investigation

By default, the table size is calculated based on the memory allocated to the node. This does not fit all workloads demands, for example in a microservice environment typically a higher number of inter-service connections could be expected without consuming a high amount of memory.

To output the current max table size:

cat /proc/sys/net/netfilter/nf_conntrack_max

To get a point in time count of the current entries in the table:

cat /proc/sys/net/netfilter/nf_conntrack_count

Note: With the conntrack package installed, you can also use conntrack -C

If the nf_conntrack_count and nf_conntrack_max are close, it is indicating that the current workload requires a larger table size.

If the current number of entries are not approaching the table size, this could indicate that a burst of workload was experienced historically, in a containerized environment this can be common. For example, if the high-traffic Pods may now running on different nodes.

Resolution

Increasing the conntrack table size is achieved with sysctl.

Calculate a higher value, this can be applied to the node immediately with:

sysctl -w net.netfilter.nf_conntrack_max=<value>

To persist through reboot, add the tunable to either /etc/sysctl.conf, or a specific config file in /etc/sysctl.d.

For example, if your Linux distribution follows the /etc/sysctl.d/ directory structure:

echo "net.netfilter.nf_conntrack_max=<value>" > /etc/sysctl.d/10-conntrack-max.conf
sysctl -p /etc/sysctl.d/10-conntrack-max.conf

This creates a new config file to set the table size at each boot.

Additionally, if you configure nodes with configuration management, UserData, or build custom images etc., you may wish to add this to your usual approach to configure this for future nodes.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020149
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center