Addressing file system performance issues on NUMA machines

This document (7008919) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 1

Situation

On some NUMA systems performance issues can occur when large amounts of data are copied. This can be mainly observed with NFS. There is a low probability that it can happen with local file systems.

Resolution

This can for example be addressed by tuning zone_reclaim in the kernel. zone_reclaim is a tunable that is set automatically on NUMA machines when the penalty for accessing remote memory is high. This is an optimal setting where all of the machine's  memory bank sets have the same size that are assigned to the CPUs.

A problem occurs if the machine is not tuned to fit within a NUMA partition. The server can stall as the system performs the work necessary to keep the process local to a node. These stalls can be significant if the process is also writing data. There are five recommendations as to how the problem may be mitigated.

1. Disable zone_reclaim if the workload has not been partitioned to fit within NUMA nodes with:
echo 0 >/proc/sys/vm/zone_reclaim_mode
2. Disable NUMA support in the BIOS, effectively using the interleave memory policy for all processes.
3. Use numactl to use the interleave memory policy on the processes writing data by preceding the process start command with

numactl --interleave=all <command>

4. Tune vm.dirty_background_bytes to a value based on the backing storage such that stalls due to IO are within an acceptable margin:
echo $((128*1024*1024)) >/proc/sys/vm/dirty_background_bytes
echo $((1024*1024*1024)) >/proc/sys/vm/dirty_bytes
To set one or several of these parameters permanently edit the file /etc/sysctl.conf

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7008919
  • Creation Date: 30-Jun-2011
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center