Blocking Oracle processes on NUMA machines after migration to SLES12

This document (7023694) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)
SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2)
SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)

Situation

After migrating Oracle workloads with large SGA to SLES12 (all Service Packs) on a NUMA Machine performance is dropping.

Thousands of Oracle processes/treads are blocking in state uninterruptible sleep and CPU statistics shows high percentage values for I/O wait :

19:10:03        CPU     %user     %nice   %system   %iowait    %steal     %idle
19:10:03        all     30.57      0.00      8.16     29.25      0.00     32.02
19:20:03        all     31.89      0.00      8.83     31.76      0.00     27.52

Also page_misses can be observed in the sar data.

Resolution

Disabling auto-numa_balancing in /proc/sys/kernel/numa_balancing solves blocking processes and I/O wait almost vanishes :

19:40:03        CPU     %user     %nice   %system   %iowait    %steal     %idle
19:40:03        all     19.85      0.00      4.43      0.41      0.00     75.32
19:50:01        all     24.33      0.00      4.54      0.36      0.00     70.76

numa_balancing can be disabled by either :

echo 0 > /proc/sys/kernel/numa_balancing
sysctl -w kernel.numa_balancing=0

(Note : this disables numa_balancing during runtime, but this change is not reboot persistent)

To make the change persistent, create a sysctl configuration file containing the settings in /etc/sysctl.d/:

echo "kernel.numa_balancing = 0" > /etc/sysctl.d/numa_balancing.conf

and enable this by invoking :

sysctl --system

Cause

NUMA (Non-Uniform Memory Access) tries to keep processes/treads memory close to the CPU it's running on to speed up access time.

Larger Oracle Database Systems with hundred/thousands of threads spreaded across multiple NUMA zones are accessing common memory (SGA =System Global Area), the system tries to move memory to the appropriate NUMA zone currently executing the active process/thread, which results in permanent memory re-organisation.

Turning of the numa_balancing mechanism for applications with common used larger memory areas in the kernel solves the permanent overhead on memory re-organization.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.