Configuring and tuning process and threads limits

This document (000020429) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP3

Situation

Process or thread creation fails, in some conditions this may affect also the creation of kernel threads which manifests as a workqueue lockup (dmesg message `BUG: workqueeue lockup` (beware not all workqueue lockups are caused by hitting the task limits)).

Resolution

Check what limits are set in the configuration files and adjust them accordingly.
 
MechanismScopeKernel defaultUserspace defaultWhere configured
pid_max
(/proc/sys/kernel/pid_max)
pid namespace (effectively global)max
(nr_cpus * 1024, 32768)
none (note: systemd > v237 propagates pids.max of the root cgroup here)/etc/sysctl.conf
/etc/sysctl.d/*.conf/usr/lib/sysctl.d/*.conf
threads-max (/proc/sys/kernel/threads-max)globalmemory / 128 KiBnone/etc/sysctl.conf
/etc/sysctl.d/*.conf
/usr/lib/sysctl.d/*.conf
pids.max (cgroup attribute of PID controller)cgroup (and hierarchical ancestors)infinitelyinfinitely (SLE-10123)/etc/systemd/system.conf /etc/systemd/logind.conf
/usr/lib/systemd/system/user-.slice.d/10-defaults.conf
or particular systemd units (TasksMax= directive)
RLIMIT_NPROC (setrlimit(2) inherited by forks)tasks of given uid (limit can be modified for chilren in forking tree)threads-max / 2none/etc/security/limits.conf DefaultLimitNPROC= (systemd/system.conf) or LimitNPROC= in systemd units
  
The HW derived defaults take into account the system at boot, nr_cpus refers to "possible cpus" (/proc/cpuinfo lists only "online" cpus, the possible CPUs are reported on x86_64 in kernel log message `[    0.020731] smpboot: Allowing 8 CPUs, 0 hotplug CPUs`).
The formula factors apply for SLE15 SP3.
Whatever the names of the mechanism suggests, the accounted entity is a kernel task.
 
It's suggested to override config files rather than to overwrite them. (https://documentation.suse.com/sles/15-SP3/html/SLES-all/cha-systemd.html#sec-boot-systemd-custom-drop-in)

Cause

The Linux kernel provides several mechanisms that limit the number of tasks that exist at given time in a particular scope. The default limit values are derived from available HW resources based on simple heuristics and these should work just fine for most workloads. (Persistent) user configuration may override these defaults, primarily dedicated to fine tuning but that may also backfire when HW capacity changes without appropriate configuration update.
The most stringent limit for the given scope applies.
 
Note: The accounted kernel resource is a task. Each thread of a process is a task.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020429
  • Creation Date: 09-Nov-2021
  • Modified Date:23-Nov-2021
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center