How to increase the inotify.max_user_watches and inotify.max_user_instances sysctls on a Linux host

This document (000020048) is provided subject to the disclaimer at the end of this document.

Situation

Task

The sysctls fs.inotify.max_user_instances and fs.inotify.max_user_watches define user limits on the number of inotify resources and inotify file watches. If these limits are reached, you may experience processes failing with error messages related to the limits, for example:

ENOSPC: System limit for number of file watchers reached...
The configured user limit (128) on the number of inotify instances has been reached 
The default defined inotify instances (128) has been reached

In the context of a Kubernetes cluster, this behaviour would exhibit as failing Pods, with inotify related errors in the Pod logs similar to the above. This article details how to check the current limits configured and how to increase these.

Pre-requisites

  • A Linux host

Resolution

Check current limits

You can check the current inotify user instance limit, with the following:

cat /proc/sys/fs/inotify/max_user_instances

Similarly, the current inotify user watch limit can be checked as follows:

cat /proc/sys/fs/inotify/max_user_watches
Update the limits

You can update the limits temporarily, with the following commands (setting the values to 8192 and 524288 respectively in this example):

sudo sysctl fs.inotify.max_user_instances=8192
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl -p

In order to make the changes permanent, i.e. to persist a reboot, you can set fs.inotify.max_user_instances=8192 and fs.inotify.max_user_watches=524288 in the file /etc/sysctl.conf.

After updating the limits, you can validate these on the host again, as above, with cat /proc/sys/fs/inotify/max_user_instances and cat /proc/sys/fs/inotify/max_user_watches.

To check the value as reflected in a running container, exec into the container and cat the files:

docker exec -it <CONTAINER ID> cat /proc/sys/fs/inotify/max_user_instances

and

docker exec -it <CONTAINER ID> cat /proc/sys/fs/inotify/max_user_watches

If the updated limits are not reflected on a host after running sysctl -p, reboot the host after setting the limits in /etc/sysctl.conf.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020048
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center