Cgroup "Misc controller" introduce instability in Kubelet
This document (000021270) is provided subject to the disclaimer at the end of this document.
A node running Cgroup V1
You can check the Runc library version of your Kubernetes version in the go.mod file in the Kubernetes repo.
Eg: See the below example for Kubernetes 1.24.6 using runc 1.1.1
kubelet_getters.go:300] "Path does not exist" path="/var/lib/kubelet/pods/<ID_NUMBER>/volumes"
These lines fill up the kubelet log ending up in gigabyte of logs.
This can also cause high CPU usage.
- Update the node to use Cgroup V2
- Upgrade to a Kubernetes version using runc libraries >= 1.1.6
It is recommended to upgrade your Kubernetes version.
Doing a change at the Kernel level may introduce unkowns to the stability of your system.
Problem comes from a discrepancy between the code creating the "Misc" Cgroup and the code cleaning the Cgroup that doesn't handle the "Misc" Cgroup, leaving it behind.
- runc integrated a fix in 1.1.6, HOWEVER
- kubelet depends on runc's cgroup libraries
- In order to clean up pods using the new "Misc" controller, runc cgroup library need to be updated to be aware of it.
- So even if your system run 1.1.6 but kubelet is not build with these library, the problem occurs.
Kubernetes 1.24.8 uses runc library 1.1.1
The first kubernetes release with runc library 1.1.6 in k8s 1.24.x is K8s 1.24.14
with the help of this PR
The following comment list all PR for all impacted Kubernetes minor releases
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000021270
- Creation Date: 13-Nov-2023
- Modified Date:15-Nov-2023
- SUSE Rancher
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com