containerd.io 1.4.4 bug advisory

This document (000020328) is provided subject to the disclaimer at the end of this document.

Environment

Recently our team has been made aware of PLEG health issues caused by a recent containerd.io package version.

This can manifest in a few ways, we usually see all Docker commands failing, but in recent cases, the Docker commands with a specific security flag fail and hang. We have isolated this to containerd.io package version 1.4.4-x, and has been logged in this GitHub issue.


Situation

 

 

******Update as of 25MAY21 ********

Our team is now aware that this issue is resolved in rc95 or higher. Please note that the issue was resolved in rc94, but there was a CVE tied to that rc version. Our recommendation for anyone experiencing this issue to move to rc95 or higher to resolve the containerd issues as well as avoiding the known CVE found in rc94. Even if you are not experiencing issues tied to this advisory, we would recommend you move to rc95. 

******End Update as of 25MAY21 ********

How do I know if I am impacted?

Customers running RKE could be impacted by this. This is impacting customers running Docker 19.03 and 20.10 where containerd.io is using 1.4.4 -x.  Nodes running containerd.io 1.4.4 may experience containers hanging on initialization after a certain number of containers with no-new-privileges are started.

Often this has come as a result of upgrading Docker with Rancher 2.5.6. The symptoms include PLEG timeout errors in the Rancher UI, CoreDNS pods failing to start, and docker inspect commands to hang on certain containers.

As the issue relates to the specific runc version (1.0.0-rc93) bundled with containerd.io, the following can be a basic test to identify if the node is running the affected runc build:

runc --version | grep -q 1.0.0-rc93 && echo "AFFECTED" || echo "NOT AFFECTED"

Resolution

Is there a workaround?

******Update as of 25MAY21 ********

The below workaround should not be used any longer.  With the release of rc95 (mentioned above) any customers experiencing this issue should upgrade to rc95 as the resolution is found there.  

******End Update as of 25MAY21 ********

 

Yes, currently our team recommends that you take the following step:

Downgrade or install the containerd.io package to a 1.4.3-x version.  There is no need to modify privileges on CoreDNS pods, once downgraded to 1.4.3 you should pin that version to not auto-update. Please ensure your team is aware of CVE-2021-21334 in 1.4.3-x. 

As examples of downgrading the containerd.io package on affected nodes:

Ubuntu:

apt install containerd.io=1.4.3-1

EL:

yum downgrade containerd.io-1.4.3-3.1.el7

As needed, drain and cordon the node, followed by restarting the Docker daemon.

For the most accurate steps, we recommend you consult the documentation for your OS on downgrading and version pinning for the specific package manager and Linux distribution.

For customers who have not upgraded their Rancher clusters to 2.5.6+, we recommend that you hold off on upgrading until this is resolved upstream. If you need to upgrade to Rancher 2.5.6+, you should be safe to upgrade to Rancher when using the above process to install and pin the containerd.io package to a 1.4.3-x version.

In the meantime, if you have any questions, please reach out to your Customer Success Manager or Rancher Support via a Support Ticket.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020328
  • Creation Date: 09-Jul-2021
  • Modified Date:09-Jul-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center