Node become NotReady state because of unhealthy PLEG

This document (000020265) is provided subject to the disclaimer at the end of this document.

Situation

The rancher UI displays the following: 

"PLEG is not healthy: pleg was last seen active 18h50m17.324752357s ago; threshold is 3m0s "

Kubelet log shows;
docker logs kubelet --tail=20 -f 
Output:- 
E0511 09:12:59.037051 10851 remote_runtime.go:312] ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (17182828 vs. 16777216)

E0511 09:12:59.037105 10851 kuberuntime_container.go:382] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (17182828 vs. 16777216)

E0511 09:12:59.037123 10851 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (17182828 vs. 16777216)

Resolution

The workaround here is to prune stopped or dead containers to reduce the message size.
docker system prune
The Paketo Buildpacks fixed this issue already.
Please refer GitHub issue #80 for more details.

Cause

The labels set on the containers build by Paketo Buildpacks are causing a huge message size.
This results in the default gRPC message buffer size of 16Mb overflowing in kubelet, causing PLEG to fail.

 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020265
  • Creation Date: 28-May-2021
  • Modified Date:31-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center