Node become NotReady state because of unhealthy PLEG

This document (000020265) is provided subject to the disclaimer at the end of this document.

Situation

The rancher UI displays the following:

"PLEG is not healthy: pleg was last seen active 18h50m17.324752357s ago; threshold is 3m0s "

Kubelet log shows;

docker logs kubelet --tail=20 -f

Output:-

E0511 09:12:59.037051 10851 remote_runtime.go:312] ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (17182828 vs. 16777216)

E0511 09:12:59.037105 10851 kuberuntime_container.go:382] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (17182828 vs. 16777216)

E0511 09:12:59.037123 10851 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (17182828 vs. 16777216)

Resolution

The workaround here is to prune stopped or dead containers to reduce the message size.

docker system prune

The Paketo Buildpacks fixed this issue already.
Please refer GitHub issue #80 for more details.

Cause

The labels set on the containers build by Paketo Buildpacks are causing a huge message size.
This results in the default gRPC message buffer size of 16Mb overflowing in kubelet, causing PLEG to fail.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.