"log unreadable. It is excluded and would be examined next time." warning messages, for kubelet and kube-proxy, in rancher-logging-fluentd Pod logs of worker nodes

This document (000020201) is provided subject to the disclaimer at the end of this document.

Situation

Issue

In a Rancher v2.x provisioned Kubernetes cluster, with Rancher Cluster Logging configured, the rancher-logging-fluentd Pod logs on Linux worker role only nodes show warning messages of the following format:

2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kubelet_5c47838dd4af749a7a0d1c457b04a6d7b905e680157718063c8e5d9eb61268fa.log unreadable. It is excluded and would be examined next time.
2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kube-proxy_d2beb2e667eefbd6d95355082af4bc61c367fc4c220d9f1d165d15a8c8be2ab1.log unreadable. It is excluded and would be examined next time.

The output of ls /var/lib/rancher/rke/log/ on affected workers shows that these files referenced in the warning log messages are broken symlinks. In addition, docker ps output shows the container ID for the currently running kubelet and kube-proxy containers does not match the IDs in these filenames.

Pre-requisites

Root cause

These warning level messages in the rancher-logging-fluentd Pod are the result of the issue tracked in Rancher GitHub issue #22549.

The container log symlinks in /var/lib/rancher/rke/log/ for cluster component containers (kubelet, kube-proxy, nginx-proxy) on worker nodes in Rancher launched clusters are not cleaned up when these components are re-created, i.e. due to a Kubernetes version upgrade, or other configuration update for these components.

As a result these broken symlinks persist and cause the log unreadable warning messages when the rancher-logging-fluentd Pod attempts to parse files in the /var/lib/rancher/rke/log/ directory.

This warning message itself is harmless and can be ignored.

Resolution

The request to handle automatic clean-up of these log symlinks on worker nodes, in Rancher provisioned clusters, is tracked in Rancher GitHub issue #22549.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020201
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center