"log unreadable. It is excluded and would be examined next time." warning messages, for kubelet and kube-proxy, in rancher-logging-fluentd Pod logs of worker nodes
This document (000020201) is provided subject to the disclaimer at the end of this document.
In a Rancher v2.x provisioned Kubernetes cluster, with Rancher Cluster Logging configured, the
rancher-logging-fluentd Pod logs on Linux worker role only nodes show warning messages of the following format:
2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kubelet_5c47838dd4af749a7a0d1c457b04a6d7b905e680157718063c8e5d9eb61268fa.log unreadable. It is excluded and would be examined next time. 2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kube-proxy_d2beb2e667eefbd6d95355082af4bc61c367fc4c220d9f1d165d15a8c8be2ab1.log unreadable. It is excluded and would be examined next time.
The output of
ls /var/lib/rancher/rke/log/ on affected workers shows that these files referenced in the warning log messages are broken symlinks. In addition,
docker ps output shows the container ID for the currently running
kube-proxy containers does not match the IDs in these filenames.
- A Rancher v2.x provisioned Kubernetes cluster, using either custom nodes or nodes hosted in an infrastructure provider.
- Rancher Cluster Logging configured on the cluster, with the 'Include System Log' option set.
- Worker role only nodes in the cluster.
These warning level messages in the
rancher-logging-fluentd Pod are the result of the issue tracked in Rancher GitHub issue #22549.
The container log symlinks in
/var/lib/rancher/rke/log/ for cluster component containers (
nginx-proxy) on worker nodes in Rancher launched clusters are not cleaned up when these components are re-created, i.e. due to a Kubernetes version upgrade, or other configuration update for these components.
As a result these broken symlinks persist and cause the
log unreadable warning messages when the
rancher-logging-fluentd Pod attempts to parse files in the
This warning message itself is harmless and can be ignored.
The request to handle automatic clean-up of these log symlinks on worker nodes, in Rancher provisioned clusters, is tracked in Rancher GitHub issue #22549.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000020201
- Creation Date: 06-May-2021
- Modified Date:06-May-2021
- SUSE Rancher
For questions or concerns with the SUSE Knowledgebase please contact: firstname.lastname@example.org