"log unreadable. It is excluded and would be examined next time." warning messages, for kubelet and kube-proxy, in rancher-logging-fluentd Pod logs of worker nodes

Issue

In a Rancher v2.x provisioned Kubernetes cluster, with Rancher Cluster Logging configured, the rancher-logging-fluentd Pod logs on Linux worker role only nodes show warning messages of the following format:

2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kubelet_5c47838dd4af749a7a0d1c457b04a6d7b905e680157718063c8e5d9eb61268fa.log unreadable. It is excluded and would be examined next time.
2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kube-proxy_d2beb2e667eefbd6d95355082af4bc61c367fc4c220d9f1d165d15a8c8be2ab1.log unreadable. It is excluded and would be examined next time.

The output of ls /var/lib/rancher/rke/log/ on affected workers shows that these files referenced in the warning log messages are broken symlinks. In addition, docker ps output shows the container ID for the currently running kubelet and kube-proxy containers does not match the IDs in these filenames.

Pre-requisites

• A Rancher v2.x provisioned Kubernetes cluster, using either custom nodes or nodes hosted in an infrastructure provider.
• Rancher Cluster Logging configured on the cluster, with the 'Include System Log' option set.
• Worker role only nodes in the cluster.

Root cause

These warning level messages in the rancher-logging-fluentd Pod are the result of the issue tracked in Rancher GitHub issue #22549.

The container log symlinks in /var/lib/rancher/rke/log/ for cluster component containers (kubelet, kube-proxy, nginx-proxy) on worker nodes in Rancher launched clusters are not cleaned up when these components are re-created, i.e. due to a Kubernetes version upgrade, or other configuration update for these components.

As a result these broken symlinks persist and cause the log unreadable warning messages when the rancher-logging-fluentd Pod attempts to parse files in the /var/lib/rancher/rke/log/ directory.

This warning message itself is harmless and can be ignored.

Resolution

The request to handle automatic clean-up of these log symlinks on worker nodes, in Rancher provisioned clusters, is tracked in Rancher GitHub issue #22549.