Logs not forwarded by Rancher Logging in Rancher v2.x when Docker daemon logging driver is not set to json-file

This document (000020067) is provided subject to the disclaimer at the end of this document.

Situation

Issue

The Rancher v2.x Logging feature enables you to configure log forwarding for Pods, as well as system component containers, in a cluster to a logging endpoint such as Elasticsearch or Splunk.

This feature works by deploying a workload to each node in the cluster that mounts the container log directory from the host to parse the Docker container json log files. This is dependent upon use of the json-file Docker logging driver. In the event that the Docker daemon is configured with an alternative logging driver, the logging feature will be unable to parse the logs and will not forward these.

In CentOS and RHEL packaged Docker 1.13.1, the default log driver configured is journald, which will prevent log forwarding functioning. Meanwhile, whilst json-file is the default log driver in the upstream Docker packages, if an alternative has been configured on nodes this will also prevent the correct functioning of the log forwarding.

You can verify the currently configured Docker logging driver on a node by running docker info | grep Logging, which will show output of the following format: Logging Driver: journald.

In the event that json-file is not the configured logging driver, the output of ls -la /var/log/containers/ on the node should also be empty. With json-file configured this would display symoblic links to paths under /var/log/pods, containing symbolic links which in turn point to the Docker container json log files.

Pre-requisites

  • Rancher v2.x managed cluster with Rancher logging enabled

Resolution

CentOS or RHEL packaged Docker
  1. Update /etc/sysconfig/docker, as shown in the screenshot below, to set --log-driver=json-file instead of journald.

    sysconfigdocker

  2. Restart the Docker daemon: systemctl restart docker

  3. You should now see symlinked logs created under /var/log/containers
Upstream Docker
  1. Configure the json-file Docker logging driver in /etc/docker/daemon.json per the Docker documentation
  2. Restart the Docker daemon: systemctl restart docker
  3. You should now see symlinked logs created under /var/log/containers

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020067
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center