How to stream k3s journal logs to Cloudwatch on Rancher

This document (000021029) is provided subject to the disclaimer at the end of this document.


Rancher 2.6.x and 2.7.x


Send the k3s journal logs to AWS CloudWatch using Rancher's v2 logging integration



    • Gathering k3s journal logs from each node in the cluster.
    • Parsing the logs to forward only the required fields.
    • Forwarding the parsed data to cloudwatch.

    Rancher uses this logging operator that comes with the below CRDS:

    • flow
    • clusterFlow
    • output
    • clusterOutput

    You can read more about them here.

    We will be using clusterFlow and clusterOutput as they are not namespaced. The clusterFlow CRD defines a logging flow for Fluentd with filters and outputs. Using this, we can define and apply filters to select only the desired data. Once parsed, data will be forwarded to the clusterOutput object. The clusterOutput CRD defines where to send the data. It supports several plugins, but we will use Cloudwatch. You can read the spec here.

    Now we have clusterFlow to parse the data and clusterOutput to define the destination of data. We need a way to get the journal logs from the nodes.

    HostTailer CRD is provided by and is supported on the Rancher. From the doc, HostTailer’s main goal is to tail custom files and transmit their changes to stdout. This way, the logging-operator can process them. Example usage is here. Similarly, you can use the file-tailer if you know the log file name.

    The difference between the two is host-tailer looks at specific systemd service logs like k3s.service logs, while for file-tailer, you need to specify the exact location of the log file like /var/log/nginx/access.log.

    Here is the YAML to get the systemd journal logs from each host. This will create a daemonset. Pods will fetch the logs from the journal log files of the specified service name and output them to stdout.

    kind: HostTailer
      name: k3s-systemd-tailer
      namespace: cattle-logging-system
        - name: k3s-systemd-tailer
          maxEntries: 100
          path: /run/log/journal/
          systemdFilter: k3s.service

    The log output will then be fed to clusterFlow, which parses the logs.

    kind: ClusterFlow
      name: host-tailer-flow
      namespace: cattle-logging-system
        - parser:
            key_name: message
            reserve_time: true
              type: json
        - record_transformer:
        - select: 
        - host-logging-cloudwatch

    Here we are matching the app name to the name of the host-tailer daemonset, which is host-tailer. Once matched, we parse them using the parser plugin. We only need the message field from the logs, so key_name is specified as the message, and the parse type is set to json. After this, we remove unwanted fields from the message field using the remove_keys spec from the record_transformer plugin.

    The globalOutputRefs is set to the name of the clusterOutput.

    kind: ClusterOutput
      name: host-logging-cloudwatch
      namespace: cattle-logging-system
        auto_create_stream: true
          type: json
          timekey: 30s
          timekey_use_utc: true
          timekey_wait: 30s
        log_group_name: hosted-group
        log_stream_name: host-logs
        region: us-west-2

    In the clusterOutput spec, we use cloudwatch and define log_group_name, log_stream_name, and region.



    This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

    • Document ID:000021029
    • Creation Date: 31-Mar-2023
    • Modified Date:31-Mar-2023
      • SUSE Rancher

    < Back to Support Search

    For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]

    SUSE Support Forums

    Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

    Join Our Community

    Support Resources

    Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

    SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
    Support FAQ

    Open an Incident

    Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

    Go to Customer Center