How to send logs to Amazon Web Services (AWS) CloudWatch with the new logging services available on Rancher v2.5.x

This document (000020008) is provided subject to the disclaimer at the end of this document.

Situation

Task

New logging service introduced in Rancher v2.5.x allows users to send logs to Amazon Web Services (AWS) Cloudwatch. This article details how to send logs to AWS CloudWatch with the new logging services available on Rancher v2.5.x, in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.5.x provisioned Kubernetes clusters.

Pre-requisites

  • A Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes cluster with Logging in Rancher 2.5 enabled
  • Rancher v2.5.x
  • AWS IAM policy with at least the following permissions, the policy is attached to either an IAM user with credentials, or an EC2 instance profile which is attached to the nodes in the cluster
    {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "logs:PutLogEvents",
                  "logs:CreateLogGroup",
                  "logs:PutRetentionPolicy",
                  "logs:CreateLogStream",
                  "logs:DescribeLogGroups",
                  "logs:DescribeLogStreams"
              ],
              "Effect": "Allow",
              "Resource": "*"
          }
      ]
    }

Steps

  1. Ensure the Rancher v2.5 logging is enabled on the cluster; visit and follow the Logging section in the Rancher docs if it is not already enabled.
  2. Optional Create a secret containing the AWS Access key ID and Secret access key in cattle-logging-system namespace:
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    data:
      id: <AWS Access key ID>
      secret: <AWS Secret access key>
    kind: Secret
    metadata:
      name: aws
      namespace: cattle-logging-system
    type: Opaque
    EOF
    > Note, this step is not required if using an EC2 instance profile
  3. Create the ClusterOutput and ClusterFlow to forward the logs to the CloudWatch.
    cat <<EOF | kubectl apply -f -
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: ClusterOutput
    metadata:
      name: cloudwatch
      namespace: cattle-logging-system
    spec:
      cloudwatch:
        auto_create_stream: true  #Set to false to disable automatically create Log Stream under the Log Group
        aws_key_id:
          valueFrom:
            secretKeyRef:
              key: id
              name: aws
        aws_sec_key:
          valueFrom:
            secretKeyRef:
              key: secret
              name: aws
        buffer:
          timekey: 30s
          timekey_use_utc: true
          timekey_wait: 30s
        log_group_name: <LOG GROUP NAME ON THE CLOUDWATCH>
        log_stream_name: <LOG STREAM NAME UNDER THE LOG GROUP>
        region: <AWS REGION>
    ---
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: ClusterFlow
    metadata:
      name: logging
      namespace: cattle-logging-system
    spec:
      globalOutputRefs:
      - cloudwatch
    EOF
    > Note, the aws_key_id and aws_sec_key should be removed if using an EC2 instance profile
  4. The logs will start sending to the CloudWatch once the ClusterOuput and ClusterFlow are created. You may visit the docs in the below to explore all the available configurations for the Rancher v2.5 logging.

Further reading

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020008
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center