How to send logs to Amazon Web Services (AWS) CloudWatch with the new logging services available on Rancher v2.5.x

Task

New logging service introduced in Rancher v2.5.x allows users to send logs to Amazon Web Services (AWS) Cloudwatch. This article details how to send logs to AWS CloudWatch with the new logging services available on Rancher v2.5.x, in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.5.x provisioned Kubernetes clusters.

Pre-requisites

• A Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes cluster with Logging in Rancher 2.5 enabled
• Rancher v2.5.x
• AWS IAM policy with at least the following permissions, the policy is attached to either an IAM user with credentials, or an EC2 instance profile which is attached to the nodes in the cluster

  {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
  }

Steps

1. Ensure the Rancher v2.5 logging is enabled on the cluster; visit and follow the Logging section in the Rancher docs if it is not already enabled.
2. Optional Create a secret containing the AWS Access key ID and Secret access key in cattle-logging-system namespace:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   data:
     id: <AWS Access key ID>
     secret: <AWS Secret access key>
   kind: Secret
   metadata:
     name: aws
     namespace: cattle-logging-system
   type: Opaque
   EOF

   > Note, this step is not required if using an EC2 instance profile
3. Create the ClusterOutput and ClusterFlow to forward the logs to the CloudWatch.

   cat <<EOF | kubectl apply -f -
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: ClusterOutput
   metadata:
     name: cloudwatch
     namespace: cattle-logging-system
   spec:
     cloudwatch:
       auto_create_stream: true  #Set to false to disable automatically create Log Stream under the Log Group
       aws_key_id:
         valueFrom:
           secretKeyRef:
             key: id
             name: aws
       aws_sec_key:
         valueFrom:
           secretKeyRef:
             key: secret
             name: aws
       buffer:
         timekey: 30s
         timekey_use_utc: true
         timekey_wait: 30s
       log_group_name: <LOG GROUP NAME ON THE CLOUDWATCH>
       log_stream_name: <LOG STREAM NAME UNDER THE LOG GROUP>
       region: <AWS REGION>
   ---
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: ClusterFlow
   metadata:
     name: logging
     namespace: cattle-logging-system
   spec:
     globalOutputRefs:
     - cloudwatch
   EOF

   > Note, the aws_key_id and aws_sec_key should be removed if using an EC2 instance profile
4. The logs will start sending to the CloudWatch once the ClusterOuput and ClusterFlow are created. You may visit the docs in the below to explore all the available configurations for the Rancher v2.5 logging.

Further reading

• Rancher Logging Documentation
• Full Configuration of CloudWatch output plugin for Fluentd