How to deploy the AWS EBS CSI driver on K3s

Task

This knowledge base article will provide the directions for deploying and testing the AWS EBS CSI driver and storage class on K3s.

Requirements

• K3s 1.18+ (may apply to other versions)
• Amazon Web Services (AWS) account with privileges to launch EC2 instances and create IAM policies.

Background

K3s has all in-tree storage providers removed since Kubernetes is shifting to out of tree providers for Container Storage Interface (CSI) and Cloud Provider Interface (CPI). While in-tree providers are convenient, they add a lot of bloat to Kubernetes and will eventually be removed from upstream Kubernetes, possibly in 2021.

This how-to guide will instruct you on installing and configuring the AWS EBS CSI driver and storage class. This will allow you to dynamically provision and attach an EBS volume to your pod without having to manually create a persistent volume (PV) and EBS volume in advance. In the event that your node crashes and your pod is re-launched on another node, your pod will be reattached to the volume assuming that node is running in the same availability zone used by the defunct node.

Solution

Assuming you want the CSI and storage class automatically deployed by K3s, copy the following YAML to a file in your manifests folder on one or all of your K3s servers. For example, /var/lib/rancher/k3s/server/manifests/aws-ebs-csi.yaml:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: aws-ebs-csi-driver
  namespace: kube-system
spec:
  chart: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/releases/download/v0.5.0/helm-chart.tgz
  version: v0.5.0
  targetNamespace: kube-system
  valuesContent: |-
    enableVolumeScheduling: true
    enableVolumeResizing: true
    enableVolumeSnapshot: true
    extraVolumeTags:
      Name: k3s-ebs
      anothertag: anothervalue
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-storageclass
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

Next, you need to give the driver IAM permissions to manage EBS volumes. This can be done one of two ways. You can either feed your AWS access key and secret key as a Kubernetes secret, or use an AWS instance profile. Since the first option involves passing sensitive keys in clear text and storing them directly in Kubernetes, the second option is usually preferred. I will go over both options. For either option, make sure your access keys or instance profile has the following permissions set in IAM:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AttachVolume",
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteSnapshot",
        "ec2:DeleteTags",
        "ec2:DeleteVolume",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DescribeVolumesModifications",
        "ec2:DetachVolume",
        "ec2:ModifyVolume"
      ],
      "Resource": "*"
    }
  ]
}

Option 1: Kubernetes Secret

You can place your AWS access key and secret key into a Kubernetes secret. Create a YAML file with the following contents and run a kubectl apply. You can also place this inside your /var/lib/rancher/k3s/server/manifests/aws-ebs-csi.yaml file. Keep in mind this is not a terribly secure option and anyone with access to these files or secrets in the kube-system namespace will be able to obtain your AWS access keys.

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: kube-system
stringData:
  key_id: "AKI**********"
  access_key: "**********"

Option 2: Instance Profile

This option to more secure and should not expose your keys in clear text or in a Kubernetes secret object. You'll need to make sure when your EC2 instances are launched, you've attached an instance profile that has the permissions defined above in the JSON block.

Verifying and Testing

You can now check your pods to see if the CSI pods are running. You should see something like this:

# kubectl get pods -n kube-system | grep ebs
ebs-snapshot-controller-0                1/1     Running   0          15m
ebs-csi-node-k2gh5                       3/3     Running   0          15m
ebs-csi-node-xdcvn                       3/3     Running   0          15m
ebs-csi-controller-6f799b5548-46jqr      6/6     Running   0          15m
ebs-csi-controller-6f799b5548-h4nbb      6/6     Running   0          15m

Time to test things out. The following command can be run that should provision a 1GB EBS and attach it to your pod:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-storageclass
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: storage-test
spec:
  containers:
  - name: "storage-test"
    image: "ubuntu:latest"
    command: ["/bin/sleep"]
    args: ["infinity"]
    volumeMounts:
      - name: myebs
        mountPath: /mnt/test
  volumes:
  - name: myebs
    persistentVolumeClaim:
      claimName: myclaim
EOF

In your AWS console, you should see a new EBS volume has been created. After about a minute, you should be able to exec into your pod and see the volume mounted in your pod:

# kubectl exec storage-test -- df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          31G  6.2G   25G  20% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/nvme2n1    976M  2.6M  958M   1% /mnt/test
/dev/root        31G  6.2G   25G  20% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           3.8G   12K  3.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.8G     0  3.8G   0% /proc/acpi
tmpfs           3.8G     0  3.8G   0% /proc/scsi
tmpfs           3.8G     0  3.8G   0% /sys/firmware

Cleaning Up

Remove the test pod by running the following:

kubectl delete pod storage-test

kubectl delete pvc myclaim

Reference

• K3s documentation
• AWS EBS CSI documentation