When to Use K3s and RKE2

Wednesday, 14 December, 2022

K3s and Rancher Kubernetes Engine (RKE2) are two Kubernetes distributions from the SUSE Rancher container platform. Either project can be used to run a production-ready cluster; however, they target different use cases and consequently possess unique characteristics.

This article will explain the similarities and differences between the projects. You’ll learn when it makes sense to use RKE2 instead of K3s and vice versa. Selecting the right choice is important because it affects the security and compliance of the containerized workloads you deploy.

K3s and RKE2

K3s provides a production-ready Kubernetes cluster from a single binary that weighs in at under 60MB. Because K3s is so lightweight, it’s a great option for running Kubernetes at the edge on IoT devices, low-power servers and your developer workstations.

Meanwhile, RKE2 is an alternative project that also runs a production-ready cluster. It offers similar simplicity to K3s while adding additional security and conformance layers, including Federal Information Processing Standard (FIPS) 140-2 compliance for use in the U.S. federal government and DISA STIG compliance.

RKE2 has evolved from the original RKE project. It’s also known as RKE Government, reflecting its suitability for the most demanding sectors. It’s not just government agencies that can benefit, though — the distribution is ideal for all organizations that prioritize security and compliance, so it continues to be primarily marketed as RKE2.

Similarities between K3s and RKE2

K3s and RKE2 are both lightweight Cloud Native Computing Foundation (CNCF)-certified Kubernetes distributions that Rancher fully supports. Although they diverge in their target use cases, the two platforms have several intentional similarities in how they’re launched and operated. Both can be deployed with Rancher Manager, and they each run your containers using the industry-standard containerd runtime.


K3s and RKE2 each offer good usability with a quick setup experience.

Starting a K3s cluster on a new host can be achieved with one command and around 30 seconds of waiting:

$ curl -sfL https://get.k3s.io | sudo sh -

The service is registered and started for you, so you can immediately run kubectl commands to interact with your cluster:

$ k3s kubectl get pods

RKE2 doesn’t fare much worse. A similarly straightforward installation script is available to download its binary:

$ curl -sfL https://get.rke2.io | sudo sh -

RKE2 doesn’t start its service by default. Run the following commands to enable and start RKE2 in server (control plane) mode:

$ sudo systemctl enable rke2-server.service
$ sudo systemctl start rke2-server.service

You can find the bundled kubectl binary at /var/lib/rancher/rke2/bin. It’s not added to your PATH by default; a kubeconfig file is deposited to /etc/rancher/rke2/rke2.yaml:

$ export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
$ /var/lib/rancher/rke2/bin/kubectl get pods

Ease of operation

In addition to their usability, K3s and RKE2 are simple to operate. You can upgrade clusters created with either project by repeating the installation script on each node:

# K3s
$ curl -sfL https://get.k3s.io | sh -

# RKE2
$ curl -sfL https://get.rke2.io | sh -

You should repeat any flags you supplied to the original installation command.

Automated upgrades are supported using Rancher’s system-upgrade-controller. After installing the controller, you can declaratively create Plan objects that describe how to migrate the cluster to a new version. Plan is a custom resource definition (CRD) provided by the controller.

Backing up and restoring data is another common Kubernetes challenge. K3s and RKE2 also mirror each other in this field. Snapshots are automatically written and retained for a configurable period. You can easily restore a cluster from a snapshot by running the following command:

# K3s
$ k3s server \
    --cluster-reset \

# RKE2
$ rke2 server \
    --cluster-reset \

Deployment model

K3s and RKE2 share their single-binary deployment model. They bundle all their dependencies into one download, allowing you to deploy a functioning cluster with minimal Kubernetes experience.

The projects also support air-gapped environments to accommodate critical machines that are physically separated from your network. Air-gapped images are provided in the release artifacts. Once you’ve transferred the images to your machine, running the regular installation script will bootstrap your cluster.

High availability and multiple nodes

K3s and RKE2 are designed to run in production. K3s is often used in the development, too, having gained a reputation as an ideal single-node cluster. It has robust multi-node management and is capable of supporting fleets of IoT devices.

Both projects can run the control plane with high availability, too. You can distribute replicas of control plane components over several server nodes and use external data stores instead of the embedded ones.

Differences between K3s and RKE2

K3s and RKE2 both offer single-binary Kubernetes, high availability and easy backups, with many commands interchangeable between the two. However, some key differences affect where and when they should be used. It’s these characteristics that justify the distributions existing as two independent projects.

RKE2 is closer to upstream Kubernetes

K3s is CNCF-certified, but it deviates from upstream Kubernetes in a few ways. It uses SQLite instead of etcd as its default data store, although an embedded etcd instance is available as an option in modern releases. K3s also bundles additional utilities, such as the Traefik Ingress controller.

RKE2 sticks closer to standard Kubernetes, promoting conformance as one of its main features. This gives you confidence that workloads developed for other distributions will run reliably in RKE2. It reduces the risk of inconvenient gotchas that can occur when K3s steps out of alignment with upstream Kubernetes. RKE2 automatically uses etcd for data storage and omits nonstandard components included in K3s.

RKE2 uses embedded etcd

The standard SQLite database in K3s is beneficial for compactness and can optimize performance in small clusters. In contrast, RKE2’s default use of etcd creates a more conformant experience, allowing you to integrate directly with other Kubernetes tools that expect an etcd data store.

While K3s can be configured with etcd, it’s an option you need to turn on. RKE2 is designed around it, which reduces the risk of misconfiguration and subpar performance.

K3s also supports MySQL and PostgreSQL as alternative storage solutions. These let you manage your Kubernetes data using your existing tooling for relational databases, such as backups and maintenance operations. RKE2 only works with etcd, offering no support for SQL-based storage.

RKE2 is more security-minded

RKE2 has a much stronger focus on security. Whereas edge operation is K3s’s specialism, RKE2 prioritizes security as its greatest strength.

Hardened against the CIS benchmark

The distribution comes configured for compatibility with the CIS Kubernetes Hardening Benchmark v1.23 (v1.6 in RKE2 v1.25 and earlier). The defaults that RKE2 applies allow your clusters to reach the standard with minimal manual work.

You’re still responsible for tightening the OS-level controls on your nodes. This includes applying appropriate kernel parameters and ensuring the etcd data directory is correctly protected.

You can enforce that a safe configuration is required by starting RKE2 with the profile flag set to cis-1.23. RKE2 will then exit with a fatal error if the operating system hasn’t been suitably hardened.

Beyond configuring the OS, you must also set up suitable network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies) and [Pod security admission rules] to secure your cluster’s workloads. The security admission controller can be configured to use profiles which meet the CIS benchmark. This will prevent non-compliant Pods from being deployed to your cluster.

Regularly scanned for threats

The safety of the RKE2 distribution is maintained within its build pipeline. Components are regularly scanned for new common vulnerabilities and exposures (CVEs) using the Trivy container vulnerability tool. This provides confidence that RKE2 itself isn’t harboring threats that could let attackers into your environment.

FIPS 140-2 compliant

K3s lacks any formal security accreditations. RKE2 meets the FIPS 140-2 standard that the U.S. government uses to approve cryptographic modules.

The project’s Go code is compiled using FIPS-validated crypto modules instead of the versions in the Go standard library. Each of the distribution’s components, from the Kubernetes API server through to kubelet and the bundled kubectl binary, are compiled with the FIPS-compatible compiler.

The FIPS mandate means RKE2 can be deployed in government environments and other contexts that mandate verifiable cryptographic performance. The entire RKE2 stack is compliant when you use the built-in components, such as the containerd runtime and etcd data store.

When to use K3s

K3s should be your preferred solution when you’re seeking a performant Kubernetes distribution to run on the edge. It’s also a good choice for single-node development clusters as well as ephemeral environments used in your CI pipelines and other build tools.

This distribution makes the most sense in situations where your primary objective is deploying Kubernetes with all dependencies from a single binary. It’s lightweight, quick to start and easy to manage, so you can focus on writing and testing your application.

When to use RKE2

You should use RKE2 whenever security is critical, such as in government services and other highly regulated industries, including finance and healthcare. As previously stated, the complete RKE2 distribution is FIPS 140-2 compliant and comes hardened with the CIS Kubernetes Benchmark. It’s also the only DISA STIG-certified Kubernetes distribution, meaning it’s approved for use in the most stringent U.S. government environments, including the Department of Defense.

RKE2 is fully certified and tightly aligned with upstream Kubernetes. It omits the K3s components that aren’t standard Kubernetes or that are unstable alpha features. This increases the probability that your deployments will be interoperable across different environments. It also reduces the risk of nonconformance that can occur through oversight when you’re manually hardening Kubernetes clusters.

Near edge computing is another primary use case for RKE2 over K3s. RKE2 ships with support for multiple CNI networking plugins, including Cilium, Calico, and Multus. Multus allows pods to have multiple network interfaces attached, making it ideal for use cases such as telco distribution centers and factories with several different production facilities. In these situations, it’s critical to have robust networking support with different network adapters. K3s bundles Flannel as its built-in CNI plugin; you can install a different provider, but all configuration has to be performed manually. RKE2’s default distribution provides integrated options for common networking solutions.


K3s and RKE2 are two popular Kubernetes distributions that overlap each other in several ways. They both offer a simple deployment experience, frictionless long-term maintenance, and high performance and compatibility.

While designed for tiny and far-edge use cases, K3s are not limited to these scenarios. It’s also widely used for development, in labs, or in resource-constrained environments. However, K3s is not focused on security, so you’ll need to secure and enforce your clusters.

RKE2 takes the usability ethos from K3s and applies it to a fully conformant distribution. It’s tailored for security, close alignment with upstream Kubernetes, and compliance in regulated environments such as government agencies. RKE2 is suitable for both data centers and near-edge situations as it offers built-in support for advanced networking plugins, including Multus.

Which you should choose depends on where your cluster will run and the workloads you’ll deploy. You should use RKE2 if you want a hardened distribution for security-critical workloads or if you need FIPS 140-2 compliance. It will help you establish and maintain a healthy security baseline. K3s remain an actively supported alternative for less sensitive situations and edge workloads. It’s a batteries-included project for when you want to focus on your apps instead of the environment that runs them.

Both distributions can be managed by Rancher and integrated with your DevOps toolbox. You can use solutions like Fleet to deploy applications at scale with GitOps strategies, then head to the Rancher dashboard to monitor your workloads.

Keeping Track of Kubernetes Deprecated Resources

Wednesday, 9 November, 2022

It’s a fact of life: as the Kubernetes API evolves, it’s periodically reorganized or upgraded. This means some Kubernetes resources can be deprecated and later removed. We deserve to keep track of those deprecations and removals easily. For that, we have just released the new deprecated-api-versions policy for Kubewarden, our efficient Kubernetes policy engine that runs policies compiled to Wasm. This policy checks for the usage of Kubernetes resources that have been deprecated or removed from the Kubernetes API.

A look at the deprecated-api-versions policy

This policy has two settings:

  1. kubernetes_version: The starting version begins with where to detect deprecated or removed Kubernetes resources. This setting is mandatory.
  2. deny_on_deprecation: If true, it will deny the operation on a resource that has been deprecated but not yet removed from the Kubernetes version specified by kubernetes_version. This setting is optional and is set to true by default.

As an example, extensions/v1beta1/Ingress was deprecated in Kubernetes 1.14.0, and removed in v1.22.0.

With the following policy settings, the policy accepts an extensions/v1beta1/Ingress in the cluster, yet the policy logs this result:

kubernetes_version: "1.19.0"
deny_on_deprecation: false

In contrast, with these other settings, the policy blocks the Ingress object:

kubernetes_version: "1.19.0"
deny_on_deprecation: true # (the default)

Don’t live in the past

Kubernetes deprecations evolve; we will update the policy as soon as there are new deprecations. The policy versioning scheme tells you up to what version of Kubernetes the policy knows about, e.g. 0.1.0-k8sv1.26.0 means that the policy knows about deprecations up to Kubernetes v1.26.0.

Back to the future

You are about to update your cluster’s Kubernetes version and wonder, will your workloads keep working? Will you be in trouble because of deprecated or removed resources in the new version? Check before updating! Just instantiate the deprecated-api-versions policy with the targetted Kubernetes version and deny_on_deprecation set to false, and get an overview of future-you problems.

In action

As usual, instantiate a ClusterAdmissionPolicy (cluster-wide) or AdmissionPolicy (namespaced) that makes use of the policy.

For this example, let’s work in a k8s cluster of version 1.24.0.

Here’s a definition of a cluster-wide policy that rejects resources that were deprecated or removed in Kubernetes version 1.23.0 and earlier:

kubectl apply -f - <<EOF
apiVersion: policies.kubewarden.io/v1
kind: ClusterAdmissionPolicy
  name: my-deprecated-api-versions-policy
  module: ghcr.io/kubewarden/policies/deprecated-api-versions:v0.1.0-k8sv1.26.0
  mutating: false
  - apiGroups: ["*"]
    apiVersions: ["*"]
    resources: ["*"]
    - CREATE
    - UPDATE
    kubernetes_version: "1.23.0"
    deny_on_deprecation: true

Info: In spec.rules we are checking every resource in every apiGroup and apiVersions. We are doing it for simplicity in this example, yet the policy metadata.yaml comes with long and complete, machine-generated spec.rules that covers just the resources that are deprecated.

You can obtain the right rules by using the kwctl scaffold command.

Our cluster is on version 1.24.0, so for example, without the policy, we could still instantiate an autoscaling/v2beta2/HorizontalPodAutoscaler, even if it is deprecated since 1.23.0 (and will be removed in 1.26.0).

Now with the policy, trying to instantiate an autoscaling/v2beta2/HorizontalPodAutoscaler resource that is already deprecated will result in its rejection:

kubectl apply -f - <<EOF
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
  name: php-apache
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10

Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Error from server: error when creating "STDIN":
admission webhook "clusterwide-my-deprecated-api-versions-policy.kubewarden.admission" denied the request:
autoscaling/v2beta2 HorizontalPodAutoscaler cannot be used. It has been deprecated starting from 1.23.0. It has been removed starting from 1.26.0. It has been replaced by autoscaling/v2.

Have ideas for new policies? Would you like more features on existing ones? Drop us a line at #kubewarden on Slack! We look forward to your feedback 🙂

Kubernetes Jobs Deployment Strategies in Continuous Delivery Scenarios

Wednesday, 5 October, 2022


Continuous Delivery (CD) frameworks for Kubernetes, like the one created by Rancher with Fleet, are quite robust and easy to implement. Still, there are some rough edges you should pay attention to. Jobs deployment is one of those scenarios where things may not be straightforward, so you may need to stop and think about the best way to process them.

We’ll explain here the challenges you may face and will give some tips about how to overcome them.

While this blog is based on Fleet’s CD implementation, most of what’s discussed here also applies to other tools like ArgoCD or Flux.

The problem

Let’s start with a small recap about how Kubernetes objects work.

There are some elements in Kubernetes objects that are immutable. That means that changes to the immutable fields are not allowed once one of those objects is created.

A Kubernetes Job is a good example as the template field, where the actual code of the Job is defined, is immutable. Once created, the code can’t be changed. If we make any changes, we’ll get an error during the deployment.
This is how the error looks when we try to update the Job:

The Job "update-job" is invalid: spec.template: Invalid value: 
core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", 
UID:"", ResourceVersion:"", 
: field is immutable

And this is how the error is shown on Fleet’s UI:

Job replace error as seen on Rancher Fleet

When we do deployments manually and update code within Jobs, we can delete the previous Job manually and relaunch our deployment. However, when using Continuous Delivery (CD), things should work without manual intervention. It’s critical to find a way for the CD process to run without errors and update Jobs in a way that doesn’t need manual intervention.

Thus we have reached the point where we have a new version of a Job code that needs to be deployed, and the old Job should not stop us from doing that in an automated way.

Things to consider before configuring your CD process

Our first recommendation, even if not directly related to the Fleet or Jobs update problem, is always to try using Helm to distribute applications.

Helm installation packages (called Charts) are easy to build. You can quickly build a Chart by putting together all your Kubernetes deployment files in a folder called templates plus some metadata (name, version, etc.) defined in a file called Chart.yaml.

Using Helm to distribute applications with CD tools like Fleet and ArgoCD offers some additional features that can be useful when dealing with Job updates.

Second, In terms of how Jobs are implemented and their internal logic behaves, they must be designed to be idempotent. Jobs are meant to be run just once, and if our CD process manages to update and recreate them on each run, we need to be sure that relaunching them doesn’t break anything.

Idempotency is an absolute must while designing Kubernetes CronJobs, but all those best practices should also be applied here to avoid undesired side effects.


Solution 1: Let Jobs self-destroy after execution

The process is well described in Kubernetes documentation:

“A way to clean up finished Jobs automatically is to use a TTL mechanism provided by a TTL controller for finished resources, by specifying the spec.ttlSecondsAfterFinished field of the Job”


apiVersion: batch/v1
kind: TestJob
  name: job-with-ttlsecondsafterfinished
  ttlSecondsAfterFinished: 5

When we add ttlSecondsAfterFinished to the spec, the TTL controller will delete the Job once it finishes. If the field is not present or the value is empty, the Job won’t be deleted, following the traditional behavior. A value of 0 will fire the deletion just after it finishes its execution. An integer value greater than 0 will define how many seconds will pass after the execution before the Job is deleted. This new feature has been stable since Kubernetes 1.23.

While this seems a pretty elegant way to clean finished jobs (future runs won’t complain about the update of immutable resources), it creates some problems for the CD tools.

The entire CD concepts rely on the fact that our external repos holding our application definition are the source of truth. That means that CD tools are constantly monitoring our deployment and notifying us about changes.

As a result, Fleet detects a change when the Job is deleted, so the deployment is marked as “Modified”:

Rancher Fleet modified Git repo

If we look at the details, we can see how Fleet detected that the Job is missing:

Rancher Fleet modified Git repo details

The change can also be seen in the corresponding Fleet Bundle status:

  - lastUpdateTime: "2022-10-02T19:39:09Z"
    message: Modified(2) [Cluster fleet-default/c-ntztj]; job.batch fleet-job-with-ttlsecondsafterfinished/job-with-ttlsecondsafterfinished
    status: "False"
    type: Ready
  - lastUpdateTime: "2022-10-02T19:39:10Z"
    status: "True"
    type: Processed
    readyClusters: 0/2
    state: Modified
  maxNew: 50
  maxUnavailable: 2
  maxUnavailablePartitions: 0
  observedGeneration: 1

This solution is really easy to implement, with the only drawback of having repositories marked as Modified, which may be confusing over time.

Solution 2: Use a different Job name on each deployment

Here is when Helm comes to our help.

Using Helm’s processing, we can easily generate a random name for the Job each time Fleet performs an update. The old Job will also be removed as it’s no longer in our Git repository.

apiVersion: batch/v1
kind: Job
  name: job-with-random-name-{{ randAlphaNum 8 | lower }}
    realname: job-with-random-name
      restartPolicy: Never


We hope what we have shared here helps to understand better how to manage Jobs in CD scenarios and how to deal with changes on immutable objects.

The code examples used in this blog are available on our GitHub repository: https://github.com/SUSE-Technical-Marketing/fleet-job-deployment-examples

If you have other scenarios that you’d want us to cover, we’d love to hear them and discuss ideas that can help improve Fleet in the future.

Please join us at the Rancher’s community Slack channel for Fleet and Continuous Delivery or at the CNCF official Slack Channel for Rancher.

Epinio and Crossplane: the Perfect Kubernetes Fit

Thursday, 18 August, 2022

One of the greatest challenges that operators and developers face is infrastructure provisioning: it should be resilient, reliable, reproducible and even audited. This is where Infrastructure as Code (IaC) comes in.

In the last few years, we have seen many tools that tried to solve this problem, sometimes offered by the cloud providers (AWS CloudFormation) or vendor-agnostic solutions like Terraform and Pulumi. However, Kubernetes is becoming the standard for application deployment, and that’s where Crossplane fits in the picture. Crossplane is an open source Kubernetes add-on that transforms your cluster into a universal control plane.

The idea behind Crossplane is to leverage Kubernetes manifests to build custom control planes that can compose and provision multi-cloud infrastructure from any provider.

If you’re an operator, its highly flexible approach gives you the power to create custom configurations, and the control plane will track any change, trying to keep the state of your infrastructure as you configured it.

On the other side, developers don’t want to bother with the infrastructure details. They want to focus on delivering the best product to their customers, possibly in the fastest way. Epinio is a tool from SUSE that allows you to go from code to URL in just one push without worrying about all the intermediate steps. It will take care of building the application, packaging your image, and deploying it into your cluster.

This is why these two open source projects fit perfectly – provisioning infrastructure and deploying applications inside your Kubernetes platform!

Let’s take a look at how we can use them together:

# Push our app 
-> % epinio push -n myapp -p assets/golang-sample-app 

# Create the service 
-> % epinio service create dynamodb-table mydynamo 

# Bind the two 
-> % epinio service bind mydynamo myapp 

That was easy! With just three commands, we have:

  1. Deployed our application
  2. Provisioned a DynamoDB Table with Crossplane
  3. Bound the service connection details to our application

Ok, probably too easy, but this was just the developer’s perspective. And this is what Epinio is all about: simplifying the developer experience.

Let’s look at how to set up everything to make it work!


I’m going to assume that we already have a Kubernetes cluster with Epinio and Crossplane installed. To install Epinio, you can refer to our documentation. This was tested with the latest Epinio version v1.1.0, Crossplane v.1.9.0 and the provider-aws v0.29.0.

Since we are using the enable-external-secret-stores alpha feature of Crossplane to enable it, we need to provide the args={--enable-external-secret-stores} value during the Helm installation:

-> % helm install crossplane \
    --create-namespace --namespace crossplane-system \
    crossplane-stable/crossplane \
    --set args={--enable-external-secret-stores}


Also, provide the same argument to the AWS Provider with a custom ControllerConfig:

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
  name: aws-config
  - --enable-external-secret-stores
apiVersion: pkg.crossplane.io/v1
kind: Provider
  name: crossplane-provider-aws
  package: crossplane/provider-aws:v0.29.0
    name: aws-config


Epinio services

To use Epinio and Crossplane together, we can leverage the Epinio Services. They provide a flexible way to add custom resources using Helm charts. The operator can prepare a Crossplane Helm chart to claim all resources needed. The Helm chart can then be added to the Epinio Service Catalog. Finally, the developers will be able to consume the service and have all the needed resources provisioned.


Prepare our catalog

We must prepare and publish our Helm chart to add our service to the catalog.

In our example, it will contain only a simple DynamoDB Table. In a real scenario, the operator will probably define a claim to a Composite Resource, but for simplicity, we are using some Managed Resource directly.

For a deeper look, I’ll invite you to take a look at the Crossplane documentation about composition.

We can see that this resource will “publish” its connection details to a secret defined with the publishConnectionDetailsTo attribute (this is the alpha feature that we need). The secret and the resource will have the app.kubernetes.io/instance label with the Epinio Service instance name. We can correlate the two with this label as Epinio services and configurations.

apiVersion: dynamodb.aws.crossplane.io/v1alpha1
kind: Table
  name: {{ .Release.Name | quote }}
  namespace: {{ .Release.Namespace | quote }}
    app.kubernetes.io/instance: {{ .Release.Name | quote }}
    name: {{ .Release.Name }}-conn
        app.kubernetes.io/instance: {{ .Release.Name | quote }}
        kubed.appscode.com/sync: "kubernetes.io/metadata.name={{ .Release.Namespace }}"
    name: aws-provider-config
    region: eu-west-1
    - key: env
      value: test
    - attributeName: Name
      attributeType: S
    - attributeName: Surname
      attributeType: S
    - attributeName: Name
      keyType: HASH
    - attributeName: Surname
      keyType: RANGE
      readCapacityUnits: 7
      writeCapacityUnits: 7


Note: You can see a kubed annotation in this Helm chart. This is because the generated secrets need to be in the same namespace as the services and applications. Since we are using directly a managed resource then the secret will be in the namespace defined in the default StoreConfig (the crossplane-system  namespace). We are using kubed to copy this secret in the release namespace.


We can now package and publish this Helm chart to a repository and add it to the Epinio Service Catalog by applying a service manifest containing the information on where to fetch the chart.

The application, .epinio.io/catalog-service-secret-types, define the list of the secret types that Epinio should look for. Crossplane will generate the secrets with their own secret type, so we need to explicit it.

apiVersion: application.epinio.io/v1
kind: Service
  name: dynamodb-table
  namespace: epinio
    application.epinio.io/catalog-service-secret-types: connection.crossplane.io/v1alpha1
  name: dynamodb-table
  shortDescription: A simple DynamoDBTable that can be used during development
  description: A simple DynamoDBTable that can be used during development
  chart: dynamodb-test
  chartVersion: 1.0.0
  appVersion: 1.0.0
    name: reponame
    url: https://charts.example.com/reponame
  values: ""


Now we can see that our custom service is available in the catalog:

-> % epinio service catalog

Create and bind a service

Now that our service is available in the catalog, the developers can use it to provision DynamoDBTables with Epinio:

-> % epinio service create dynamodb-table mydynamo

We can check that a dynamo table resource was created and that the corresponding table is available on AWS:

-> % kubectl get tables.dynamodb.aws.crossplane.io
-> % aws dynamodb list-tables

We can now create an app with the epinio push command. Once deployed, we can bind it to our service with the epinio service bind:

-> % epinio push -n myapp -p assets/golang-sample-app
-> % epinio service bind mydynamo myapp
-> % epinio service list

And that’s it! We can see that our application was bound to our service!

The bind command did a lot of things. It fetched the secrets generated by Crossplane and labeled them as Configurations. It also redeployed the application mounting these configurations inside the container.

We can check this with some Epinio commands:

-> % epinio configuration list

-> % epinio configuration show x937c8a59fec429c4edeb339b2bb6-conn

The shown access path is available in the application container. We can use exec in the app and see the content of that files:

-> % epinio app exec myapp



In this blog post, I’ve shown you it’s possible to create an Epinio Service that will use Crossplane to provide external resources to your Epinio application. We have seen that once the heavy lifting is done, the provision of a resource is just a matter of a couple of commands.

While some of these features are not ready, the Crossplane team is working hard on them, and I think they will be available soon!

Next Steps: Learn More at the Global Online Meetup on Epinio

Join our Global Online Meetup: Epinio on Wednesday, September 14th, 2022, at 11 AM EST. Dimitris Karakasilis and Robert Sirchia will discuss the Epinio GA 1.0 release and how it delivers applications to Kubernetes faster, along with a live demo! Sign up here.

Secure Supply Chain: Verifying Image Signatures in Kubewarden

Friday, 20 May, 2022

Secure Supply Chain: Verifying image signatures


After these last releases Kubewarden now has support for verifying the integrity and authenticity of artifacts within Kubewarden using the Sigstore project. In this post, we shall focus on verifying container image signatures using the new verify-image-signatures policy.

To learn more about how Sigstore works, take a look at our previous post

Verify Image Signatures Policy

This policy validates Pods by checking their container images for signatures (that is, containers, init containers and ephemeral containers in the pod)

The policy can inspect all the container images defined inside of a Pod or it can just analyze the ones that are matching a pattern provided by the user.

Container image tags are mutable, they can be changed to point to a completely different content. That’s why it’s a good security practice to reference container images by their immutable checksum.

This policy can rewrite the image definitions that are using a tag to instead reference the image by its checksum.

The policy will:

  • Ensure the image referenced by a tag is satisfying the signature requested by the operator
  • Extract the immutable reference of the image from the signatures
  • Rewrite the image reference to be in the form <image ref>@sha256:<digest>

Let’s see it in action!

For this example, a Kubernetes cluster with Kubewarden already installed is required. The installation process is described in the quick start guide.

We need an image with a signature that we can verify. You can use cosign to sign your images. For this example we’ll use the image ghcr.io/viccuad/app-example:v0.1.0 that was signed using keyless verification.

Obtain the issuer and subject using cosign.

COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/viccuad/app-example:v0.1.0
"Issuer": "https://token.actions.githubusercontent.com",
"Subject": "https://github.com/viccuad/app-example/.github/workflows/ci.yml@refs/tags/v0.1.0"

Let’s create a cluster-wide policy that will verify all images, and let’s use the issuer and subject for verification:

kubectl apply -f - <<EOF
apiVersion: policies.kubewarden.io/v1alpha2
kind: ClusterAdmissionPolicy
  name: verify-image-signatures
  module: ghcr.io/kubewarden/policies/verify-image-signatures:v0.1.4
  - apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    - CREATE
    - UPDATE
  mutating: true
      - image: "*"
          - issuer: "https://token.actions.githubusercontent.com"
            subject: "https://github.com/viccuad/app-example/.github/workflows/ci.yml@refs/tags/v0.1.0"

Wait for the policy to be active:

kubectl wait --for=condition=PolicyActive clusteradmissionpolicies verify-image-signatures

Verify we can create pods with containers that are signed:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
  name: verify-image-valid
  - name: test-verify-image
    image: ghcr.io/viccuad/app-example:v0.1.0

Then check that the image was modified with the digest:

kubectl get pod verify-image-valid -o=jsonpath='{.spec.containers[0].image}'

This will produce the following output:


Finally, let’s try to create a pod with an image that it is not signed:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
  name: verify-image-invalid
  - name: test-verify-image
    image: ghcr.io/kubewarden/test-verify-image-signatures:unsigned

We will get the following error:

Error from server: error when creating "STDIN": admission webhook "clusterwide-verify-image-signatures.kubewarden.admission" denied the request: Pod verify-image-invalid is not accepted: verification of image ghcr.io/kubewarden/test-verify-image-signatures:unsigned failed: Host error: Callback evaluation failure: no signatures found for image: ghcr.io/kubewarden/test-verify-image-signatures:unsigned 


This policy is designed to meet all your needs. However, if you prefer you can build your own policy using one of the SDKs Kubewarden provides. We will show how to do this in an upcoming blog! Stay tuned!

Secure Supply Chain: Securing Kubewarden Policies

Wednesday, 4 May, 2022

With recent releases, the Kubewarden stack supports verifying the integrity and authenticity of content using the Sigstore project.

In this post, we focus on Kubewarden Policies and how to create a Secure Supply Chain for them.


Since a full Sigstore dive is not within the scope for this post, we recommend checking out their nice docs.

In short, Sigstore provides an automatable workflow to match the distributed Open Source development model. The workflow specifies how to digitally sign and verify artifacts which in our case are Kubewarden Policies. It also provides a transparency log to monitor such signatures. The workflow allows to sign artifacts with traditional Public-Private key pairs, or in Keyless mode.

In the keyless mode, signatures are created with short-lived certs using an OpenID Connect (OIDC) service as identity provider. Those short-lived certs are issued by Sigstore’s PKI infrastructure, Fulcio.

Fulcio acts as a Registration Authority, authenticating that you are who you say you are by using an OIDC service (SSO via your own Okta instance, GitHub, Google, etc). Once authenticated, Fulcio acts as a Certificate Authority, issuing the short-lived certificate that you will use to sign artifacts.

These short-lived certificate include the identity information obtained by the OIDC service inside of the certificate extensions attributes. The private key associated with the certificate is then used to sign the object while the certificate itself has a public key that can be used to verify the signatures produced by the private key.

The certificates issued by Fulcio have a short validity because they are generated to be short-lived. This is an interesting property that we will discuss shortly.

Once the artifact is signed, the proof of signature is then sent to an append-only transparency log, Rekor, that allows monitoring of such signatures and protects against timing attacks. The proof of signature is signed by Rekor and this information is stored inside of the signature itself.

By using the timestamp found inside of the proof of signature, the verifier can ensure that the signing action has been performed during the limited lifetime of the certificate.

Due to this the private key associated with the certificate doesn’t need to be safely stored. It can be discarded at the end of the signature process. An attacker could even reuse the private key, but the signature would not be considered valid if used outside of the limited lifetime of the certificate.

Nobody – developers, project leads, or sponsors, needs to have access to keys and Sigstore never obtains your private key. Hence the term keyless. Additionaly, one doesn’t need expensive infra for creating and validating signatures.

Since there’s no need for key secrets and the like in Keyless mode, it is easily automated inside CIs and implemented and monitored in the open. This is one of the reasons that makes it so interesting.

Building a Rust Sigstore stack

The policy server and libs within the Kubewarden stack are responsible for instantiating and running policies. They are written in Rust and therefore, we needed a good Rust implementation of Sigstore features. Since there weren’t any available, we are glad to announce that we have created a new crate, sigstore-rs, under the Sigstore org. This was done in an upstream-first manner and we’re happy to report that it is now taking a life of its own.

Securing kubewarden policies

As you may already know, Kubewarden Policies are small wasm-compiled binaries (~1 to ~6 MB) that are distributed via container registries as OCI artifacts. Let us see how Kubewarden protects policies against Secure Supply Chain attacks by signing and verifying them before they run.

Signing your Kubewarden Policy

Signing a Policy is done in the same way as signing a container image. This means just adding a new layer within the signature to a dedicated signature object managed by Sigstore. In the Sigstore workflow, one can sign with Public-Private keypair, or Keyless. Both can also add key=value annotations to the signatures.

The Public-Private key pair signing is straightforward, using sigstore/cosign:

$ COSIGN_PASSWORD=yourpass cosign generate-key-pair

Private key written to cosign.key
Public key written to cosign.pub

$ COSIGN_PASSWORD=yourpass cosign sign \
  --key cosign.key --annotations blog=yes \

Pushing signature to: ghcr.io/kubewarden/policies/user-group-psp

The Keyless mode is more interesting:

$ COSIGN_EXPERIMENTAL=1 cosign sign \
  --annotations blog=yes \

Generating ephemeral keys...
Retrieving signed certificate...
Your browser will now be opened to:
Successfully verified SCT...
tlog entry created with index: (...)
Pushing signature to: ghcr.io/viccuad/policies/volumes-psp

What happened? cosign prompted us for an OpenID Connect provider on the browser, which authenticated us, and instructed Fulcio to generate an ephemeral private key and a x509 certificate with the associated public key.

If this were to happen in a CI, the CI would provide the OIDC identity token in its environment. cosign has support for detecting some automated environments and producing an identity token. Currently that covers GitHub And Google Cloud, but one can always use a flag.

We shall now detail how it works for policies built by the Kubewarden team in GitHub Actions. First, we call cosign, and sign the policy in keyless mode. The certificate issued by Fulcio includes the following details about the identity of the signer inside of its x503v extensions:

  • An issuer, telling you who certified the image:
  • A subject related to the specific workflow and worker, for example:

If you are curious, and want to see the contents of one of the certificates issued by Fulcio, install the crane cli tool, jq and openssl and execute the following command:

crane manifest \
  $(cosign triangulate ghcr.io/kubewarden/policies/pod-privileged:v0.1.10) | \
  jq -r '.layers[0].annotations."dev.sigstore.cosign/certificate"' | \
  openssl x509 -noout -text -in -

The end result is the same. A signature is added as a new image layer of a special OCI object that is created and managed by Sigstore. You can view those signatures as added layers,with sha256-<sha>.sig in the repo.

Even better, you can use tools like crane or the CLI tool, kwctl to perform the same action as demonstrated below.

kwctl pull <policy_url>; kwctl inspect <policy_url>

If you want to verify policies locally, you now can use kwctl verify:

$ kwctl verify --github-owner kubewarden registry://ghcr.io/kubewarden/policies/pod-privileged:v0.1.10
$ echo $?

When testing policies locally with kwctl pull or kwctl run, you can also enable signature verification by using any verification related flag. For example:

$ kwctl pull --github-owner kubewarden registry://ghcr.io/kubewarden/policies/pod-privileged:v0.1.10
$ echo $?

All the policies from the Kubewarden team are signed in keyless mode by the workers of the CI job, specifically the CI job of Github. We don’t leave certs around and they are verifiable by third parties.

Enforcing signature verification for instantiated Kubewarden policies

You can now configure PolicyServers to enforce that all policies being run need to be signed. When deploying Kubewarden via Helm charts, you can do it so for the default PolicyServer installed by kubewarden-defaults chart.

For this, the PolicyServers have a new spec.VerificationConfig argument. Here, you can put the name of a ConfigMap containing a “verification config”, to specify the needed signatures.

You can obtain a default verification config for policies from the Kubewarden team with:

$ kwctl scaffold verification-config
# Default Kubewarden verification config
# With this config, the only valid policies are those signed by Kubewarden
# infrastructure.
# This config can be saved to its default location (for this OS) with:
#   kwctl scaffold verification-config > /home/youruser/.config/kubewarden/verification-config.yml
# Providing a config in the default location enables Sigstore verification.
# See https://docs.kubewarden.io for more Sigstore verification options.
apiVersion: v1
  - kind: githubAction
    owner: kubewarden
    repo: ~
    annotations: ~
anyOf: ~

The verification config format has several niceties, see its reference docs. For example, kind: githubAction with owner and repo, instead of checking the issuer and subject strings blindly. Or anyOf a list of signatures, with anyOf.atLeast a number of them: this allows for accepting at least a specific number of signatures, and makes migration between signatures in your cluster easy. It’s the little things ?.

If you want support for other CIs (such as GitLab, Jenkins, etc) drop us a note on Slack or file a GitHub issue!

Once you have crafted your verification config, create your ConfigMap:

$ kubectl create configmap my-verification-config \
  --from-file=verification-config=./my-verification-config.yml \

And pass it to your PolicyServers in spec.VerificationConfig, or if using the default PolicyServer from the kubewarden-defaults chart, set it there with for example:

$ helm upgrade --set policyServer.verificationConfig=my-verification-config \
  --wait -n kubewarden kubewarden-defaults ./kubewarden-defaults


Using cosign sign policy authors can sign or author their policies. All the policies owned by the Kubewarden team have already been signed in this way.

With kwctl verify, operators can verify them, and with kwctl inspect (and other tools such as crane manifest), operators can inspect the signatures. We can keep using kwctl pull and kwctl run to test policies locally as in the past, plus now verify their signatures too. Once we are satisfied, we can deploy Kubewarden PolicyServers so they enforce those signatures. If we want, the same verification config format can be used for kwctl and the cluster stack.

This way we are sure that the policies come from their stated authors, and have not been tampered with. Phew!

We, the Kubewarden team, are curious on how you approach this. What workflows are you interested in? What challenges do you have? Drop us a word in our Slack channel or foile a GitHub issue!

There are more things to secure in the chain and we’re excited for what lays ahead. Stay tuned for more blog entries on how to secure your supply chain with Kubewarden!

Managing Sensitive Data in Kubernetes with Sealed Secrets and External Secrets Operator (ESO)

Thursday, 31 March, 2022

Having multiple environments that can be dynamically configured has become akin to modern software development. This is especially true in an enterprise context where the software release cycles typically consist of separate compute environments like dev, stage and production. These environments are usually distinguished by data that drives the specific behavior of the application.

For example, an application may have three different sets of database credentials for authentication (AuthN) purposes. Each set of credentials would be respective to an instance for a particular environment. This approach essentially allows software developers to interact with a developer-friendly database when carrying out their day-to-day coding. Similarly, QA testers can have an isolated stage database for testing purposes. As you would expect, the production database environment would be the real-world data store for end-users or clients.

To accomplish application configuration in Kubernetes, you can either use ConfigMaps or Secrets. Both serve the same purpose, except Secrets, as the name implies, are used to store very sensitive data in your Kubernetes cluster. Secrets are native Kubernetes resources saved in the cluster data store (i.e., etcd database) and can be made available to your containers at runtime.

However, using Secrets optimally isn’t so straightforward. Some inherent risks exist around Secrets. Most of which stem from the fact that, by default, Secrets are stored in a non-encrypted format (base64 encoding) in the etcd datastore. This introduces the challenge of safely storing Secret manifests in repositories privately or publicly. Some security measures that can be taken include: encrypting secrets, using centralized secrets managers, limiting administrative access to the cluster, enabling encryption of data at rest in the cluster datastore and enabling TLS/SSL between the datastore and Pods.

In this post, you’ll learn how to use Sealed Secrets for “one-way” encryption of your Kubernetes Secrets and how to securely access and expose sensitive data as Secrets from centralized secret management systems with the External Secrets Operator (ESO).


Using Sealed Secrets for one-way encryption

One of the key advantages of Infrastructure as Code (IaC) is that it allows teams to store their configuration scripts and manifests in git repositories. However, because of the nature of Kubernetes Secrets, this is a huge risk because the original sensitive credentials and values can easily be derived from the base64 encoding format.

``` yaml

apiVersion: v1

kind: Secret


  name: my-secret

type: Opaque


  username: dXNlcg==

  password: cGFzc3dvcmQ=


Therefore, as a secure workaround, you can use Sealed Secrets. As stated above, Sealed Secrets allow for “one-way” encryption of your Kubernetes Secrets and can only be decrypted by the Sealed Secrets controller running in your target cluster. This mechanism is based on public-key encryption, a form of cryptography consisting of a public key and a private key pair. One can be used for encryption, and only the other key can be used to decrypt what was encrypted. The controller will generate the key pair, publish the public key certificate to the logs and expose it over an HTTP API request.

To use Sealed Secrets, you have to deploy the controller to your target cluster and download the kubeseal CLI tool.

  • Sealed Secrets Controller – This component extends the Kubernetes API and enables lifecycle operations of Sealed Secrets in your cluster.
  • kubeseal CLI Tool – This tool uses the generated public key certificate to encrypt your Secret into a Sealed Secret.

Once generated, the Sealed Secret manifests can be stored in a git repository or shared publicly without any ramifications. When you create these Sealed Secrets in your cluster, the controller will decrypt it and retrieve the original Secret, making it available in your cluster as per norm. Below is a step-by-step guide on how to accomplish this.

To carry out this tutorial, you will need to be connected to a Kubernetes cluster. For a lightweight solution on your local machine, you can use Rancher Desktop.

To download kubeseal, you can select the binary for your respective OS (Linux, Windows, or Mac) from the GitHub releases page. Below is an example for Linux.

``` bash

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.17.3/kubeseal-linux-amd64 -O kubeseal

sudo install -m 755 kubeseal /usr/local/bin/kubeseal


Installing the Sealed Secrets Controller can either be done via Helm or kubectl. This example will use the latter. This will install Custom Resource Definitions (CRDs), RBAC resources, and the controller.

``` bash

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml

kubectl apply -f controller.yaml


You can ensure that the relevant Pod is running as expected by executing the following command:

``` bash

kubectl get pods -n kube-system | grep sealed-secrets-controller


Once it is running, you can retrieve the generated public key certificate using kubeseal and store it on your local disk.

``` bash

kubeseal --fetch-cert > public-key-cert.pem


You can then create a Secret and seal it with kubeseal. This example will use the manifest detailed at the start of this section, but you can change the key-value pairs under the data field as you see fit.

``` bash

kubeseal --cert=public-key-cert.pem --format=yaml < secret.yaml > sealed-secret.yaml


The generated output will look something like this:

``` yaml

apiVersion: bitnami.com/v1alpha1

kind: SealedSecret


  creationTimestamp: null

  name: my-secret

  namespace: default



    password: AgBvA5WMunIZ5rF9...

    username: AgCCo8eSORsCbeJSoRs/...


    data: null


      creationTimestamp: null

      name: my-secret

      namespace: default

    type: Opaque


This manifest can be used to create the Sealed Secret in your cluster with kubectl and afterward stored in a git repository without the concern of any individual accessing the original values.

``` bash

kubectl create -f sealed-secret.yaml


You can then proceed to review the secret and fetch its values.

``` bash

kubectl get secret my-secret -o jsonpath="{.data.user}" | base64 --decode

kubectl get secret my-secret -o jsonpath="{.data.password}" | base64 --decode



Using External Secrets Operator (ESO) to access Centralized Secrets Managers

Another good practice for managing your Secrets in Kubernetes is to use centralized secrets managers. Secrets managers are hosted third-party platforms used to store sensitive data securely. These platforms typically offer encryption of your data at rest and expose an API for lifecycle management operations such as creating, reading, updating, deleting, or rotating secrets. In addition, they have audit logs for trails and visibility and fine-grained access control for operations of stored secrets. Examples of secrets managers include HashiCorp Vault, AWS Secrets Manager, IBM Secrets Manager, Azure Key Vault, Akeyless, Google Secrets Manager, etc. Such systems can put organizations in a better position when centralizing the management, auditing, and securing secrets. The next question is, “How do you get secrets from your secrets manager to Kubernetes?” The answer to that question is the External Secrets Operator (ESO).

The External Secrets Operator is a Kubernetes operator that enables you to integrate and read values from your external secrets management system and insert them as Secrets in your cluster. The ESO extends the Kubernetes API with the following main API resources:

  • SecretStore – This is a namespaced resource that determines how your external Secret will be accessed from an authentication perspective. It contains references to Secrets that have the credentials to access the external API.
  • ClusterSecretStore – As the name implies, this is a global or cluster-wide SecretStore that can be referenced from all namespaces to provide a central gateway to your secrets manager.
  • ExternalSecret – This resource declares the data you want to fetch from the external secrets manager. It will reference the SecretStore to know how to access sensitive data.

Below is an example of how to access data from AWS Secrets Manager and make it available in your K8s cluster as a Secret. As a prerequisite, you will need to create an AWS account. A free-tier account will suffice for this demonstration.

You can create a secret in AWS Secrets Manager as the first step. If you’ve got the AWS CLI installed and configured with your AWS profile, you can use the CLI tool to create the relevant Secret.

``` bash

aws secretsmanager create-secret --name <name-of-secret> --description <secret-description> --secret-string <secret-value> --region <aws-region>


Alternatively, you can create the Secret using the AWS Management Console.

As you can see in the images above, my Secret is named “alias” and has the following values:

``` json


  "first": "alpha",

  "second": "beta"



After you’ve created the Secret, create an IAM user with programmatic access and safely store the generated AWS credentials (access key ID and a secret access key). Make sure to limit this user’s service and resource permissions in a custom IAM Policy.

``` json


  "Version": "2012-10-17",

  "Statement": [


      "Effect": "Allow",

      "Action": [






      "Resource": [







Once that is done, you can install the ESO with Helm.

``` bash

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \

   external-secrets/external-secrets \

    -n external-secrets \



Next, you can create the Secret that the SecretStore resource will reference for authentication. You can optionally seal this Secret using the approach demonstrated in the previous section that deals with encrypting Secrets with kubeseal.

``` yaml

apiVersion: v1

kind: Secret


  name: awssm-secret

type: Opaque


  accessKeyID: PUtJQTl11NKTE5...

  secretAccessKey: MklVpWFl6f2FxoTGhid3BXRU1lb1...


If you seal your Secret, you should get output like the code block below.

``` yaml

apiVersion: bitnami.com/v1alpha1

kind: SealedSecret


  creationTimestamp: null

  name: awssm-secret

  namespace: default



    accessKeyID: Jcl1bC6LImu5u0khVkPcNa==...

    secretAccessKey: AgBVMUQfSOjTdyUoeNu...


    data: null


      creationTimestamp: null

      name: awssm-secret

      namespace: default

    type: Opaque


Next, you need to create the SecretStore.

``` yaml

apiVersion: external-secrets.io/v1alpha1

kind: SecretStore


  name: awssm-secretstore




      service: SecretsManager

      region: eu-west-1




            name: awssm-secret

            key: accessKeyID


            name: awssm-secret

            key: secretAccessKey


The last resource to be created is the ExternalSecret.

``` yaml

apiVersion: external-secrets.io/v1alpha1

kind: ExternalSecret


  name: awssm-external-secret


  refreshInterval: 1440m


    name: awssm-secretstore

    kind: SecretStore


    name: alias-secret

    creationPolicy: Owner


  - secretKey: first


      key: alias

      property: first

  - secretKey: second


      key: alias

      property: second


You can then chain the creation of these resources in your cluster with the following command:

``` bash

kubectl create -f sealed-secret.yaml,secret-store.yaml,external-secret.yaml


After this execution, you can review the results using any of the approaches below.

``` bash

kubectl get secret alias-secret -o jsonpath="{.data.first}" | base64 --decode

kubectl get secret alias-secret -o jsonpath="{.data.second}" | base64 --decode


You can also create a basic Job to test its access to these external secrets values as environment variables. In a real-world scenario, make sure to apply fine-grained RBAC rules to Service Accounts used by Pods. This will limit the access that Pods have to the external secrets injected into your cluster.

``` yaml

apiVersion: batch/v1

kind: Job


  name: job-with-secret





        - name: busybox

          image: busybox

          command: ['sh', '-c', 'echo "First comes $ALIAS_SECRET_FIRST, then comes $ALIAS_SECRET_SECOND"']


            - name: ALIAS_SECRET_FIRST



                  name: alias-secret

                  key: first

            - name: ALIAS_SECRET_SECOND



                  name: alias-secret

                  key: second

      restartPolicy: Never

  backoffLimit: 3


You can then view the logs when the Job has been completed.


In this post, you learned that using Secrets in Kubernetes introduces risks that can be mitigated with encryption and centralized secrets managers. Furthermore, we covered how Sealed Secrets and the External Secrets Operator can be used as tools for managing your sensitive data. Alternative solutions that you can consider for encryption and management of your Secrets in Kubernetes are Mozilla SOPS and Helm Secrets. If you’re interested in a video walk-through of this post, you can watch the video below.

Let’s continue the conversation! Join the SUSE & Rancher Community, where you can further your Kubernetes knowledge and share your experience.

Automate Deployments to Amazon EKS with Skaffold and GitHub Actions

Monday, 28 February, 2022

Creating a DevOps workflow to optimize application deployments to your Kubernetes cluster can be a complex journey. I recently demonstrated how to optimize your local K8s development workflow with Rancher Desktop and Skaffold. If you haven’t seen it yet, you can watch it by viewing the video below.

You might be wondering, “What happens next?” How do you extend this solution beyond a local setup to a real-world pipeline with a remote cluster? This tutorial responds to that question and will walk you through how to create a CI/CD pipeline for a Node.js application using Skaffold and GitHub Actions to an EKS cluster.

All the source code for this tutorial can be found in this repository.


By the end of this tutorial, you’ll be able to:

1. Configure your application to work with Skaffold

2. Configure a CI stage for automated testing and building with GitHub Actions

3. Connect GitHub Actions CI with Amazon EKS cluster

4. Automate application testing, building, and deploying to an Amazon EKS cluster.


To follow this tutorial, you’ll need the following:

-An AWS account.

-AWS CLI is installed on your local machine.

-AWS profile configured with the AWS CLI. You will also use this profile for the CI stage in GitHub Actions.

-A DockerHub account.

-Node.js version 10 or higher installed on your local machine.

-kubectl is installed on your local machine.

-Have a basic understanding of JavaScript.

-Have a basic understanding of IaC (Infrastructure as Code).

-Have a basic understanding of Kubernetes.

-A free GitHub account, with git installed on your local machine.

-An Amazon EKS cluster. You can clone this repository that contains a Terraform module to provision an EKS cluster in AWS. The repository README.md file contains a guide on how to use the module for cluster creation. Alternatively, you can use `eksctl` to create a cluster automatically. Running an Amazon EKS cluster will cost you $0.10 per hour. Remember to destroy your infrastructure once you are done with this tutorial to avoid additional operational charges.

Understanding CI/CD Process

Getting your CI/CD process right is a crucial step in your team’s DevOps lifecycle. The CI step is essentially automating the ongoing process of integrating the software from the different contributors in a project’s version control system, in this case, GitHub. The CI automatically tests the source code for quality checks and makes sure the application builds as expected.

The continuous deployment step picks up from there and automates the deployment of your application using the successful build from the CI stage.

Create Amazon EKS cluster

As mentioned above, you can clone or fork this repository that contains the relevant Terraform source code to automate the provisioning of an EKS cluster in your AWS account. To follow this approach, ensure that you have Terraform installed on your local machine. Alternatively, you can also use eksctl to provision your cluster. The AWS profile you use for this step will have full administrative access to the cluster by default. To communicate with the created cluster via kubectl, ensure your AWS CLI is configured with the same AWS profile.

You can view and confirm the AWS profile in use by running the following command:

aws sts get-caller-identity

Once your K8s cluster is up and running, you can verify the connection to the cluster by running `kubectl cluster-info` or `kubectl config current-context`.

Application Overview and Dockerfile

The next step is to create a directory on your local machine for the application source code. This directory should have the following folder structure (in the code block below). Ensure that the folder is a git repository by running the `git init` command.

Application Source Code

To create a package.json file from scratch, you can run the `npm init` command in the root directory and respond to the relevant questions you are prompted with. You can then proceed to install the following dependencies required for this project.

npm install body-parser cors express 
npm install -D chai mocha supertest nodemon

After that, add the following scripts to the generated package.json:

scripts: {
  start: "node src/index.js",
  dev: "nodemon src/index.js",
  test: "mocha 'src/test/**/*.js'"

Your final package.json file should look like the one below.

  "name": "nodejs-express-test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node src/index.js",
    "dev": "nodemon src/index.js",
    "test": "mocha 'src/test/**/*.js'"
  "repository": {
    "type": "git",
    "url": "git+<your-github-uri>"
  "author": "<Your Name>",
  "license": "ISC",
  "dependencies": {
    "body-parser": "^1.19.0",
    "cors": "^2.8.5",
    "express": "^4.17.1"
  "devDependencies": {
    "chai": "^4.3.4",
    "mocha": "^9.0.2",
    "nodemon": "^2.0.12",
    "supertest": "^6.1.3"

Update the app.js file to initialize the Express web framework and add a single route for the application.

// Express App Setup
const express = require('express');
const http = require('http');
const bodyParser = require('body-parser');
const cors = require('cors');

// Initialization
const app = express();

// Express route handlers
app.get('/test', (req, res) => {
  res.status(200).send({ text: 'Simple Node App Is Working As Expected!' });

module.exports = app;

Next, update the index.js in the root of the src directory with the following code to start the webserver and configure it to listen for traffic on port `8080`.

const http = require('http');
const app = require('./app');

// Server
const port = process.env.PORT || 8080;
const server = http.createServer(app);
server.listen(port, () => console.log(`Server running on port ${port}`));

The last step related to the application is the test folder which will contain the index.js file with code to test the single route you’ve added to our application.

You can redirect to the index.js file in the test folder and add code to test the route you added to the application.

const { expect } = require('chai');
const { agent } = require('supertest');
const app = require('../app');

const request = agent;

describe('Some controller', () => {
  it('Get request to /test returns some text', async () => {
    const res = await request(app).get('/test');
    const textResponse = res.body;
    expect(textResponse.text).to.equal('Simple Node App Is Working As Expected!');

Application Dockerfile

Later on, we will configure Skaffold to use Docker to build our container image. You can proceed to create a Dockerfile with the following content:

FROM node:14-alpine
WORKDIR /usr/src/app
COPY ["package.json", "package-lock.json*", "npm-shrinkwrap.json*", "./"]
RUN npm install 
COPY . .
RUN chown -R node /usr/src/app
USER node
CMD ["npm", "start"]

Kubernetes Manifest Files for Application

The next step is to add the manifest files with the resources that Skaffold will deploy to your Kubernetes cluster. These files will be deployed continuously based on the integrated changes from the CI stage of the pipeline. You will be deploying a Deployment with three replicas and a LoadBalancer service to proxy traffic to the running Pods. These resources can be added to a single file called manifests.yaml.

apiVersion: apps/v1
kind: Deployment
 name: express-test
 replicas: 3
     app: express-test
       app: express-test
     - name: express-test
       image: <your-docker-hub-account-id>/express-test
            memory: 128Mi
            cpu: 500m
       - containerPort: 8080
apiVersion: v1
kind: Service
  name: express-test-svc
    app: express-test
  type: LoadBalancer
  - protocol: TCP
    port: 8080
    targetPort: 8080

Skaffold Configuration File

In this section, you’ll populate your Skaffold configuration file (skaffold.yaml). This file will determine how your application is built and deployed by the Skaffold CLI tool in the CI stage of your pipeline. Your file will specify Docker as the image builder with the Dockerfile you created earlier to define the steps of how the image should be built. By default, Skaffold will use the gitCommit to tag the image create the Deployment manifest file with this image tag.

This configuration file will also contain a step for testing the application’s container image by executing the `npm run test` command that we added to the scripts section of the package.json file. Once the image has been successfully built and tested, it will be pushed to your Docker Hub account in the repository that you specify in the tag prefix.

Finally, we’ll specify that we want Skaffold to use kubectl to deploy the manifest file resources in the manifest.yaml file.

The complete configuration file will look like this:

apiVersion: skaffold/v2beta26
kind: Config
  name: nodejs-express-test
  - image: <your-docker-hub-account-id>/express-test
      dockerfile: Dockerfile
  - context: .
    image: <your-docker-hub-account-id>/express-test
      - command: npm run test
    - manifests.yaml

GitHub Secrets and GitHub Actions YAML File

In this section, you will create a remote repository for your project in GitHub. In addition to this, you will add secrets for your CI environment and a configuration file for the GitHub Actions CI stage.

Proceed to create a repository in GitHub and complete the fields you will be presented with. This will be the remote repository for the local one you created in an earlier step.

After you’ve created your repository, go to the repo Settings page. Under Security, select Secrets > Actions. In this section, you can create sensitive configuration data that will be exposed during the CI runtime as environment variables.

Proceed to create the following secrets:

-AWS_ACCCESS_KEY_ID – This is the AWS-generated Access Key for the profile you used to provision your cluster earlier.

-AWS_SECRET_ACCESS_KEY – This is the AWS-generated Secret Access Key for the profile you used to provision your cluster earlier.

-DOCKER_ID – This is the Docker ID for your DockerHub account.

-DOCKER_PW – This is the password for your DockerHub account.

-EKS_CLUSTER – This is the name you gave to your EKS cluster.

-EKS_REGION – This is the region where your EKS cluster has been provisioned.

Lastly, you are going to create a configuration file (main.yml) that will declare how the pipeline will be triggered, the branch to be used, and the steps that your CI/CD process should follow. As outlined at the start, this file will live in the .github/workflows folder and will be used by GitHub Actions.

The steps that we want to define are as follows:

-Expose our Repository Secrets as environment variables

-Install Node.js dependencies for the application

-Log in to Docker registry

-Install kubectl

-Install Skaffold

-Cache skaffold image builds & config

-Check that the AWS CLI is installed and configure your profile

-Connect to the EKS cluster

-Build and deploy to the EKS cluster with Skaffold

-Verify deployment

You can proceed to update the main.yml file with the following content.

name: 'Build & Deploy to EKS'
      - main
  EKS_CLUSTER: ${{ secrets.EKS_CLUSTER }}
  EKS_REGION: ${{ secrets.EKS_REGION }}
  DOCKER_ID: ${{ secrets.DOCKER_ID }}
  DOCKER_PW: ${{ secrets.DOCKER_PW }}
    name: Deploy
    runs-on: ubuntu-latest
      # Install Node.js dependencies
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
          node-version: '14'
      - run: npm install
      - run: npm test
      # Login to Docker registry
      - name: Login to Docker Hub
        uses: docker/login-action@v1
          username: ${{ secrets.DOCKER_ID }}
          password: ${{ secrets.DOCKER_PW }}
      # Install kubectl
      - name: Install kubectl
        run: |
          curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
          echo "$(<kubectl.sha256) kubectl" | sha256sum --check

          sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
          kubectl version --client
      # Install Skaffold
      - name: Install Skaffold
        run: |
          curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 && \
          sudo install skaffold /usr/local/bin/
          skaffold version
      # Cache skaffold image builds & config
      - name: Cache skaffold image builds & config
        uses: actions/cache@v2
          path: ~/.skaffold/
          key: fixed-${{ github.sha }}
      # Check AWS version and configure profile
      - name: Check AWS version
        run: |
          aws --version
          aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
          aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
          aws configure set region $EKS_REGION
          aws sts get-caller-identity
      # Connect to EKS cluster
      - name: Connect to EKS cluster 
        run: aws eks --region $EKS_REGION update-kubeconfig --name $EKS_CLUSTER
      # Build and deploy to EKS cluster
      - name: Build and then deploy to EKS cluster with Skaffold
        run: skaffold run
      # Verify deployment
      - name: Verify the deployment
        run: kubectl get pods

Once you’ve updated this file, you can commit all the changes in your local repository and push them to the remote repository you created.

git add .
git commit -m "other: initial commit"
git remote add origin <your-remote-repository>
git push -u origin <main-branch-name>

Reviewing Pipeline Success

After pushing your changes, you can track the deployment in the Actions page of the remote repository you set up in your GitHub profile.


This tutorial taught you how to create automated deployments to an Amazon EKS cluster using Skaffold and GitHub Actions. As mentioned in the introduction, all the source code for this tutorial can be found in this repository. If you’re interested in a video walk-through of this post, you can watch the video below.

Make sure to destroy the following infrastructure provisioned in your AWS account:

-Load Balancer created by service resource in Kubernetes.

-Amazon EKS cluster

-VPC and all networking infrastructure created to support EKS cluster

Let’s continue the conversation! Join the SUSE & Rancher Community where you can further your Kubernetes knowledge and share your experience.

Stupid Simple Kubernetes: Service Mesh

Wednesday, 16 February, 2022

We covered the what, when and why of Service Mesh in a previous post. Now I’d like to talk about why they are critical in Kubernetes. 

To understand the importance of using service meshes when working with microservices-based applications, let’s start with a story.  

Suppose that you are working on a big microservices-based banking application, where any mistake can have serious impacts. One day the development team receives a feature request to add a rating functionality to the application. The solution is obvious: create a new microservice that can handle user ratings. Now comes the hard part. The team must come up with a reasonable time estimate to add this new service.  

The team estimates that the rating system can be finished in 4 sprints. The manager is angry. He cannot understand why it is so hard to add a simple rating functionality to the app.  

To understand the estimate, let’s understand what we need to do in order to have a functional rating microservice. The CRUD (Create, Read, Update, Delete) part is easy — just simple coding. But adding this new project to our microservices-based application is not trivial. First, we have to implement authentication and authorization, then we need some kind of tracing to understand what is happening in our application. Because the network is not reliable (unstable connections can result in data loss), we have to think about solutions for retries, circuit breakers, timeouts, etc.  

We also need to think about deployment strategies. Maybe we want to use shadow deployments to test our code in production without impacting the users. Maybe we want to add A/B testing capabilities or canary deployments. So even if we create just a simple microservice, there are lots of cross-cutting concerns that we have to keep in mind.  

Sometimes it is much easier to add new functionality to an existing service than create a new service and add it to our infrastructure. It can take a lot of time to deploy a new service, add authentication and authorization, configure tracing, create CI/CD pipelines, implement retry mechanisms and more. But adding the new feature to an existing service will make the service too big. It will also break the rule of single responsibility, and like many existing microservices projects, it will be transformed into a set of connected macroservices or monoliths. 

We call this the cross-cutting concerns burden — the fact that in each microservice you must reimplement the cross-cutting concerns, such as authentication, authorization, retry mechanisms and rate limiting. 

What is the solution to this burden? Is there a way to implement all these concerns once and inject them into every microservice, so the development team can focus on producing business value? The answer is Istio.  

Set Up a Service Mesh in Kubernetes Using Istio  

Istio solves these issues using sidecars, which it automatically injects into your pods. Your services won’t communicate directly with each other — they’ll communicate through sidecars. The sidecars will handle all the cross-cutting concerns. You define the rules once, and these rules will be injected automatically into all of your pods.   

Sample Application 

Let’s put this idea into practice. We’ll build a sample application to explain the basic functionalities and structure of Istio.  

In the previous post, we created a service mesh by hand, using envoy proxies. In this tutorial, we will use the same services, but we will configure our Service Mesh using Istio and Kubernetes.  

The image below depicts that application architecture.  


  1. Kubernetes(we used the 1.21.3 version in this tutorial) 
  1. Helm (we used the v2) 
  1. Istio (we used 1.1.17) - setup tutorial 
  1. Minikube, K3s or Kubernetes cluster enabled in Docker 

Git Repository 

My Stupid Simple Service Mesh in Kubernetes repository contains all the scripts for this tutorial. Based on these scripts you can configure any project. 

Running Our Microservices-Based Project Using Istio and Kubernetes 

As I mentioned above, step one is to configure Istio to inject the sidecars into each of your pods from a namespace. We will use the default namespace. This can be done using the following command: 

kubectl label namespace default istio-injection=enabled 

In the second step, we navigate into the /kubernetes folder from the downloaded repository, and we apply the configuration files for our services: 

kubectl apply -f service1.yaml 
kubectl apply -f service2.yaml 
kubectl apply -f service3.yaml 

After these steps, we will have the green part up and running: 


For now, we can’t access our services from the browser. In the next step, we will configure the Istio Ingress and Gateway, allowing traffic from the exterior. 

The gateway configuration is as follows: 

apiVersion: networking.istio.io/v1alpha3 
kind: Gateway 
    name: http-gateway 
        istio: ingressgateway 
        - port: 
            number: 80 
            name: http 
            protocol: HTTP 
        hosts:    - “*”  

Using the selector istio: ingressgateway, we specify that we would like to use the default ingress gateway controller, which was automatically added when we installed Istio. As you can see, the gateway allows traffic on port 80, but it doesn’t know where to route the requests. To define the routes, we need a so-called VirtualService, which is another custom Kubernetes resource defined by Istio. 

apiVersion: networking.istio.io/v1b 
kind: VirtualService 
    name: sssm-virtual-services 
    hosts:  - "*" 
    gateways:  - http-gateway 
        - match: 
            - uri: 
                prefix: /service1 
                - destination: 
                    host: service1 
                        number: 80 
        - match: 
            - uri: 
                prefix: /service2 
                - destination: 
                    host: service2 
                        number: 80 

The code above shows an example configuration for the VirtualService. In line 7, we specified that the virtual service applies to the requests coming from the gateway called http-gateway and from line 8 we define the rules to match the services where the requests should be sent. Every request with /service1 will be routed to the service1 container while every request with /service2 will be routed to the service2 container. 

At this step, we have a working application. Until now there is nothing special about Istio — you can get the same architecture with a simple Kubernetes Ingress controller, without the burden of sidecars and gateway configuration.  

Now let’s see what we can do using Istio rules. 

Security in Istio 

Without Istio, every microservice must implement authentication and authorization. Istio removes the responsibility of adding authentication and authorization from the main container (so developers can focus on providing business value) and moves these responsibilities into its sidecars. The sidecars can be configured to request the access token at each call, making sure that only authenticated requests can reach our services. 

apiVersion: authentication.istio.io/v1beta1 
kind: Policy 
    name: auth-policy 
        - name: service1   
        - name: service2   
        - name: service3  
        - name: service4   
        - name: service5   
    - jwt:       
        issuer: "{YOUR_DOMAIN}"      
        jwksUri: "{YOUR_JWT_URI}"   
    principalBinding: USE_ORIGIN 

As an identity and access management server, you can use Auth0, Okta or other OAuth providers. You can learn more about authentication and authorization using Auth0 with Istio in this article. 

Traffic Management Using Destination Rules 

Istio’s official documentation says that the DestinationRule “defines policies that apply to traffic intended for a service after routing has occurred.” This means that the DestionationRule resource is situated somewhere between the Ingress controller and our services. Using DestinationRules, we can define policies for load balancing, rate limiting or even outlier detection to detect unhealthy hosts.  


Shadowing, also called Mirroring, is useful when you want to test your changes in production silently, without affecting end users. All the requests sent to the main service are mirrored (a copy of the request) to the secondary service that you want to test. 

Shadowing is easily achieved by defining a destination rule using subsets and a virtual service defining the mirroring route.  

The destination rule will be defined as follows: 

apiVersion: networking.istio.io/v1beta1 
kind: DestinationRule 
    name: service2 
    host: service2 
    - name: v1      
          version: v1 
    - name: v2     
          version: v2 

As we can see above, we defined two subsets for the two versions.  

Now we define the virtual service with mirroring configuration, like in the script below: 

apiVersion: networking.istio.io/v1alpha3 
kind: VirtualService 
    name: service2 
    - service2   
    - route:     
        - destination:         
          host: service2 
          subset: v1            
            host: service2 
            subset: v2 

In this virtual service, we defined the main destination route for service2 version v1. The mirroring service will be the same service, but with the v2 version tag. This way the end user will interact with the v1 service, while the request will also be sent also to the v2 service for testing. 

Traffic Splitting 

Traffic splitting is a technique used to test your new version of a service by letting only a small part (a subset) of users to interact with the new service. This way, if there is a bug in the new service, only a small subset of end users will be affected.  

This can be achieved by modifying our virtual service as follows: 

apiVersion: networking.istio.io/v1alpha3 
kind: VirtualService 
    name: service2 
    - service2  
    - route:     
        - destination:         
              host: service2         
              subset: v1       
         weight: 90            
         - destination:         
               host: service2 
               subset: v2       
         weight: 10    

The most important part of the script is the weight tag, which defines the percentage of the requests that will reach that specific service instance. In our case, 90 percent of the request will go to the v1 service, while only 10 percent of the requests will go to v2 service. 

Canary Deployments 

In canary deployments, newer versions of services are incrementally rolled out to users to minimize the risk and impact of any bugs introduced by the newer version. 

This can be achieved by gradually decreasing the weight of the old version while increasing the weight of the new version. 

A/B Testing 

This technique is used when we have two or more different user interfaces and we would like to test which one offers a better user experience. We deploy all the different versions and we collect metrics about the user interaction. A/B testing can be configured using a load balancer based on consistent hashing or by using subsets. 

In the first approach, we define the load balancer like in the following script: 

apiVersion: networking.istio.io/v1alpha3 
kind: DestinationRule 
    name: service2 
    host: service2 
                httpHeaderName: version 

As you can see, the consistent hashing is based on the version tag, so this tag must be added to our service called “service2”, like this (in the repository you will find two files called service2_v1 and service2_v2 for the two different versions that we use): 

apiVersion: apps/v1 
kind: Deployment 
    name: service2-v2   
        app: service2 
            app: service2   
        type: Recreate   
                app: service2         
                version: v2     
            - image: zoliczako/sssm-service2:1.0.0         
              imagePullPolicy: Always         
              name: service2         
              - containerPort: 5002         
                      memory: "256Mi"             
                      cpu: "500m" 

The most important part to notice is the spec -> template -> metadata -> version: v2. The other service has the version: v1 tag. 

The other solution is based on subsets. 

Retry Management 

Using Istio, we can easily define the maximum number of attempts to connect to a service if the initial attempt fails (for example, in case of overloaded service or network error). 

The retry strategy can be defined by adding the following lines to the end of our virtual service: 

    attempts: 5 
    perTryTimeout: 10s 

With this configuration, our service2 will have five retry attempts in case of failure and it will wait 10 seconds before returning a timeout. 

Learn more about traffic management in this article. You’ll find a great workshop to configure an end-to-end service mesh using Istio here. 


In this chapter, we learned how to set up and configure a service mesh in Kubernetes using Istio. First, we configured an ingress controller and gateway and then we learned about traffic management using destination rules and virtual services.  

Want to Learn More from our Stupid Simple Series?

Read our eBook: Stupid Simple Kubernetes. Download it here!