Is it possible to perform etcd snapshots to an s3 endpoint with a certificate signed by a custom CA?

This document (000020232) is provided subject to the disclaimer at the end of this document.

Situation

Question

Is it possible to perform etcd snapshots to an S3 endpoint with a certificate signed by a custom certificate authority (CA)?

Pre-requisites

Kubernetes clusters provisioned via the RKE CLI v0.2.x, or Rancher launched Kubernetes clusters via Rancher v2.2.x.

Answer

Rancher v2.2.0 - v2.2.4 and RKE CLI v0.2.0 - v0.2.4

In Rancher v2.2.0 - v2.2.4 and RKE CLI v0.2.0 - v0.2.4 it is not possible to configure etcd snapshots to use a custom CA. Attempts to perform etcd snapshots to an S3 endpoint with a certificate signed by a custom CA will fail with an error similar to the following:

FATA[0002] Failed to take one-time snapshot, exit code [1]: time="2019-04-29T08:37:15Z" level=fatal msg="faield to set s3 server: failed to check s3 bucket:rke, err:Get https://s3.example.com/rke/?location=: x509: certificate signed by unknown authority"

Rancher v2.2.5+

In Rancher v2.2.5 and above it is possible to specify a custom CA for the S3 endpoint within the S3 backup options. Expanding 'Show advanced options' under the 'Edit Cluster' view, a 'Custom CA Certificate' field is shown when the s3 backup target is selected, enabling you to enter the certificate or upload this from file.

RKE v0.2.5+

With the RKE CLI v0.2.5 and above it also possible to specify a custom CA for the S3 endpoint within the S3 backup options. To do you specify the certificate via the custom_ca field in the s3backupconfig block of the cluster configuration YAML. The cert should be provided as string, with newlines replaced with \n, per the example below:

services:
  etcd:
    backup_config:
      interval_hours: 12
      retention: 6
      s3backupconfig:
        access_key: S3_ACCESS_KEY
        secret_key: S3_SECRET_KEY
        bucket_name: s3-bucket-name
        region: ""
        endpoint: s3.amazonaws.com
        custom_ca: "-----BEGIN CERTIFICATE-----\nMIIDazCCAlOgAwIBAgIUMoCmUpa4u2UJWqNIkizFbpeJkwowDQYJKoZIhvcNAQEL\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0xOTA5MTgwOTI4NDBaFw0yMjA3\nMDgwOTI4NDBaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQDIW8aN2vszkiNAqykYvqivZgWPRqEukPSAZz39Qtyx\nkv2wl3B29chBzw5+vjG6veaUnWufOpGeiwglL2PEBOMI0a62zmmm3ttyJDy1lY+A\ncuxZ1+hveWjWrA2B2bN69/wdkQTQu6ZLoguk+8mRFBZ7ghu6YTZQfczBsHlDxUpA\n77qQunE4RmcQzOBHoWmMkSSxSGMBsVIj2rRihtVqpgbrMr3/LtCqzqsF+UcroJPC\nIIBd8bSFlcgkWLnJdqlSa8s1PUodcKD3q6mbMZPDudraszuRgLyC5pIylGQOk+XF\nMjf2I8zkkAV4QtfSpgBpNXbZEZ3a6CPhveDZqoZN4rxTAgMBAAGjUzBRMB0GA1Ud\nDgQWBBTD/EagPfxclAlfViV5kKLq0YwBYzAfBgNVHSMEGDAWgBTD/EagPfxclAlf\nViV5kKLq0YwBYzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQB0\nyJ6vjtmuvBEKuNgWwIJLh2CqZubUL+lUQGi1NhdFzkXj7+fLeLjqsmbi2Xj/qQ5n\nooI/p4MeHfYrUqqS7nqTBIsRZQZDZcKUYTZWzDRBdQZtxvEsB1WUq5+nsCQqVuZO\n+ICsXQFL45xDKaWOoRMH8z9JksYf2CSKeRWViAFElC/IDwf8d5mtufe17h5vlyPR\nLaIMJ37vyAosN6h8icztVHRzfcIjp1KLqwaGfaOrNSCv8zja9YsD6kbYL64lKND4\nHiOJy3oSjjjTNdnXjIO44Ngo7L4TWF1CshFlsRF3a5/Jw+NmsEV46Vq41YcuRX9E\n5JYZWzGRsPDeG4vrzWrV\n-----END CERTIFICATE-----"

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.