Release notes usually only list changes that happened between two subsequent releases. Certain important entries from the release notes documents of previous product versions may be repeated. To make such entries easier to identify, they contain a note to that effect.

2 SUSE CaaS Platform #

SUSE CaaS Platform is an enterprise-ready Kubernetes-based container management solution used by application development and DevOps teams to more easily and efficiently deploy and manage containerized applications and services. Enterprises use SUSE CaaS Platform to reduce application delivery cycle times and improve business agility.

3 Supported Platforms #

This release supports deployment on:

SUSE OpenStack Cloud 8
VMware ESXi 6.7
KVM
Bare Metal x86_64
Amazon Web Services (technological preview)

(SUSE CaaS Platform 4.5.5 supports hardware that is certified for SLES through the YES certification program. You will find a database of certified hardware at https://www.suse.com/yessearch/.)

4 Changes in 4.5.5 #

4.1 New release of addon images #

4.5.5 upgrades Kubernetes to 1.18.20 including a backport for relevant security bugs.

4.2 Required Actions #

In order to use the latest skuba fixes, you need to update the admin workstation. For detailed instructions, see the Administration Guide
In order to update to Kubernetes 1.18.20 follow the instructions in the Admin Guide

4.3 Bugs Fixed in 4.5.5 since 4.5.4 #

bsc#1189416 [kubernetes] CVE-2021-25741: Symlink Exchange Can Allow Host Filesystem Access
bsc#1182185 [kubernetes] CVE-2021-3121: gogo/protobuf: plugin/unmarshal/unmarshal.go lacks certain index validation

5 Changes in 4.5.4 #

5.1 New release of addon images #

4.5.4 brings a new fresh rebuild of container images including all SLES patches up to the end of April 2021.

5.2 Required Actions #

In order to use the latest skuba fixes, you need to update the admin workstation. For detailed instructions, see the Administration Guide
Run skuba addons upgrade apply to update addons to the new release.

5.3 Bugs Fixed in 4.5.4 since 4.5.3 #

bsc#1183541 [cri-o] Warning ImageGCFailed node
bsc#1181585 [kubernetes] Invalid value: provided port is already allocated

6 Changes in 4.5.3 #

6.1 New release of Cilium addon #

4.5.3 brings a new release of cilium image including some security fixes. See the list below.

6.2 Required Actions #

In order to use the latest skuba fixes, you need to update the admin workstation. For detailed instructions, see the Administration Guide
Run skuba addons upgrade apply to update addons to the new release.

6.3 Bugs Fixed in 4.5.3 since 4.5.2 #

bsc#1173559 [envoy-proxy] VUL-0: CVE-2020-12605,CVE-2020-8663,CVE-2020-12603,CVE-2020-12604: multiple resource exhaustion issues
bsc#1177348 [bpftool] Bpftool is installing a program with bpf_probe_write_user helper that may corrupt user memory
bsc#1178931 [cilium] Add missing ClusterRoles rules

6.4 Known Issues #

Kubernetes public announcement: CVE-2020-8554 Man in the middle using LoadBalancer or ExternalIPs:

Multi-tenant clusters that grant tenants the ability to create and update services and pods are vulnerable. This issue is a design flaw that cannot be mitigated without user-facing changes. With this public announcement, we can begin conversations about a long-term fix.

To restrict the use of external IPs we are providing an admission webhook container: k8s.gcr.io/multitenancy/externalip-webhook:v1.0.0. The source code and deployment instructions are published at https://github.com/kubernetes-sigs/externalip-webhook.

7 Changes in 4.5.2 #

7.1 Kubernetes Update #

4.5.2 brings in a Kubernetes update, precisely to 1.18.10. The list of features and bug fixes is long, see:

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v11810

7.2 Helm 3 is default helm #

4.5.2 makes Helm 3 the default helm. For further details see Installing Helm.

7.3 Required Actions #

To apply the fix for the CRI-O bug mentioned below, you must perform skuba-update. See https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_cluster_updates.html#_base_os_updates for more details.
In order to use the latest skuba fixes, you need to update the admin workstation. For detailed instructions, see the Administration Guide
Run skuba addons upgrade apply to update addons to the new release.

7.4 Bugs Fixed in 4.5.2 since 4.5.1 #

bsc#1174951 [etcd] - VUL-0: CVE-2020-15106,CVE-2020-15112: etcd: a large slice causes panic in decodeRecord method and improper checks in entry index
bsc#1174219 [cri-o-1.18] - Mixed /etc/sysconfig/kubelet changes between older/newer CaaSP4 nodes
bsc#1178785 [cri-o-1.18] - cri-o bug# 3972 - failed to get image stats: rpc error: code = Unknown desc = layer not known
bsc#1173165 [gangway] - Insecure SSL/TLS versions in Gangway
bsc#1176904 [skuba] - cilium-secret does not get monitored or renewed causing cluster downtime
bsc#1175352 [skuba] - Regular pod is assigned to psp kucero by default
bsc#1176903 [skuba] - skuba does not renew the client-cert of the admin.conf in the skuba cluster directory
bsc#1176578 [skuba] - Complain about certificate in /etc/kubernetes/kubelet.conf
bsc#1173055 [skuba] - After changing dex to AD configuration - error 401 after successful login
bsc#1176225 [skuba] - Upgraded v4.5 cluster is running etcd from v4 namespace
bsc#1172270 [skuba] - cilium-init:1.6.6 does not exist in registry
bsc#1177361 [skuba] - VUL-0: CVE-2020-8030: skuba: Insecure /tmp usage when joining node to cluster
bsc#1177362 [skuba] - VUL-0: CVE-2020-8029: skuba: Insecure handling of private key
bsc#1177661 [kubernetes-1.18] - VUL-0: CVE-2020-8565: kubernetes: Incomplete fix for CVE-2019-11250 allows for token leak in logs when logLevel >= 9
bsc#1177660 [kubernetes-1.18] - VUL-0: CVE-2020-8564: kubernetes: Docker config secrets leaked when file is malformed and loglevel >= 4

7.5 Documentation Changes #

Added migration instructions for the new Helm 3 default.

7.6 Known Issues #

8 Changes in 4.5.1 #

8.1 Deprecations in 4.5.1 #

None

8.2 Required Actions #

Run skuba addons upgrade apply to update Cilium images to rev3 which has the bug fixes to be installed.
In order to use the latest skuba fixes, you need to update the admin workstation. For detailed instructions, see the Administration Guide
Envoy security fixes will be updated with skuba addons upgrade apply. The bugs and security fixes applied are listed in the following sections.

8.3 Bugs Fixed in 4.5.1 since 4.5.0 #

bsc#1173559 [envoy] - CVE-2020-12605,CVE-2020-8663,CVE-2020-12603,CVE-2020-12604: envoy-proxy, cilium-proxy: multiple resource exhaustion issues
bsc#1176755 [helm3] - CVE-2020-15184: helm3: alias field on a Chart.yaml is not properly sanitized
bsc#1176754 [helm] - CVE-2020-15185: helm3: Helm repository can contain duplicates of the same chart
bsc#1176752 [helm3] - CVE-2020-15187: helm3: plugin can contain duplicates of the same entry
bsc#1174075 [kubernetes] - Changing %{_libexecdir} breaks some packages which are misusing the macro
bsc#1167073 [envoy] - CaaSPv5: envoy-proxy doesn’t build on SLE15SP2
bsc#1176753 [helm3] - CVE-2020-15186: helm3: plugin names are not sanitized properly

8.4 Security issues fixed in 4.5.1 since 4.5.0 #

CVE-2020-12603: "Envoy version 1.14.2, 1.13.2, 1.12.4 or earlier may consume excessive amounts of memory when proxying HTTP/2 requests or responses with many small (i.e. 1 byte) data frames."
CVE-2020-12604: "Envoy version 1.14.2, 1.13.2, 1.12.4 or earlier is susceptible to increased memory usage in the case where an HTTP/2 client requests a large payload but does not send enough window updates to consume the entire stream and does not reset the stream."
CVE-2020-12605: "Envoy version 1.14.2, 1.13.2, 1.12.4 or earlier may consume excessive amounts of memory when processing HTTP/1.1 headers with long field names or requests with long URLs."
CVE-2020-8663: "Envoy version 1.14.2, 1.13.2, 1.12.4 or earlier may exhaust file descriptors and/or memory when accepting too many connections."
CVE-2020-15187: "In Helm before version 3.3.2, a Helm plugin can contain duplicates of the same entry, with the last one always used. If a plugin is compromised, this lowers the level of access that an attacker needs to modify a plugin’s install hooks, causing a local execution attack."
CVE-2020-15185: "In Helm before version 3.3.2, a Helm repository can contain duplicates of the same chart, with the last one always used. If a repository is compromised, this lowers the level of access that an attacker needs to inject a bad chart into a repository."
CVE-2020-15184: "In Helm before version 3.3.2 there is a bug in which the alias field on a Chart.yaml is not properly sanitized. This could lead to the injection of unwanted information into a chart."
CVE-2020-15186: "In Helm before version 3.3.2 plugin names are not sanitized properly. As a result, a malicious plugin author could use characters in a plugin name that would result in unexpected behavior, such as duplicating the name of another plugin or spoofing the output to hellm --help."

8.5 Documentation Changes #

None

8.6 Known Issues #

https://bugzilla.suse.com/show_bug.cgi?id=1176225 - Upgraded v4.5 cluster is running etcd from v4 namespace
https://bugzilla.suse.com/show_bug.cgi?id=1172270 - cilium-init:1.6.6 does not exist in registry
Kubeproxy is not fully deprecated since envoyproxy requires support of Linux Kernel 5.3 and upwards.
If the cluster node(s) was bootstrapped/joined before kubernetes version 1.17, you have to manually modify the contents of /etc/kubernetes/kubelet.conf to point to the automatically rotated kubelet client certificates by replacing client-certificate-data and client-key-data with:
```
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
```

9 What Is New in 4.5.0 #

9.1 Migrating to 4.5.0 #

If you are coming from the latest release, just follow the instructions described here. Otherwise, if you are coming from older releases, you will have to upgrade to the previous SUSE CaaS Platform release first, and then follow the link above.

9.2 Base Operating System Is Now SLES 15 SP2 #

SUSE CaaS Platform 4 uses standard SLES 15 SP2 as the base platform OS. SUSE CaaS Platform can be installed as an extension on top of that. Because SLES 15 is designed to address both cloud-native and legacy workloads. these changes make it easier for customers who want to modernize their infrastructure by moving existing workloads to a Kubernetes framework.

9.2.1 Changes in the Installation Media #

Please pay attention to the change in the installation media of SLES 15 SP2. The Unified Installer and Packages DVDs known from SUSE SLES 15 SP1 are deprecated and have been replaced by Online Installation Media and Full Installation Media, for further details see this section of the SLES 15 SP2 Release Notes.

For further information on notable changes when going from SLES 15 SP1 to SLES 15 SP2, also refer to the SLES 15 SP2 Release Notes.

9.3 Support disabling routable check in VMware #

VMware Terraform config now supports the option wait_for_guest_net_routable to disable Terraform from waiting for a routable network on VM creation. This is useful to prevent an upstream issue in which Terraform hangs waiting for a routable IP even when the VM has one. For more information check the bug report

9.4 Changes to the Kubernetes Stack #

9.4.1 Updated Kubernetes #

SUSE CaaS Platform 4.5.0 comes with Kubernetes 1.18. The new Kubernetes version includes for example notable upgrades to kubectl, storage enhancements, Horizontal Pod Autoscaler (HPI) and advanced scheduling. Some of these features have been available in SUSE CaaS Platform for a couple of versions now.

You can find a list of changes directly relevant to SUSE CaaS Platform here: Changes from Kubernetes 1.17 to 1.18

For a more generalized summary, you can view the SUSE blog or see the full version in the official Kubernetes documentation.

9.4.2 New Kubernetes package names #

From SUSE CaaS Platform version 4.5.0 onwards, Kubernetes packages are built with new package names; for example:

kubernetes-1.18-1.18.6-1.1.src.rpm
kubernetes-1.18-apiserver-1.18.6-1.1.x86_64.rpm
kubernetes-1.18-client-1.18.6-1.0.x86_64.rpm
kubernetes-1.18-controller-manager-1.18.6-1.1.x86_64.rpm
kubernetes-1.18-extra-1.18.6-1.1.x86_64.rpm
kubernetes-1.18-kubeadm-1.18.6-1.1.x86_64.rpm
kubernetes-1.18-kubelet-1.18.6-1.1.x86_64.rpm
kubernetes-1.18-proxy-1.18.6-1.1.x86_64.rpm
kubernetes-1.18-scheduler-1.18.6-1.1.x86_64.rpm

In SUSE CaaS Platform Kubernetes packages use the kubernetes-1.18-name pattern. The reason for this is because the openSUSE Kubic project already uses the pattern kubernetes1.18-name but with different package (dependency) semantics underneath.

Having two identically named packages from SUSE (one from SUSE CaaS Platform, one from Kubic) with different semantics could lead to confusion and conflicts/problems with installation of the software from either project.

9.4.3 CRI-O #

This release upgrades CRI-O to 1.18, which brings support for Kubernetes 1.18. This new version, though, requires some manual steps for the upgrade, since the configuration files have been migrated from sysconfig to /etc/crio/crio.conf.d/. In order to upgrade, then, you will have to perform the following command before upgrading each node:

skuba cluster upgrade localconfig

After this you will be able to perform all the skuba commands that you would call normally when upgrading SUSE CaaS Platform.

9.5 Kubernetes Audit Log with rsyslog agent #

Starting with this version of SUSE CaaS Platform, the Kubernetes audit log will be forwarded to the centralized logging service using log-agent-rsyslog.

9.6 Helm 3 #

Helm 3 is now available in the SUSE CaaS Platform zypper repositories. Helm 3 includes many enhancements over Helm 2.

Removal of Tiller, which simplifies deployment and management.
Configuration is now stored in Kubernetes Secrets rather than ConfigMaps.
Chart releases are now scoped to Kubernetes namespaces.
Charts can be searched for in Helm Hub or repositories using the helm search command.
The chart apiVersion has been bumped to "v2" to allow specification changes.
Several helm command line options have been changed, notably the name for a release is now a required parameter.

These changes and more are documented by the community in the Helm FAQs and Migrating 2 to 3 document.

Helm 2 is still installed by default to avoid forcing a migration of chart releases when upgrading SUSE CaaS Platform. The Helm 3 version is not a drop in upgrade of Helm 2, though most charts written for "apiVersion: v1" will work with Helm 3. However, as the stated end of community support for Helm 2 is November 13, 2020, you should migrate to Helm 3 as soon as reasonable.

Please Note: We recommend using Helm 3 for fresh installations. See Official Helm blog.

The Helm 2 and Helm 3 packages have been named helm2 and helm3 to allow both versions to be installed to facilitate migration. Through the use of update-alternatives the version used by 'helm' command line can be set. For detailed steps to select alternatives, see the Administration Guide.

In the next SUSE CaaS Platform release, the default Helm version will be switched to Helm 3, and in the following release Helm 2 will be dropped.

9.7 Required Actions #

9.7.1 Update CRI-O configuration format #

You must update the CRI-O configuration files on each node before upgrading, please refer to Section 9.4.3, “CRI-O”.

9.7.2 (VMWare) Enable "Jumbo Frames" #

To avoid running into problems with certain integrations and components, VMWare users must enable "Jumbo Frames" and configure maximum transmission unit (MTU) to 9000. Refer to: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-53F968D9-2F91-41DA-B7B2-48394D997F2A.html

10 Documentation Changes #

Added the migration instructions from SUSE CaaS Platform 4.0.x/4.2.x
Split the air gap deployment guide into a separate document: Airgap Deployment Guide
Installation steps for Helm 3 and alternate command lines for Helm 3 documented in the Administration Guide.
Instructions on how to migrate from Helm 2 to 3, see the Administration Guide.
New chapter on Addon Certificate Rotation in the Administration Guide.
New chapter on GPU-Dependent Workloads in the Administration Guide.
Completely overhauled logging section, reordered Admin guide to incorporate this change.
Various other fixes and improvements, refer to: https://github.com/SUSE/doc-caasp/releases/tag/release-4.5

11 Known Issues #

11.1 In the upgrade process, after the restart of CRI-O and kubelet, some pods might not run properly #

This can happen when there are multiple instances of a PodSandbox in a "NotReady" state. As a workaround please make sure to remove any pod in the "NotReady" state using crictl rmp <podid>. Further it is advisable to drain the node that is being upgrade before actually starting the upgrade procedure.

The upstream fix is https://github.com/cri-o/cri-o/pull/4006 which will be included in the next release.

Reference: https://github.com/SUSE/avant-garde/issues/1808

11.2 etcd: CVE-2020-15106 and CVE-2020-15112 #

Note the version of etcd shipped with CaaSP 4.5.0 contains two security issues identified as CVE-2020-15106 and CVE-2020-15112

The etcd endpoints should only be accessible inside the cluster if you have set up the firewall rules / network segmentation, following our suggestions in the admin guide; etcd should only be accessible by k8s nodes (or by trusted nodes). Exploiting this vulnerability requires an attacker to take control of the etcd leader in order to send crafted WAL entries, which means access to the SSL certs or local machine access.

Fixes for these will be provided as a maintenance update.

11.3 envoy: CVE-2020-12605,CVE-2020-8663,CVE-2020-12603 and CVE-2020-12604 #

Note that the version of envoy shipped with CaaSP 4.5.0 contains security issues idendified as CVE-2020-12605,CVE-2020-8663,CVE-2020-12603 and CVE-2020-12604

These are "Denial of Service" vulnerabilities, and do not expose systems to unauthorized access or data exfiltration. A fix for them will be provided as a maintenance update.

11.4 Helm 2to3 migration plugin requires internet connection to install #

The installer for the Helm 2to3 plugin is written to pull the plugin from the official community github site at github.com/helm/helm-2to3. This could cause a problem in an air-gapped SUSE CaaS Platform installation where an open internet connection is not available.

The simplest workaround is to move the management system (such as a laptop) out of the internal network, install the plugin, then move back in to perform the migration.

11.5 Upgraded v4.5 cluster is running etcd from v4 namespace (bsc#1176225) #

After performing upgrade from CaaSP 4.2.3 to CaaSP 4.5.0 GMC3 the migrated cluster is running etcd from v4 namespace then. If 4.5.0 is deployed from scratch the cluster is using correct image from v4.5 namespace.

The outdated etcd image from v4 namespace doesn’t have any impact for the cluster functionality.

12 Legal Notices #

SUSE makes no representations or warranties with regard to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, SUSE reserves the right to revise this publication and to make changes to its content, at any time, without the obligation to notify any person or entity of such revisions or changes.

Further, SUSE makes no representations or warranties with regard to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, SUSE reserves the right to make changes to any and all parts of SUSE software, at any time, without any obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may be subject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classifications to export, re-export, or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as specified in U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical/biological weaponry end uses. Refer to https://www.suse.com/company/legal/ for more information on exporting SUSE software. SUSE assumes no responsibility for your failure to obtain any necessary export approvals.

This release notes document is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC-BY-SA-4.0). You should have received a copy of the license along with this document. If not, see https://creativecommons.org/licenses/by-sa/4.0/.

SUSE has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed at https://www.suse.com/company/legal/ and one or more additional patents or pending patent applications in the U.S. and other countries.

For SUSE trademarks, see SUSE Trademark and Service Mark list (https://www.suse.com/company/legal/). All third-party trademarks are the property of their respective owners.