Developer Sandbox Upgraded Retrospective

A few weeks ago we upgraded the Developer Sandbox from Cloud Application Platform 2.0 to 2.1 successfully! It went so well that I wanted to wait a little while to write this so I could be sure that it did indeed go as smoothly as it appeared to!

Prior to the upgrade, I had a few hesitations about the upgrade because we were also upgrading the Kubernetes version from 1.14 to 1.18 which meant laddering up worker and control plane versions. This meant compute nodes being destroyed and recreated several times in fairly rapid succession. Is also meant an upgrade to Minibroker from 1.0 to 1.1 and an upgrade to the latest version of Stratos as well! All of these moving parts meant a significant amount of risk as well as an unknown amount of downtime.

On the day of the upgrade, we started with laddering up the underlying Kubernetes. The process to do this is

• Upgrade control plane to n+1 version
• Create new worker nodes with the n+2 version
• Cordon then drain old nodes
• Repeat with control plane n+3
• etc…

Every step of that process has the potential to bring an unsuspecting platform to its knees…

I’m happy to report that there was never a platform downtime and applications with at least 2 replicas stayed available as well. This is very impressive!

Next, we upgraded the Platform version to 2.1. This is a two step process: upgrade the Quarks operator, then upgrade the platform spec. As with the K8s upgrade, this did not result in any platform downtime and only single replica workloads saw any interruption.

As always, Stratos and Minibroker were as easy as a helm upgrade.

Congratulations to the engineering teams behind the product! This shows that having a team who truly understands what it takes to build a production ready Platform as a Service can generate real value.