Improving trust in the cloud with OpenStack and AMD SEV
This post contains an exciting announcement, but first I need to provide some context!
Ever heard that joke “the cloud is just someone else’s computer”?
Of course it’s a gross over-simplification, but there’s more than a grain of truth in it. And that raises the question: if your applications are running in someone else’s data-centre, how can you trust that they’re not being snooped upon, or worse, invasively tampered with?
Until recently, the answer was “you can’t”. Well, that’s another over-simplification. You could design your workload to be tamperproof; for example even if individual mining nodes in Bitcoin or Ethereum are compromised, the blockchain as a whole will resist the attack just fine. But there’s still the snooping problem.
Hardware to the rescue?
However, there’s some good news on this front. Intel and AMD realised this was a problem, and have both introduced new hardware capabilities to help improve the level to which cloud users can trust the environment in which their workloads are executed, e.g.:
AMD SEV (Secure Encrypted Virtualization) which can encrypt the memory of a running VM with a key which is only accessible to the owner of that VM. This is done on-chip so that even if you have physical access to the machine, it makes it a lot harder to snoop in on the running VM1.
It can also provide the guest owner with an attestation which cryptographically proves that the memory was encrypted correctly and can only be decrypted by the owner.
- Intel MKTME (Multi-Key Total Memory Encryption) which is a similar approach.
But even with that hardware support, there is the question to what degree anyone can trust public clouds run on proprietary technology. There is a growing awareness that Free (Libre) / Open Source Software tends to be inherently more secure and trustworthy, since its transparency enables unlimited peer review, and its openness allows anyone to contribute improvements.
And these days, OpenStack is pretty much the undisputed king of the Open Source cloud infrastructure world.
An exciting announcement
So I’m delighted to be able to announce a significant step forward in trustworthy cloud computing: as of this week, OpenStack is now able to launch VMs with SEV enabled! (Given the appropriate AMD hardware, of course.)
While this is “only” an MVP and far from the end of the journey (see below), it’s an important milestone in a strong partnership between my employer SUSE and AMD. We started work on adding SEV support into OpenStack around a year ago:
This resulted in one of the most in-depth technical specification documentations I’ve ever had to write, plus many months of intense collaboration on the code and several changes in design along the way.
I’d like to thank not only my colleagues at SUSE and AMD for all their work so far, but also many members of the upstream OpenStack community, especially the Nova team. In particular I enjoyed fantastic support from the PTL (Project Technical Lead) Eric Fried, and several developers at Red Hat, which I think speaks volumes to how well the “coopetition” model works in the Open Source world.
The rest of this post gives a quick tour of the implementation via screenshots and brief explanations, and then concludes with what’s planned next.
OpenStack’s Compute service (nova) will automatically detect the presence of the SEV feature on any compute node which is configured to support it. You can optionally configure how many slots are available on the memory controller for encryption keys. One is used for each guest, so this effectively acts as the maximum number of guest VMs which can concurrently use SEV. Here you can see the configuration of this option, and how nova handles the inventory. Note that it also registers an SEV trait on the compute host, so that in the future if the cloud has a mix of hardware offering different guest memory encryption technologies, you’ll be able to choose which one you want for any given guest, if you need to.
SEV can be enabled by the operator by adding a new
hw:mem_encryption “extra spec” which is a property on nova’s flavors. As already shown in the screenshot above, this can be done through Horizon, OpenStack’s web dashboard. However it can also be set per-image via a similarly-named property
and of course this can all be done via the command-line too:
Notice the presence of a few other image properties which are crucial for SEV to function correctly. (These are explained fully in the documentation.)
Once booted, an SEV VM instance looks and behaves pretty much like any other OpenStack VM:
However there are some limitations, e.g. it cannot yet be live-migrated or suspended:
Behind the scenes, nova takes care of quite a few important details in how the VM is configured in libvirt. Firstly it performs sanity checks on the flavor and image properties. Then it adds a crucial new
and also enables IOMMU for
This area of technology is new and rapidly evolving, so there is still plenty of work left to be done, especially on the software side.
Of course we’ll be adding this functionality to SUSE OpenStack Cloud, initially as a technical preview for our customers to try out.
Probably the most important feature needed next on the SEV side is the ability to verify the attestation which cryptographically proves that the memory was encrypted correctly and can only be decrypted by the owner. In addition specification of the work required to add support to OpenStack for Intel’s MKTME already started, so I would expect that to continue.
There are still potential attacks, e.g. snooping unencrypted memory cache or CPU registers. Work by AMD and others is ongoing to address these.