SUSE Linux Enterprise, SUSE OpenStack Cloud, SUSE Enterprise Storage, and other SUSE-based products

System Performance Implications of Meltdown, Spectre, and L1TF Vulnerabilities in SUSE-based Products

This document provides information about released mitigations for Meltdown, Spectre, and L1 Terminal Fault (L1TF) in SUSE Linux Enterprise-based products. The purpose of this document is to help customers evaluate how best to performance test their deployments.

Author: Sheilagh Morlan, Manager Software Engineering, SUSE
Author: Bryan Stephenson, SUSE OpenStack Cloud Security Engineer, SUSE
Author: T.R. Bosworth, Senior Product Manager SUSE OpenStack Cloud, SUSE
Author: Jiri Kosina, Director, SUSE Labs Core, SUSE
Author: Vojtech Pavlik, VP SUSE Labs, SUSE
Author: Olaf Kirch, VP SUSE Linux Enterprise, SUSE
Publication Date: March 21, 2019

1 Background

To improve chipset performance, in the 1990s, Intel introduced a new architecture to its x386 chips. This architecture allowed for predictive calculations to speed up processes. A simple explanation is that this design included the ability for processes to read buffers and caches not directly related to their specific calculations. A study funded by the NSA in 1995 identified both the covert channel and transaction lookaside buffer vulnerabilities at the heart of Meltdown, Spectre and L1 Terminal Fault (L1TF). But at the time there were no easy exploits and the industry was more interested in performance than security. Twenty years later the rise of cloud computing, containers, multi-tenancy, and the drive for performance have created an environment where it is now possible to exploit these vulnerabilities.

There is always a balance between performance and security. Enhanced security means siloed computations, message and load encryption, secure threading, and secure database transactions all of which take a toll on performance. In a highly competitive industry where speed is market, security gets downgraded. When a security problem is identified and mitigated, performance takes a hit.

Since January 2018, chipset manufacturers produced microcode updates to the chips for each specific vulnerability. These microcode updates make the chips more secure at the expense of performance.

Note
Note: Chipset Vulnerabilities

The Meltdown, Spectre and L1TF vulnerabilities are chipset hardware problems. They will only be completely resolved when the chipset manufacturers develop a new architecture and burn new silicon that makes its way into enterprise and consumer hardware available to the market.

The challenge for operating system and software providers now is that machines that contain the affected chipsets (all of them) have patched vulnerabilities that now need to be accommodated through the software that runs on it. This holds specifically true for the operating systems, but to a lesser extent also for the applications that run on top of those operating systems.

Ever since Meltdown and Spectre vulnerabilities were disclosed in January of 2018, customers have been asking for guidance on the performance impact of the mitigating patches for these hardware vulnerabilities. The most burning questions are:

  • Do these patches impact performance?

  • If so, by how much?

The highly unsatisfactory answer is: it depends on the workload. Some types of workloads are hardly affected, while others are greatly affected.

The main determinants of whether a workload will be affected are:

  • The workload itself:

    • Workloads that execute more system calls are more highly affected

      • Heavy database operations

      • File management

      • Transfer status information

      • Attach/detach remote devices

      • Send/receive messages

      • Workloads that do numeric computations are less affected

  • The CPU vendor and family:

    • All modern CPUs were affected but not equally

    • Some CPUs are optimized for certain types of transactions/processes

  • The operating system in use:

    • Different CPUs will respond differently to different patches

      • Tuning operating system builds is an ongoing process

In our own testing, we have seen performance benchmarks that regressed just a little, and we have seen some that regressed by 15 percent or even more. Because different benchmarks exercise different areas of an operating system, it is no surprise that they show different degrees of performance regression. What may be more surprising is that most benchmarks showed a high variance in the relative impact across different CPU vendors and models, sometimes by a factor of 2 or more. This means a benchmark that would show a 5 percent impact on one platform would show a 10 percent impact or more on another.

Thus we believe that it would be misleading if we provided customers with a table stating that workload X will suffer a performance regression of N percent as a result of security fixes for Spectre, Meltdown and L1TF vulnerabilities. Given that microcode changes play a major part in the mitigation, the performance impact you experience will very much depend on what sort of hardware you have. And until there is a final set of microcode updates from all vendors, any performance measurements must be considered preliminary anyway.

To make a definitive assessment, you need to benchmark your workloads to find out how they are impacted. However, the following sections will explain more in detail what causes the performance hit to illustrate how different classes of applications will be impacted differently. They also discuss the work SUSE is doing to recover some of the lost performance.

2 Meltdown

This vulnerability, among others, can be exploited by a rogue user space process to read kernel memory locations. It affects Intel CPUs, some ARM licensees, and IBM POWER processors to some degree. The only way to mitigate it is by changing the operating system to help the CPU to forget its kernel address mapping whenever switching from kernel to user space – for example, when returning from a system call.

The patch set to help forget its kernel address mapping on the x86-64 architecture is called Page Table Isolation (PTI). It adds (a lot of) code to the system call entry and exit code plus a few other places to fully isolate the kernel page tables from user space access. This involves removing all address mapping information about kernel memory from the CPU, and performing a flush on the address translation cache (called a TLB flush). This adds a more or less constant cost to each system call, interrupt, etc. The operation itself is not hugely expensive, but it is not fast either. The workload itself may add additional latency for any TLB hits without PTI, so this is another example of the performance being workload specific.

Our patch set includes a slight optimization on Intel CPUs that relies on a somewhat recent CPU feature called Process Context ID. This ID can be used to speed up the page table isolation a bit. When returning from a system call, we still need to remove all mapping information about kernel memory – but we can avoid the other part of this operation, the TLB flush. In several of the benchmarks we have run, using the PCID feature would cut the performance impact by up to a half. This PTI optimization is enabled by default when we find that the CPU supports it.

This mitigation is not necessary for AMD CPUs as they will be automatically detected and disabled during the boot process. For more details, see the Technical Information Document at https://www.suse.com/support/kb/doc/?id=7022512.

3 Spectre

Spectre is not so much a single vulnerability, but several vulnerabilities rolled into one. There are many variants of this family of attacks. A complete list can be found in the SUSE blog article at https://www.suse.com/c/overview-of-meltdown-spectre/. In some form, Spectre affects all current CPUs on which SUSE Linux Enterprise products are supported.

Spectre 1 exploits rely on the ability to have the CPU (or a hypervisor) speculatively access memory through pointers that can be controlled by the attacker. To a large degree the mitigation employed at the kernel and hypervisor level involves finding code that accesses user controlled pointers in an exploitable way, and protecting these with speculation barriers that flush the branch prediction logic, which is a costly operation. A primary example is the extended Berkeley Packet Filtering facility, which suffered a significant hit in the first round of updates.

SUSE kernel developers are working on improving some of these changes to soften the impact where possible. For example, we expect eBPF performance to come back to almost the previous levels with the next round of kernel updates that are in preparation, by replacing the (hardware) barrier with a software based mitigation. SUSE defaults to retpoline mitigation on all CPUs except for SkyLake.

Spectre 2 exploits rely on the ability to actively confuse the branch target prediction inside the CPU. This is essentially a cache that is used to predict where indirect calls will end up jumping to. By poisoning this cache, an attacker can cause the CPU to speculatively execute code at an address controlled by an attacker, with arguments controlled by the attacker. This vulnerability cannot be addressed by finding bits of code that can be abused, like it has been done for Spectre 1, because the number of combinations is virtually limitless.

The mitigation currently present in our update kernels is based on three new CPU features (introduced by the recent microcode updates and the retpoline software-based mitigation):

  • Indirect Branch Restricted Speculation (IBRS):

    • Flushes the cache between privilege levels (user to kernel)

    • Disables branch prediction on the sibling CPU thread

    • Enabled on kernel entry; disabled on kernel exit

    • Can be enabled at boot time or runtime

  • Single Thread Indirect Branch Predictors STIBP:

    • Disables branch prediction on the sibling CPU thread

    • May adversely affect hyper-threading

    • November 2018 release allows this to run surgically

      • Only when processes request it

      • SECCOMP threads

  • Indirect Branch Prediction Barrier (IBPB):

    • Flushes cache for code running at same privilege level

  • “Return Trampoline” (Retpoline):

    • Potential alternative for IBRS

    • Emits different machine code sequence for indirect calls

    • Replacement of all indirect calls also results in slowdown

    • Slowdowns expected to be less than IBRS

    • Microcode updates still required to take advantage of IBPB mitigations

4 L1 Terminal Fault

Exploiting L1 Terminal Fault (L1TF) will bypass protective barriers and read privileged memory the attackers would not otherwise have access to. This includes data stored on physical machines and on virtual machines in a multi-tenant environment. It exploits modern CPUs’ ability to read memory inside the L1 cache. The data that can be exposed resides in:

  • Intel’s Software Guard Extensions (SGX) secure enclaves

  • System Management Mode (SMM) memory or the operating system’s kernel

  • Data from hypervisors/Virtual Machine Monitors (VMMs)

  • Data from other virtual machines running on the same hardware

Vulnerable processors are:

  • SGX-enabled Intel Core and Xeon Processors (Skylake and Kaby Lake)

Intel Atom family processors with SGX, AMD, and ARM chips do not seem to be affected.

Also available from Intel is the flush_l1d feature which hypervisors can use to flush the L1 data cache before executing potentially malicious code in the VM.

We have released additional L1TF mitigation in the form of the l1tf kernel boot parameter for use on both bare metal and virtual machines. Recommended mitigations include support for new CPU features and vary depending on environment. L1TF mitigation is controlled through the l1tf kernel boot parameter.

We have released Linux kernel updates that include a new sysfs variable that will show the state of available mitigations. These states are available for non-VMX systems, systems using KVM/VMX, and CPU microcode protection levels.

More information on the updates can be found in the SUSE blog article at https://www.suse.com/c/suse-addresses-the-l1-terminal-fault-issue/.

5 Resources

For more detailed information, refer to the following documents and articles:

6 Legal Notice

Copyright ©2006–2019 SUSE LLC and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled GNU Free Documentation License.

SUSE, the SUSE logo and YaST are registered trademarks of SUSE LLC in the United States and other countries. For SUSE trademarks, see http://www.suse.com/company/legal/. Linux is a registered trademark of Linus Torvalds. All other names or trademarks mentioned in this document may be trademarks or registered trademarks of their respective owners.

This article is part of a series of documents called "SUSE Best Practices". The individual documents in the series were contributed voluntarily by SUSE's employees and by third parties.

All information found in this document has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy.

Therefore, we need to specifically state that neither SUSE LLC, its affiliates, the authors, nor the translators may be held liable for possible errors or the consequences thereof. Below we draw your attention to the license under which the articles are published.

Print this page