Security Vulnerability: "L1 Terminal Fault" (L1TF) – Hypervisor Information (CVE-2018-3620, CVE-2018-3646, XSA-273).

This document (7023078) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11

Situation

As described in TID 7023077, CVE-2018-3620, CVE-2018-3646, and XSA-273, Terminal Faults can be used to populate the L1 data cache through the speculative execution capability of Intel CPUs.

As this data cache is shared between sibling hyper-threads, malicious processes may be able to read this data through side-channel attacks.

While this vulnerability is serious on bare-metal machines, Hypervisor environments (including Xen and KVM) which host multiple virtual machines have an increasingly serious risk of leaking sensitive data from the host and/or across guests which run on sibling hyper-threads.

Resolution

Full mitigation for this issue requires a combination of hardware and software changes. Depending on the guest type, software changes may be required at both the Hypervisor and guest level.

Updated Intel microcode (provided through your hardware / BIOS vendor or by SUSE) introduces a new feature called “flush_l1d”. Hypervisors and bare-metal kernels use this feature to flush the L1 data cache during operations which may be susceptible to data leakage (e.g. when switching between VMs in Hypervisor environments).

Software mitigations exist for the Linux Kernel and for Hypervisors. These mitigations include support for new CPU features, passing these features to guests, and support for enabling/disabling/tuning the mitigations. Recommended mitigations vary depending on the environment.

For the Linux kernel (on both bare metal and virtual machines) L1TF mitigation is controlled through the “l1tf” kernel boot parameter. For complete information on this parameter, see TID 7023077.

KVM

For KVM host environments, mitigation can be achieved through L1D cache flushes, and/or disabling Extended Page Tables (EPT) and Simultaneous MultiThreading (SMT).

The L1D cache flush behavior is controlled through the "kvm-intel.vmentry_l1d_flush" kernel command line option:

kvm-intel.vmentry_l1d_flush=always
The L1D cache is flushed on every VMENTER.

kvm-intel.vmentry_l1d_flush=cond
The L1D cache is flushed on VMENTER only when there can be leak of host memory between VMEXIT and VMENTER. This could still leak some host data, like address space layout.

kvm-intel.vmentry_l1d_flush=never
Disables the L1D cache flush mitigation.

The default setting here is "cond".
The l1tf "full" setting overrides the settings of this configuration variable.

L1TF can be used to bypass Extended Page Tables (EPT). To mitigate this risk, it is possible to disable EPT and use shadow pages instead. This mitigation is available through the "kvm-intel.ept" option:

kvm-intel.ept=0
The Extended Page tables support is switched off.

As shadow pages are much less performant than EPT, SUSE recommends leaving EPT enabled, and use L1D cache flush and SMT tuning for full mitigation.

To eliminate the risk of untrusted processes or guests exploiting this vulnerability on a sibling hyper-thread, Simultaneous MultiThreading (SMT) can be disabled completely.

SMT can be controlled through kernel boot command line parameters, or on-the-fly through sysfs:

On the kernel boot command line:

nosmt
SMT is disabled, but can be later re-enabled in the system.

nosmt=force
SMT is disabled, and can not be re-enabled in the system.

If this option is not passed, SMT is enabled. Any SMT options used with the "l1tf" kernel parameter option overrides this “nosmt” option.

SMT can also be controlled through sysfs:

/sys/devices/system/cpu/smt/control

This file allows to read the current control state and allows to disable or (re)enable SMT.

Possible states are:
on
SMT is supported and enabled.

off
SMT is supported, but disabled. Only primary SMT threads can be onlined.

forceoff
SMT is supported, but disabled. Further control is not possible.

notsupported
SMT is not supported.

Potential values that can be written into this file:
on
off
forceoff

/sys/devices/system/cpu/smt/active

This file contains the state of SMT, if it is enabled and active, where active means that multiple threads run on 1 core.

Xen

For Xen hypervisor environments, mitigation is enabled by default and varies based on guest type. Manual adjustment of the “smt=” parameter is recommended, but the remaining parameters are best left at default values.A description of all relevant parameters are provided in the event any changes are necessary.

PV guests achieve mitigation at the Xen Hypervisor level. If a PV guest attempts to write an L1TF-vulnerable PTE, the hypervisor will force shadow mode and prevent the vulnerability. PV guests which fail to switch to shadow mode (e.g. due to a memory shortage at the hypervisor level) are intentionally crashed.

pv-l1tf=[ <bool>, dom0=<bool>, domu=<bool> ]

By default, pv-l1tf is enabled for DomU environments and, for stability and performance reasons, disabled for Dom0.

HVM guests achieve mitigation through a combination of L1D flushes, and disabling SMT.

spec-ctrl=l1d-flush=<bool>

This parameter determines whether or not the Xen hypervisor performs L1D flushes on VMEntry. Regardless of this setting, this feature is virtualized and passed to HVM guests for in-guest mitigation.

smt=<bool>

This parameter can be used to enable/disable SMT from the hypervisor. Xen environments hosting any untrusted HVM guests, or guests not under the full control of the host admin, should either disable SMT (through BIOS or smt=<bool> means), or ensure HVM guests use shadow mode (hap=0) in order to fully mitigate L1TF. It is also possible to reduce the risk of L1TF through the use of CPU pinning, custom CPU pools and/or soft-offlining of some hyper-threads. These approaches are beyond the scope of this TID, but are documented in the standard Xen documentation.

WARNING – The combination of Meltdown mitigation (KPTI) and shadow mode on hardware which supports PCID can result in a severe performance degradation.

NOTE – Efforts are ongoing to implement scheduling improvements that allow hyper-thread siblings to be restricted to threads from a single guest. This will reduce the exposure of L1TF, and the requirement to disable SMT in many environments.

Cause

CVE-2018-3620
CVE-2018-3646
XSA-273
p { margin-bottom: 0.1in; direction: ltr; color: rgb(0, 0, 10); line-height: 120%; text-align: left; }p.western { font-family: "Calibri", serif; font-size: 11pt; }p.cjk { font-family: "Calibri"; font-size: 11pt; }p.ctl { font-family: "DejaVu Sans"; font-size: 11pt; }a:link { color: rgb(0, 0, 255); }

Additional Information

Validating mitigations

Bare Metal and KVM

The state of the mitigation on bare-metal and KVM hosts can be seen through:

/sys/devices/system/cpu/vulnerabilities/l1tf

If this file is not present, the kernel does not support this mitigation.

Potential values here are:

Not affected
The processor is not affected by this problem.

Mitigation: PTE Inversion
In-kernel protection is active.

If KVM / VMX is enabled in the kernel, the following strings can appear appended to the strings above :

VMX: SMT vulnerable
SMT is enabled and vulnerable

VMX: SMT disabled
SMT is disabled

Also the state of the microcode protection is shown:

L1D vulnerable
L1D flushing is disabled

L1D conditional cache flushes
L1D flush is conditionally enabled, flushing conditionally after exiting VMs

L1D cache flushes
L1D flush is unconditionally enabled, flushing unconditionally before entering VMs

L1D EPT disabled
L1D flush is disabled, as Extended Page Tables are either not present or disabled

Xen environments

In Xen environments, the state of mitigation at the hypervisor level can be seen using:

# xl dmesg | grep -A7 Speculative

This output shows the state of L1TF in three separate lines.

On the “Xen settings” line, Hypervisor support of L1D_FLUSH is available if listed in the “Other:” section.The hardware’s vulnerability to L1TF is described on an “L1TF” line. Finally, the state of L1TF for PV guests is shown on the “PV L1TF shadowing” line.

For example, the following output shows L1D_FLUSH support enabled at the hypervisor and DomU level, but disabled on Dom0:

# xl dmesg | grep -A7 Speculative
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk JMP, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB L1D_FLUSH
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 46, Safe address 300000000000
(XEN)   Support for VMs: PV: MSR_SPEC_CTRL RSB EAGER_FPU, HVM: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU enabled

KVM and Xen guest level mitigation matches that of the Linux kernel, and can be viewed through the sysfs interface (/sys/devices/system/cpu/vulnerabilities/l1tf).p { margin-bottom: 0.1in; direction: ltr; color: rgb(0, 0, 10); line-height: 120%; text-align: left; }p.western { font-family: "Calibri", serif; font-size: 11pt; }p.cjk { font-family: "Calibri"; font-size: 11pt; }p.ctl { font-family: "DejaVu Sans"; font-size: 11pt; }a:link { color: rgb(0, 0, 255); }

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.