Interrupts not distributed as specified in smp_affinity
This document (7018478) is provided subject to the disclaimer at the end of this document.
Environment
SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2)
SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)
SUSE Linux Enterprise Server 12
Situation
The bitmask stored in /proc/irq/<IRQ#>/smp_affinity defines on which CPU(s) a given interrupt is allowed to run.
By manipulating the bitmask it is possible to change the set of permitted CPUs.
This is briefly documented in /usr/src/linux/Documentation/IRQ-affinity.txt.
For more details see the "Additional information" section.
However the outcome is not always as expected.
As an example it has been observed that when defining a group of CPUs, only the least bit CPU increments interrupts. In other words if for example CPUs 4-7 are masked (hex f0), then only CPU 4 is being used.
By manipulating the bitmask it is possible to change the set of permitted CPUs.
This is briefly documented in /usr/src/linux/Documentation/IRQ-affinity.txt.
For more details see the "Additional information" section.
However the outcome is not always as expected.
As an example it has been observed that when defining a group of CPUs, only the least bit CPU increments interrupts. In other words if for example CPUs 4-7 are masked (hex f0), then only CPU 4 is being used.
Resolution
The possibility to distribute IRQs across multiple CPUs by masking a group depends on the number of CPUs and which APIC model the kernel chooses at boot.
To establish whether the kernel has switched APIC model or it is operating in flat mode, grep in the kernel log.
Examples :
[...] x2apic enabled
[...] Switched APIC routing to cluster x2apic.
or
[...] Switched APIC routing to physical flat.
The catch is that if nothing is mentioned about switching APIC routing, then flat mode is used.
By default QEMU/KVM enables x2apic for virtual machines.
Make sure to have a maximum of 8 CPUs available and that the "flat" mode is used as per above.
For systems enabling x2apic, add the "nox2apic" option to the kernel command line in the bootloader configuration.
It seems not possible to make a system, where the kernel switches to "physical flat", run in "flat" mode.
To establish whether the kernel has switched APIC model or it is operating in flat mode, grep in the kernel log.
Examples :
# dmesg | grep -i apiccan (amongst other) show information like
[...] x2apic enabled
[...] Switched APIC routing to cluster x2apic.
or
[...] Switched APIC routing to physical flat.
The catch is that if nothing is mentioned about switching APIC routing, then flat mode is used.
By default QEMU/KVM enables x2apic for virtual machines.
Make sure to have a maximum of 8 CPUs available and that the "flat" mode is used as per above.
For systems enabling x2apic, add the "nox2apic" option to the kernel command line in the bootloader configuration.
It seems not possible to make a system, where the kernel switches to "physical flat", run in "flat" mode.
Cause
It is only possible to distribute interrupts across multiple CPUs on systems that have up to 8 CPUs.
Furthermore it depends on the APIC (Advanced Programmable Interrupt Controller) model chosen by the Linux kernel during boot.
This model depends on the actual hardware APIC present in the system.
Examples of common APIC models are flat, physical flat, physical x2apic, cluster x2apic.
By default Linux is going to use "flat" APIC model, which is the only one enabling interrupt distribution across multiple CPUs.
During early probing in the boot process the kernel decides if "flat" is the correct APIC model to be used with the system. If not, it switches the model.
Furthermore it depends on the APIC (Advanced Programmable Interrupt Controller) model chosen by the Linux kernel during boot.
This model depends on the actual hardware APIC present in the system.
Examples of common APIC models are flat, physical flat, physical x2apic, cluster x2apic.
By default Linux is going to use "flat" APIC model, which is the only one enabling interrupt distribution across multiple CPUs.
During early probing in the boot process the kernel decides if "flat" is the correct APIC model to be used with the system. If not, it switches the model.
Additional Information
The bitmask is stored in hex with each digit representing a group of four CPUs. The rightmost group is the least significant, so the first bit represents CPU 0.
Looking at the rightmost hexadecimal digit, the following "table" illustrates the representations of the bitmasks.
CPU Binary Hex
CPU 0 0001 1
CPU 1 0010 2
CPU 2 0100 4
CPU 3 1000 8
By combining these bit patterns (simply adding the hex values) multiple CPUs can be defined.
For example a hex value of 02 translates into CPU 1, 20 equals CPU 5 and 0f means CPU 0-3.
Looking at the rightmost hexadecimal digit, the following "table" illustrates the representations of the bitmasks.
CPU Binary Hex
CPU 0 0001 1
CPU 1 0010 2
CPU 2 0100 4
CPU 3 1000 8
By combining these bit patterns (simply adding the hex values) multiple CPUs can be defined.
For example a hex value of 02 translates into CPU 1, 20 equals CPU 5 and 0f means CPU 0-3.
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:7018478
- Creation Date: 11-Jan-2017
- Modified Date:03-Mar-2020
-
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com
