Interrupts not distributed as specified in smp_affinity

This document (7018478) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2)
SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)
SUSE Linux Enterprise Server 12

Situation

The bitmask stored in /proc/irq/<IRQ#>/smp_affinity defines on which CPU(s) a given interrupt is allowed to run.
By manipulating the bitmask it is possible to change the set of permitted CPUs.
This is briefly documented in /usr/src/linux/Documentation/IRQ-affinity.txt.
For more details see the "Additional information" section.

However the outcome is not always as expected.
As an example it has been observed that when defining a group of CPUs, only the least bit CPU increments interrupts. In other words if for example CPUs 4-7 are masked (hex f0), then only CPU 4 is being used.

Resolution

The possibility to distribute IRQs across multiple CPUs by masking a group depends on the number of CPUs and which APIC model the kernel chooses at boot.

To establish whether the kernel has switched APIC model or it is operating in flat mode, grep in the kernel log.
Examples :
# dmesg | grep -i apic 
can (amongst other) show information like
[...] x2apic enabled
[...] Switched APIC routing to cluster x2apic.
or
[...] Switched APIC routing to physical flat.

The catch is that if nothing is mentioned about switching APIC routing, then flat mode is used.

By default QEMU/KVM enables x2apic for virtual machines.

Make sure to have a maximum of 8 CPUs available and that the "flat" mode is used as per above.
For systems enabling x2apic, add the "nox2apic" option to the kernel command line in the bootloader configuration.

It seems not possible to make a system, where the kernel switches to "physical flat", run in "flat" mode.

Cause

It is only possible to distribute interrupts across multiple CPUs on systems that have up to 8 CPUs.
Furthermore it depends on the APIC (Advanced Programmable Interrupt Controller) model chosen by the Linux kernel during boot.
This model depends on the actual hardware APIC present in the system.
Examples of common APIC models are flat, physical flat, physical x2apic, cluster x2apic.
By default Linux is going to use "flat" APIC model, which is the only one enabling interrupt distribution across multiple CPUs.
During early probing in the boot process the kernel decides if "flat" is the correct APIC model to be used with the system. If not, it switches the model.

Additional Information

The bitmask is stored in hex with each digit representing a group of four CPUs. The rightmost group is the least significant, so the first bit represents CPU 0.
Looking at the rightmost hexadecimal digit, the following "table" illustrates the representations of the bitmasks.
CPU      Binary       Hex
CPU 0    0001         1
CPU 1    0010         2
CPU 2    0100         4
CPU 3    1000         8

By combining these bit patterns (simply adding the hex values) multiple CPUs can be defined.
For example a hex value of 02 translates into CPU 1, 20 equals CPU 5 and 0f means CPU 0-3.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7018478
  • Creation Date: 11-Jan-2017
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center