Inoperative Xen Guest devices due to IRQ mapping failures

This document (000021158) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP4
Xen 4.16.0
Xen 4.17.0

Situation

In instances with a large number of guest devices, errors like the following might be seen:

ice 0000:60:00.1: xen map irq failed -28 for 32752 domain
ice 0000:60:00.1: ice_init_interrupt_scheme failed, error: -28 (-ENOSPC)
mlx5_core 0000:9b:00.0: xen map irq failed -28 for 32752 domain
mlx5_core 0000:9b:00.0: probe_one:2139:(pid 1751): mlx5_init_one failed with
    error code -28
mlx5_core: probe of 0000:9b:00.0 failed with error -28

Resolution

Increasing the number of IRQs reserved for the guest using the "extra_guest_irqs" parameter resolves this problem.

Edit /etc/default/grub and add/edit the Xen parameter as follows:

GRUB_CMDLINE_XEN_DEFAULT=”extra_guest_irqs=,32768"

Run "update-bootloader --refresh" to effect the change, then reboot the system.

Cause

Xen has not allocated enough IRQs for the large number of guest devices, the default is just 32 extra IRQs.

Additional Information

Xen has a limit on the amount of IRQs per guest it can handle, but an invalid value isn't properly reported as unsupported as of version 4.17. Internally the IRQs are limited by the use of a single-page bitmap (meaning a maximum of 32768 IRQs on architectures with 4 KB pages).

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.