I/O to LUNs hang / stall under high load when using xen-blkfront

This document (7018590) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 4 (SLES 11 SP4)

SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4)

Situation

Under high I/O load, for example when running database stress tests on the Virtual Machines, I/O hangs / stalls are seen.

Resolution

Increase the default "gnttab_max_frames" of "32" to a higher value by starting the Hypervisor (Dom0) with the kernel parameter "gnttab_max_frames=xxx".

Cause

The default setting can be insufficient for high I/O environments.

Additional Information

General recommendation for determining the proper value for "gnttab_max_frames" is to multiply by 2 the number of attached disks. Alternatively the following can be used to determine the appropriate value:

gnttab_max_frames=(6*#NICs + 8*#discs) * 1.5

To adjust the parameter edit the "/boot/grub/menu.lst" file of the Hypervisor (Dom0) and look for a line similar to the below example:

kernel /xen.gz

If the desired value is determined to be 256, adjust the line as follows and save the file:

kernel /xen.gz gnttab_max_frames=256

Currently if the issue is encountered, it will be logged to the system log with:

xen/grant-table: max_grant_frames reached ...

To verify the current number of used grants:

# xl debug-keys g; xl dmesg

(XEN) gnttab_usage_print_all [ key 'g' pressed
(XEN)       -------- active --------       -------- shared --------
(XEN) [ref] localdom mfn      pin          localdom gmfn     flags
(XEN) grant-table for remote d0 (v1)
(XEN)   1 frames (64 max), 5 maptrack frames (1024 max)
(XEN) no active grant table entries
(XEN)       -------- active --------       -------- shared --------
(XEN) [ref] localdom mfn      pin          localdom gmfn     flags
(XEN) grant-table for remote d1 (v1)
(XEN)   3 frames (32 max), 0 maptrack frames (1024 max)
(XEN) [0x000]      0 0x65d24 0x00000002          0 0x0fefff 0x19
(XEN) [0x001]      0 0x65d27 0x00000002          0 0x0feffc 0x19
(XEN) [0x008]      0 0x53632 0x00000002          0 0x036c32 0x19
(XEN) [0x300]      0 0x06b74 0x00000001          0 0x03b774 0x19
(XEN) [0x301]      0 0x06fef 0x00000001          0 0x03b3ef 0x19
(XEN) [0x500]      0 0x53254 0x00000001          0 0x037054 0x19
(XEN) [0x501]      0 0x53255 0x00000001          0 0x037055 0x19
(XEN) gnttab_usage_print_all ] done

To change the value for guests add "max_grant_frames=xx" to their configuration file or add the entry to "/etc/xen/xl.conf" in order to set the default for guests without having "max_grant_frames" in their configuration.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.