Boot drops to emergency shell due udev running out of memory

This document (000019906) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 SP5 on s390x or x86_64
SUSE Linux Enterprise Server 12 SP4 on s390x or x86_64

Situation

When system has lots of LUNs and each LUN has multiple paths, it can lead to the event that the boot process does not complete successfully. Instead, the boot process stops and asks to drop to emergency shell. Accessing emergency shell and just exiting it will cause the boot process to continue.

Resolution

Blacklist the device driver for HBA that has the huge amount of LUNs allocated. If for example on s390x the LUNs are behind an zFCP HBA, omit the loading of zfcp driver as outlined:

Method 1

Add modprobe.blacklist=zfcp in /etc/default/grub to the GRUB_CMDLINE_LINUX_DEFAULT arguments .

Method 2

Blacklist the zFCP driver from being included in the initrd

echo 'omit_drivers+=" zfcp "' > /etc/dracut.conf.d/50-zfcp.conf

Next, add an action to load the zfcp module after system has done initial booting. For example, add the following line in /etc/initd.d/boot.local file

modprobe zfcp

Finally, recreate the intird image with mkinitrd or dracut -f command, then refresh bootloader

update-bootloader --refresh

Cause

As part for disk detection during boot process for each path udev worker is spawned. The underlying problem is that the individual udev workers will receive an OOM during booting, as the calculation on how many udev workers should be spawned is very fragile.

Background: udev does not do any memory allocation, but rather allocates everything on the stack. So an OOM can not be captured as one would normally when 'malloc' returns NULL; rather the OOM will directly kill the process with no way out. This leads to events not being processed, and a hanging system waiting for these events.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.