multipath system unable to boot after installing dracut-037-98.2.x86_64

SUSE Linux Enterprise Server 12 Service Pack 1 (SLES 12 SP1)
SUSE Linux Enterprise Server for SAP Applications

Applying updates to the current patch level of mirrored repositories without the required /etc/dracut.conf.d/10-mp.conf can lead to a race condition in the boot process.

After updating to the following versions:
 
dracut-037-98.2.x86_64
kernel-default-3.12.74-60.64.40.1.x86_64
kpartx-0.5.0-62.1.x86_64
multipath-tools-0.5.0-62.1.x86_64

a boot failure will result in being dropped to the emergency shell.

##Error from boot log

May 26 12:12:38 <hostname> kernel: device-mapper: multipath service-time: version
May 26 12:12:38 <hostname> multipathd[1191]: 3600c0ff000275a1178741d5901000000: l
May 26 12:12:38 <hostname> multipathd[1191]: 3600c0ff000275a1178741d5901000000: e
May 26 12:12:38 <hostname> kernel: bio: create slab <bio-1> at 1
May 26 12:12:38 <hostname> systemd-fsck[1200]: /sbin/fsck.xfs: XFS file system.
May 26 12:12:38 <hostname> systemd[1]: Started File System Check on /dev/disk/by-
May 26 12:12:38 <hostname> systemd[1]: Mounting /sysroot...
May 26 12:12:38 <hostname> kernel: bio: create slab <bio-2> at 2
May 26 12:12:38 <hostname> kernel: SGI XFS with ACLs, security attributes, realti
May 26 12:12:38 <hostname> mount[1220]: mount: /dev/sdc3 is already mounted or /s
May 26 12:12:38 <hostname> systemd[1]: [1;39msysroot.mount mount process exited, code=e
May 26 12:12:38 <hostname> systemd[1]: [1;31mFailed to mount /sysroot.
May 26 12:12:38 <hostname> systemd[1]: [1;31mDependency failed for Initrd Root File Sys
[7mlines 2969-2991
[27m
[KMay 26 12:12:38 <hostname> systemd[1]: 
[1;31mDependency failed for Reload Configuration
May 26 12:12:38 <hostname> systemd[1]: Triggering OnFailure= dependencies of init
May 26 12:12:38 <hostname> systemd[1]: Triggering OnFailure= dependencies of init
May 26 12:12:38 <hostname> systemd[1]: 
[1;39mUnit sysroot.mount entered failed state.
May 26 12:12:38 <hostname> systemd[1]: 

Ensure the initrd is built with the required dm-multipath modules and the rd.driver.pre=dm_multipath is present.
 
The file /etc/dracut.conf.d/10-mp.conf should exist _before_ the updates are applied to avoid the issue.
This is mentioned in the Section 15.11.2 "Enabling Multipath I/O for an Existing Root Device" in the Storage Admin Guide.

If updates were applied without the required file then you can do the following in the Emergency Shell:

       'dracut --add multipath'

in order to create a multipath-enabled initrd and try booting again.

Or  backrev the versions to:

kpartx-0.5.0-55.1.x86_64.rpm
multipath-tools-0.5.0-55.1.x86_64.rpm
dracut-037-84.1.x86_64.rpm

This will return to normal booting condition where you can do the following to ensure proper boot after updates:

1. Create the required /etc/dracut.conf.d/10-mp.conf file by entering:
     
     echo 'force_drivers+=" dm-multipath "' >/etc/dracut.conf.d/10-mp.conf
(the empty space before and after " dm-multipath " is intended)

2. Apply updates with 'zypper patch'

3. Verify that you see rd.driver.pre=dm_multipath in the "Stored Kernel Commandline" of the mkinitrd log.
The mkinitrd will be the last in the process of patching and will be on the screen when zypper finishes.
Make sure that the "rd.driver.pre=dm_multipath" is present in the initrd build before attempting to reboot.

Example snip of a good initrd build:

I: Stored kernel commandline:
I: rd.driver.pre=dm_multipath       
I:  rd.lvm.lv=vg_alfsd2p1/lv_swap
 rd.lvm.lv=rhel/swap
I: rd.driver.pre=scsi_dh_alua rd.driver.pre=scsi_dh_emc rd.driver.pre=scsi_dh_rdac

The problem is a miss-configuration of not having the required /etc/dracut.conf.d/10-mp.conf file containing
    force_drivers+=" dm-multipath "

Without it the new initrd created by the kernel update process will not have the required pre-checkers and dm-multipath.