16.15 Troubleshooting MPIO

This section describes some known issues and possible solutions for MPIO.

16.15.1 The System Exits to Emergency Shell at Boot When Multipath Is Enabled

During boot the system exits into the emergency shell with messages similar to the following:

[  OK  ] Listening on multipathd control socket.
         Starting Device-Mapper Multipath Device Controller...
[  OK  ] Listening on Device-mapper event daemon FIFOs.
         Starting Device-mapper event daemon...
         Expecting device dev-disk-by\x2duuid-34be48b2\x2dc21...32dd9.device...
         Expecting device dev-sda2.device...
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
         Expecting device dev-disk-by\x2duuid-1172afe0\x2d63c...5d0a7.device...
         Expecting device dev-disk-by\x2duuid-c4a3d1de\x2d4dc...ef77d.device...
[  OK  ] Started Create list of required static device nodes ...current kernel.
         Starting Create static device nodes in /dev...
[  OK  ] Started Collect Read-Ahead Data.
[  OK  ] Started Device-mapper event daemon.
[  OK  ] Started udev Coldplug all Devices.
         Starting udev Wait for Complete Device Initialization...
[  OK  ] Started Replay Read-Ahead Data.
         Starting Load Kernel Modules...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Started Create static devices
[   13.682489] floppy0: no floppy controllers found
[*     ] (1 of 4) A start job is running for dev-disk-by\x2du...(7s / 1min 30s)
[*     ] (1 of 4) A start job is running for dev-disk-by\x2du...(7s / 1min 30s)

...

Timed out waiting for device dev-disk-by\x2duuid-c4a...cfef77d.device.
[DEPEND] Dependency failed for /opt.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Postfix Mail Transport Agent.
Welcome to emergency shell
Give root password for maintenance
(or press Control-D to continue):

This issue occurs in the following situations:

  • A non-multipath root file system is not blacklisted when multipath is enabled. See Emergency Shell: Blacklist File Systems

  • On a system with a multipath root file system, multipath has been enabled/disabled without rebuilding the initrd. See Emergency Shell: Rebuild the initrd.

  • Network-storage drivers are missing in the initrd on a system with network-attached storage and multipath enabled.

Emergency Shell: Blacklist File Systems

This fix is required if the root file system is not on multipath but multipath is enabled. In such a setup, multipath tries to set its paths for all devices that are not blacklisted. Since the device with the root file system is already mounted, it is inaccessible for multipath and causes it to fail. Fix this issue by configuring multipath correctly by blacklisting the root device in /etc/multipath.conf:

  1. Run multipath -v2 in the emergency shell and identify the device for the root file system. It will result in an output similar to:

    multipath -v2
    Dec 18 10:10:03 | 3600508b1001030343841423043300400: ignoring map

    The string between | and : is the WWID needed for blacklisting.

  2. Open /etc/multipath.conf and add the following:

    blacklist {
      wwid "WWWID"
    }

    Replace WWWID with the ID you retrieved in the previous step. For more information see Section 16.8, Blacklisting Non-Multipath Devices.

  3. Exit the emergency shell and reboot the server by pressing Ctrl+D.

Emergency Shell: Rebuild the initrd

This fix is required if the multipath status (enabled or disabled) differs between initrd and system. To fix this, rebuild the initrd:

  1. If multipath has been enabled in the system, rebuild the initrd with multipath support with this command:

    dracut --force --add multipath

    In case Multipath has been disabled in the system, rebuild the initrd with Multipath support with this command:

    dracut --force -o multipath
  2. Exit the emergency shell and reboot the server by pressing Ctrl+D.

Emergency Shell: Rebuild the initrd

This fix is required if the initrd does not contain drivers to access network attached storage. This may, for example, be the case when the system was installed without multipath or when the respective hardware was added or replaced.

  1. Add the required driver(s) to the variable force_drivers in the file /etc/dracut.conf.d/01-dist.conf. For example, if your system contains a RAID controller accessed by the hpsa driver and multipathed devices connected to a QLogic controller accessed by the driver qla23xx, this entry would look like:

    force_drivers+="hpsa qla23xx"
  2. Rebuild the initrd using the following command:

    dracut -f --add-multipath
  3. To prevent the system from booting into emergency mode if attaching the network storage fails, it is recommended to add the mount option _netdev to the respective entries in /etc/fstab.

  4. Exit the emergency shell and reboot the server by pressing Ctrl+D.

16.15.2 PRIO Settings for Individual Devices Fail After Upgrading to Multipath 0.4.9 or Later

Multipath Tools from version 0.4.9 onward uses the prio setting in the defaults{} or devices{} section of the /etc/multipath.conf file. It silently ignores the keyword prio when it is specified for an individual multipath definition in the multipaths{) section.

Multipath Tools 0.4.8 allowed the prio setting in the individual multipath definition in the multipaths{) section to override the prio settings in the defaults{} or devices{} section.

16.15.3 PRIO Settings with Arguments Fail After Upgrading to multipath-tools-0.4.9 or Later

When you upgrade from multipath-tools-0.4.8 to multipath-tools-0.4.9, the prio settings in the /etc/multipath.conf file are broken for prioritizers that require an argument. In multipath-tools-0.4.9, the prio keyword is used to specify the prioritizer, and the prio_args keyword is used to specify the argument for prioritizers that require an argument. Previously, both the prioritizer and its argument were specified on the same prio line.

For example, in multipath-tools-0.4.8, the following line was used to specify a prioritizer and its arguments on the same line.

prio "weightedpath hbtl [1,3]:.:.+:.+ 260 [0,2]:.:.+:.+ 20"

After upgrading to multipath-tools-0.4.9 or later, the command causes an error. The message is similar to the following:

<Month day hh:mm:ss> | Prioritizer 'weightedpath hbtl [1,3]:.:.+:.+ 260
[0,2]:.:.+:.+ 20' not found in /lib64/multipath

To resolve this problem, use a text editor to modify the prio line in the /etc/multipath.conf file. Create two lines with the prioritizer specified on the prio line, and the prioritizer argument specified on the prio_args line below it:

prio "weightedpath"
prio_args "hbtl [1,3]:.:.+:.+ 260 [0,2]:.:.+:.+ 20"

Restart the multipathd daemon for the changes to become active by running sudo systemctl restart multipathd.

16.15.4 Technical Information Documents

For information about troubleshooting multipath I/O issues on SUSE Linux Enterprise Server, see the following Technical Information Documents (TIDs) in the SUSE Knowledgebase: