16.15 Troubleshooting MPIO

This section describes some known issues and possible solutions for MPIO.

16.15.1 The System Exits to Emergency Shell at Boot When Multipath Is Enabled

During boot the system exits into the emergency shell with messages similar to the following:

[  OK  ] Listening on multipathd control socket.
         Starting Device-Mapper Multipath Device Controller...
[  OK  ] Listening on Device-mapper event daemon FIFOs.
         Starting Device-mapper event daemon...
         Expecting device dev-disk-by\x2duuid-34be48b2\x2dc21...32dd9.device...
         Expecting device dev-sda2.device...
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
         Expecting device dev-disk-by\x2duuid-1172afe0\x2d63c...5d0a7.device...
         Expecting device dev-disk-by\x2duuid-c4a3d1de\x2d4dc...ef77d.device...
[  OK  ] Started Create list of required static device nodes ...current kernel.
         Starting Create static device nodes in /dev...
[  OK  ] Started Collect Read-Ahead Data.
[  OK  ] Started Device-mapper event daemon.
[  OK  ] Started udev Coldplug all Devices.
         Starting udev Wait for Complete Device Initialization...
[  OK  ] Started Replay Read-Ahead Data.
         Starting Load Kernel Modules...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Started Create static devices
[   13.682489] floppy0: no floppy controllers found
[*     ] (1 of 4) A start job is running for dev-disk-by\x2du...(7s / 1min 30s)
[*     ] (1 of 4) A start job is running for dev-disk-by\x2du...(7s / 1min 30s)

...

Timed out waiting for device dev-disk-by\x2duuid-c4a...cfef77d.device.
[DEPEND] Dependency failed for /opt.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Postfix Mail Transport Agent.
Welcome to emergency shell
Give root password for maintenance
(or press Control-D to continue):

This issue is a logical consequence of the multipath integration in systemd and occurs if the root file system is not on multipath but multipath is enabled. In such a setup, multipath tries to set its paths for all devices that are not blacklisted. Since the device with the root file system is already mounted, it is inaccessible for multipath and causes it to fail.

Fix this issue by configuring multipath correctly by blacklisting the root device in /etc/multipath.conf:

  1. Run multipath -v2 in the emergency shell and identify the device for the root file system. It will result in an output similar to:

    multipath -v2
    Dec 18 10:10:03 | 3600508b1001030343841423043300400: ignoring map

    The string between | and : is the WWID needed for blacklisting.

  2. Open /etc/multipath.conf and add the following:

    blacklist {
      wwid "WWWID"
    }

    Replace WWWID with the ID you retrieved in the previous step. For more information see Section 16.8, Blacklisting Non-Multipath Devices.

  3. Exit the emergency shell and reboot the server by pressing Ctrl+D.

16.15.2 Enabling boot.multipath

Multipath must be loaded before LVM to ensure that multipath maps are built correctly. Loading multipath after LVM can result in incomplete device maps for a multipath device because LVM locks the device, and MPIO cannot create the maps properly.

If the system device is a local device that does not use MPIO and LVM, you can disable both boot.multipath and boot.lvm. After the server starts, you can manually start multipath before you start LVM, then run a pvscan command to recognize the LVM objects.

16.15.3 Troubleshooting MPIO Mapping for LVM Devices

Timing is important for starting the LVM process. If LVM starts before MPIO maps are done, LVM might use a fixed path for the device instead of its multipath. The device works, so you might not be aware that the device’s MPIO map is incomplete until that fixed path fails. You can help prevent the problem by enabling boot.multipath and following the instructions in Enabling boot.multipath.

To troubleshoot a mapping problem, you can use dmsetup to check that the expected number of paths are present for each multipath device. As the root user, enter the following at a command prompt:

sudo dmsetup ls --tree

In the following sample response, the first device has four paths. The second device is a local device with a single path. The third device has two paths. The distinction between active and passive paths is not reported through this tool.

sudo dmsetup ls --tree
  vg910-lv00 (253:23)
    └─ 360a980006465576657346d4b6c593362 (253:10)
      |- (65:96)
      |- (8:128)
      |- (8:240)
      └─ (8:16)
  vg00-lv08 (253:9)
    └─ (8:3)
  system_vg-data_lv (253:1)
    └─36006016088d014007e0d0d2213ecdf11 (253:0)
      ├─ (8:32)
      └─ (8:48)

An incorrect mapping typically returns too few paths and does not have a major number of 253. For example, the following shows what an incorrect mapping looks like for the third device:

  system_vg-data_lv (8:31)
     └─ (8:32)

16.15.4 PRIO Settings for Individual Devices Fail After Upgrading to Multipath 0.4.9 or Later

Multipath Tools from version 0.4.9 onward uses the prio setting in the defaults{} or devices{} section of the /etc/multipath.conf file. It silently ignores the keyword prio when it is specified for an individual multipath definition in the multipaths{) section.

Multipath Tools 0.4.8 allowed the prio setting in the individual multipath definition in the multipaths{) section to override the prio settings in the defaults{} or devices{} section.

16.15.5 PRIO Settings with Arguments Fail After Upgrading to multipath-tools-0.4.9 or Later

When you upgrade from multipath-tools-0.4.8 to multipath-tools-0.4.9, the prio settings in the /etc/multipath.conf file are broken for prioritizers that require an argument. In multipath-tools-0.4.9, the prio keyword is used to specify the prioritizer, and the prio_args keyword is used to specify the argument for prioritizers that require an argument. Previously, both the prioritizer and its argument were specified on the same prio line.

For example, in multipath-tools-0.4.8, the following line was used to specify a prioritizer and its arguments on the same line.

prio "weightedpath hbtl [1,3]:.:.+:.+ 260 [0,2]:.:.+:.+ 20"

After upgrading to multipath-tools-0.4.9 or later, the command causes an error. The message is similar to the following:

<Month day hh:mm:ss> | Prioritizer 'weightedpath hbtl [1,3]:.:.+:.+ 260
[0,2]:.:.+:.+ 20' not found in /lib64/multipath

To resolve this problem, use a text editor to modify the prio line in the /etc/multipath.conf file. Create two lines with the prioritizer specified on the prio line, and the prioritizer argument specified on the prio_args line below it:

prio "weightedpath"
prio_args "hbtl [1,3]:.:.+:.+ 260 [0,2]:.:.+:.+ 20"

Restart the multipathd daemon for the changes to become active by running sudo systemctl restart multipathd.

16.15.6 Technical Information Documents

For information about troubleshooting multipath I/O issues on SUSE Linux Enterprise Server, see the following Technical Information Documents (TIDs) in the SUSE Knowledgebase: