Using multipath devices for OSDs

This document (7023110) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 5
Device Mapper Multipath (MPIO)

Situation

A ceph storage node uses an MPIO device as an object storage device (OSD). When the storage node reboots, some or all of the MPIO OSDs fail to start or activate.

The MPIO OSDs are marked down.

# /usr/bin/ceph --connect-timeout=5 osd tree
ID  CLASS WEIGHT  TYPE NAME            STATUS REWEIGHT PRI-AFF 
 -1       0.59921 root default                                 
<snip/>
-11       0.13138     host ses5node108                         
 12   hdd 0.04379         osd.12         down        0 1.00000 
 13   hdd 0.04379         osd.13           up  1.00000 1.00000 
 14   hdd 0.04379         osd.14           up  1.00000 1.00000

Running ceph-disk activate-all brings the OSDs up.

Resolution

Run ceph-disk suppress-activate <device path> for each non-DM device on each MPIO device on the affected OSD storage node. For example, if:

osdnode1:~ # multipath -ll
mpath-data10 (14945540000000000736c6531322d636c737472325f313047) dm-6 IET,VIRTUAL-DISK
size=10G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='queue-length 0' prio=1 status=active
  |- 3:0:0:2 sdg 8:96  active ready running
  `- 2:0:0:2 sdd 8:48  active ready running

/dev/dm-6 and /dev/mapper/mpath-data10 are device mapper devices (DM), while /dev/sdg and /dev/sdd are non-DM devices. In this example, do the following:

osdnode1:~ # ceph-disk suppress-activate /dev/sdg
osdnode1:~ # ceph-disk suppress-activate /dev/sdd

Cause

The non-DM devices for an MPIO device where not suppressed for ceph activation.

Additional Information

Using SAN drives for OSD drives is generally counter productive. Ceph clusters are great for work loads that don't require low latency, using commodity hardware. SAN with MPIO drives are usually reserved for the low latency requirements.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.