Using multipath devices for OSDs

This document (7023110) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 5
Device Mapper Multipath (MPIO)

Situation

A ceph storage node uses an MPIO device as an object storage device (OSD). When the storage node reboots, some or all of the MPIO OSDs fail to start or activate.

The MPIO OSDs are marked down.
# /usr/bin/ceph --connect-timeout=5 osd tree
ID  CLASS WEIGHT  TYPE NAME            STATUS REWEIGHT PRI-AFF
 -1       0.59921 root default                                
<snip/>
-11       0.13138     host ses5node108                        
 12   hdd 0.04379         osd.12         down        0 1.00000
 13   hdd 0.04379         osd.13           up  1.00000 1.00000
 14   hdd 0.04379         osd.14           up  1.00000 1.00000

Running ceph-disk activate-all brings the OSDs up.

Resolution

Run ceph-disk suppress-activate <device path> for each non-DM device on each MPIO device on the affected OSD storage node. For example, if:
osdnode1:~ # multipath -ll
mpath-data10 (14945540000000000736c6531322d636c737472325f313047) dm-6 IET,VIRTUAL-DISK
size=10G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='queue-length 0' prio=1 status=active
  |- 3:0:0:2 sdg 8:96  active ready running
  `- 2:0:0:2 sdd 8:48  active ready running

/dev/dm-6 and /dev/mapper/mpath-data10 are device mapper devices (DM), while /dev/sdg and /dev/sdd are non-DM devices. In this example, do the following:
osdnode1:~ # ceph-disk suppress-activate /dev/sdg
osdnode1:~ # ceph-disk suppress-activate /dev/sdd

Cause

The non-DM devices for an MPIO device where not suppressed for ceph activation.

Additional Information

Using SAN drives for OSD drives is generally counter productive. Ceph clusters are great for work loads that don't require low latency, using commodity hardware. SAN with MPIO drives are usually reserved for the low latency requirements.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7023110
  • Creation Date: 18-Jun-2018
  • Modified Date:03-Mar-2020
    • SUSE Enterprise Storage
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center