Running replace.osd on Storage Cluster Fails to Repalce Destroyed Disk

This document (7023996) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 5 (SES5)
SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

Situation

After following the documentation steps to replace an OSD drive in SES5 using ceph replace.osd, the disk was not replaced as expected.


Resolution

Extra steps need to be taken to replace a destroyed drive. In the example below, the follow values will be used.

ses5node111 - the cluster admin node
ses5node114 - the storage node on which the affected OSD resides.
/dev/sda - the physcial block device for osd.15
/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_ses5node114_osdq - the symbolic link to /dev/sda
osd.15 - the osd ID assoicated with /dev/sda

The following were used to simulate a destroyed OSD disk

ses5node114:~ # sfdisk -f --delete /dev/sda
ses5node114:~ # reboot

The /dev/sda device was removed and a new /dev/sda device added

Replace the destroyed disk
1. ses5node111:~ # salt-run disengage.safety
2. ses5node111:~ # salt-run replace.osd 15
3. ses5node114:~ # rm /etc/ceph/destroyedOSD.yml
4. ses5node111:~ # vi /srv/pillar/ceph/proposals/profile-default/stack/default/ceph/minions/ses5node114.yml
Add "replace: True" to the affected disk in the *.yml file. For example,
Change:
      /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_ses5node114_osdq:
        format: bluestore

to:

      /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_ses5node114_osdq:
        format: bluestore
        replace: True

5. ses5node111:~ # salt-run state.orch ceph.stage.2
6. ses5node111:~ # salt-run state.orch ceph.stage.3

Cause

The ceph replace.osd option expects the /dev/disk/by-path reference to the physical device to stay the same. When the disk partition was destroyed, the /dev/disk/by-path link changed.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7023996
  • Creation Date: 09-Jul-2019
  • Modified Date:03-Mar-2020
    • SUSE Enterprise Storage
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center