Removing an OSD using ceph orch requires some additional cleanup

This document (000020642) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7

Situation

When removing an OSD using "ceph orch osd rm <id>" as per the Online Documentation , the removed OSD is still reported by "ceph orch ps" and on the node that hosted the OSD systemctl still shows the service in a failed state.

Resolution

It is needed to clean up the removed OSD by executing:
 
ceph orch daemon rm osd.<id>
 
If needed, also wipe the disk:
 
cephadm ceph-volume lvm zap --destroy --osd-id <id>

Cause

The "ceph orch osd rm <id>" command does not completely clear all references to the removed OSD.

Additional Information

Note:

1. The OSD is removed from the cluster to the point that it is not visible anymore in the crush map and its auth entry (ceph auth ls) is removed.

2. Example "cephadm shell -- timeout --verbose 10 ceph --connect-timeout=5 orch ps --format yaml" excerpt, in this case the OSD ID removed was OSD.10:
 
daemon_type: osd
daemon_id: '10'
hostname: shaxx
container_image_name: <ip_or_dns>:5000/registry.suse.com/ses/7/ceph/ceph
status: 0
status_desc: stopped
osdspec_affinity: ''
is_active: false
last_refresh: '<date>T<time>'
events:
- '<date>T<time> daemon:osd.10 [ERROR] "auth get failed: failed to find
  osd.10 in keyring retval: -2"'

3. systemd still shows the service in a failed state on the OSD host, excerpt:
 
ceph-<cluster_fsid>@osd.10.service   loaded   inactive dead   Ceph osd.10 for <cluster_fsid>

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020642
  • Creation Date: 21-Apr-2022
  • Modified Date:21-Apr-2022
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center