Removing an OSD using ceph orch requires some additional cleanup

This document (000020642) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7

Situation

When removing an OSD using "ceph orch osd rm <id>" as per the Online Documentation , the removed OSD is still reported by "ceph orch ps" and on the node that hosted the OSD systemctl still shows the service in a failed state.

Resolution

It is needed to clean up the removed OSD by executing:

ceph orch daemon rm osd.<id>

If needed, also wipe the disk:

cephadm ceph-volume lvm zap --destroy --osd-id <id>

Cause

The "ceph orch osd rm <id>" command does not completely clear all references to the removed OSD.

Additional Information

Note:

1. The OSD is removed from the cluster to the point that it is not visible anymore in the crush map and its auth entry (ceph auth ls) is removed.

2. Example "cephadm shell -- timeout --verbose 10 ceph --connect-timeout=5 orch ps --format yaml" excerpt, in this case the OSD ID removed was OSD.10:

daemon_type: osd
daemon_id: '10'
hostname: shaxx
container_image_name: <ip_or_dns>:5000/registry.suse.com/ses/7/ceph/ceph
status: 0
status_desc: stopped
osdspec_affinity: ''
is_active: false
last_refresh: '<date>T<time>'
events:
- '<date>T<time> daemon:osd.10 [ERROR] "auth get failed: failed to find
osd.10 in keyring retval: -2"'

3. systemd still shows the service in a failed state on the OSD host, excerpt:

ceph-<cluster_fsid>@osd.10.service loaded inactive dead Ceph osd.10 for <cluster_fsid>

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.