After reboot some OSDs are reported as being down.

This document (7021460) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 4

Situation

After rebooting a storage node, "ceph osd tree" output shows not all the OSDs (Object Storage Daemons) belonging to that node are up:

-2 1.33096     host ses-node-X                                  
 1 0.21329         osd.1           down  1.00000          1.00000
 4 0.21329         osd.4           up  1.00000          1.00000
 7 0.90439         osd.7           down  1.00000          1.00000 

Resolution

A possible workaround is to execute the following command:

ceph-disk activate-all

Cause

With a larger number of OSDs per host, it can happen that the ceph-disk timeout is reached resulting in not all OSDs being activated after a server reboot.

Additional Information

To verify this is the case, the journalctl output for affected OSD devices (in the below example excerpt output for the device sdi2) will show the following:

:~ # journalctl -u ceph-disk@dev-sdi2.service
-- Logs begin at di 2017-07-11 14:06:49 CEST, end at wo 2017-07-19
21:17:45 CEST. --
jul 11 14:07:09 sesnode-4 systemd[1]: Stopped Ceph disk activation: /dev/sdi2.
jul 11 14:07:09 sesnode-4 systemd[1]: Starting Ceph disk activation: /dev/sdi2...
...
jul 11 14:09:10 sesnode-4 systemd[1]: ceph-disk@dev-sdi2.service: Main process exited, code=exited, status=124/n/a
jul 11 14:09:10 sesnode-4 systemd[1]: Failed to start Ceph disk activation: /dev/sdi2.
jul 11 14:09:10 sesnode-4 systemd[1]: ceph-disk@dev-sdi2.service: Unit entered failed state.
jul 11 14:09:10 sesnode-4 systemd[1]: ceph-disk@dev-sdi2.service: Failed with result 'exit-code'.

The "code=exited, status=124" in the above output indicates the ceph-disk timeout was reached.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7021460
  • Creation Date: 18-Sep-2017
  • Modified Date:03-Mar-2020
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center