How to speed up / slow down deep-scrubbing

This document (000019684) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 5.5
SUSE Enterprise Storage 6

Situation

Customer has noticed that deep-scrubbing has fallen behind. 

Resolution

If impacting ceph performance is not a concern, the following could be done:
Configure *osd max scrubs* and *osd scrub load threshold* to do more scrubbing work, but be careful as it will have a huge impact on performance.

Get current configuration by running the following command on an OSD node:

# ceph daemon osd.$osdID config show | egrep "osd_max_scrubs|osd_scrub_load"
Example:
# ceph daemon osd.22 config show | egrep "osd_max_scrubs|osd_scrub_load"
    "osd_max_scrubs": "1",
    "osd_scrub_load_threshold": "0.500000",

Assuming that impacting cluster clients is of no concern.
If system load average is above .5, then increase osd_scrub_load_threshold, 
For system load average see: uptime, cat /proc/loadavg, top, sar -q 1 5, w
if load average is above consider increasing "osd scrub load threshold=", but may want to check randomly through out the day.
salt -I roles:storage cmd.shell "sar -q 1 5"
salt -I roles:storage cmd.shell "cat /proc/loadavg"
salt -I roles:storage cmd.shell "uptime"

Otherwise increase osd_max_scrubs:
ceph tell 'osd.*' injectargs --osd_max_scrubs=2 --osd_scrub_load_threshold=3

Validate with:
# ceph daemon $osdID config show | egrep "osd_max_scrubs|osd_scrub_load"
or
# ceph daemon $osdID config diff

Values will vary on each cluster.
Monitor for desired results with "ceph -s", etc....
If need drop back values back down closer to defaults if the commands negatively impact the cluster.

Cause

Cluster may have been backfilling.
Cluster may have client load.

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000019684
  • Creation Date: 14-Aug-2020
  • Modified Date:14-Aug-2020
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center