How to speed up or slow down osd recovery
This document (000019693) is provided subject to the disclaimer at the end of this document.
SUSE Enterprise Storage 6
The values can be increased if it is needed for a cluster to recover quicker as these help OSDs to perform recovery faster.
- osd max backfills: This is the maximum number of backfill operations allowed to/from an OSD. The higher the number, the quicker the recovery, which might impact overall cluster performance until recovery finishes.
- osd recovery max active: This is the maximum number of active recover requests. The higher the number, the quicker the recovery, which might impact the overall cluster performance until recovery finishes.
- osd recovery op priority: This is the priority set for recovery operation. The lower the number, the higher the recovery priority. Higher recovery priority might cause performance degradation until recovery completes.
As mentioned above, again note that changing these values can impact the overall performance of the cluster and clients may see lower performance.
The default values for the above settings are:
osd_max_backfills = 1
osd_recovery_max_active = 3
osd_recovery_op_priority = 3
ceph tell 'osd.*' injectargs --osd-max-backfills=3 --osd-recovery-max-active=9
NOTE: The above commands will return something like the below message, this can be confusing but in the case of these specific settings can be ignored as they are actually saved and in use:
IMPORTANT: Setting the values to high can cause OSDs to restart, causing the cluster to become unstable, also see the "Additional Information" section below.
To set back to default:
With SES 6, "ceph config set" can alternatively be used:
To set back to default:
ceph config rm osd osd_max_backfills
To view the current settings:
After increasing the settings, should any OSDs become unstable (restarting) or clients are negatively impacted by the additional recovery overhead then reduce the values or set them back to the defaults.
Once the cluster is finished with recovery and back in a HEALTH_OK state, set the values back to default.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000019693
- Creation Date: 06-Apr-2022
- Modified Date:06-Apr-2022
- SUSE Enterprise Storage
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com