Scheduling deep-scrubbing schedule for off peak hours

This document (000019714) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Stoarge 5.5
SUSE Enterprise Stoarge 6

Situation

Customer needs to schedule deep-scrubbing for off peak hours to help minimize impact on cluster. 

Resolution

It's possible to set up a deep-scrubbing schedule if there are off peak hours for the deep-scrubbing to occur. 

Default behavior is for all pg's to be deep-scrubbed once a week.  It may be necessary to increase interval values  to every other week to ensure that pg's are deep-scrubbed periodically. 

Example:
Start scrubbing at 8am, go until 17 (5pm), begin_week_day (Monday), end_week_day (Friday), the configuration would be as below.

osd_scrub_begin_hour = 8
osd_scrub_end_hour = 17
osd_scrub_begin_week_day = 1
osd_scrub_end_week_day = 6
osd_scrub_max_interval = 1209600
osd_scrub_min_interval = 259200
osd_deep_scrub_interval = 1209600

Note: “deep scrubbing” process will continue after “osd_scrub_end_hour” parameter until the “deep scrubbing” for the pg is completed.

In some cases the cluster is already busy, so deep-scrubbing does not occur frequent as needed.
See: https://www.suse.com/support/kb/doc/?id=000019684

Also see:
https://www.suse.com/support/kb/doc/?id=000019621

General ceph documentation reference:
https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/

In some cases, customers choose to use their on scripts to ensure deep-scrubbing occurs on their own schedule. 

Cause

Deep-scrubbing can impact client access to the cluster. 

Status

Top Issue

Additional Information

Other information which can help determine deep-scrubbing status

Scrubbing & Deep+Scrubbing Distribution by hours, day of week, or date
Columns 22 & 23 are scrub history
Columns 25 & 26 are for deep-scrub history
These columns will change, if "ceph pg dump" output changes.

# ceph pg dump | head -n 8 | grep "active"
dumped all
1.7f          0                  0        0         0       0     0           0          0    0        0 active+clean 2020-09-16 02:24:25.675595          0'0     1753:247  [31,1,15]         31  [31,1,15]             31          0'0 2020-09-16 02:24:25.675555             0'0 2020-09-16 02:24:25.675555             0

________________________________________
Scrubbing Distribution by Days of the week and hours of the day.
Get Scrubbing distribution by days of the week:
# for date in `ceph pg dump | grep active | awk '{print $22}'`; do date +%A -d $date; done | sort | uniq -c
dumped all
    329 Tuesday
    175 Wednesday
Get Scrubbing distribution per hours of the day:
# for date in `ceph pg dump | grep active | awk '{print $23}'`; do date +%H -d $date; done | sort | uniq -c
dumped all
   23 00
   15 01     
   22 02     
   20 03     
   28 04     
   27 05
  ---[cut here]---

________________________________________
Get Deep+Scrubbing Distribution by Days of the week and hours of the day.
Get Deep+Scrubbing distribution by days of the week:
# for date in `ceph pg dump | grep active | awk '{print $25}'`; do date +%A -d $date; done | sort | uniq -c
dumped all
     51 Friday
     60 Monday
     70 Saturday
     75 Sunday
     45 Thursday
    132 Tuesday
     71 Wednesday

Get Deep+Scrubbing distribution per hours of the day:
# for date in `ceph pg dump | grep active | awk '{print $26}'`; do date +%H -d $date; done | sort | uniq -c
dumped all
     28 00
     20 01
     19 02
     17 03
     24 04
     24 05
---[cut here]---
________________________________________
Get Deep_Scrubbing distribution by date:
# ceph pg dump | grep active | awk '{print $25}' | sort | uniq -c
dumped all
      7 2020-09-08
     25 2020-09-09
     45 2020-09-10
     50 2020-09-11
     70 2020-09-12
     73 2020-09-13
     60 2020-09-14
    125 2020-09-15
     49 2020-09-16

________________________________________
Sort pg by "DEEP_SCRUB_STAMP":

Date & Time:
ceph pg dump | grep active | awk '{print $1,$25,$26}' | sort -k2,1 -k2

Date only:
ceph pg dump | grep active | awk '{print $1,$25}' | sort -k2

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000019714
  • Creation Date: 04-Aug-2021
  • Modified Date:04-Aug-2021
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center