1 large objects found in pool 'default.rgw.meta'

This document (000019698) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 5.5

Situation

The following "ceph -s" output is observed with  "1 large omap objects" 

#==[ Command ]======================================#
# /usr/bin/ceph --connect-timeout=5 -s
cluster:
health: HEALTH_WARN
1 large omap objects
services:
mon: 3 daemons, quorum mon1,mon2,mon3
mgr: mon2(active), standbys: mon1
osd: 120 osds: 120 up, 120 in
rgw: 1 daemon active
data:
pools: 6 pools, 6400 pgs
objects: 13.10M objects, 99.3GiB
usage: 447GiB used, 152GiB / 599GiB avail
pgs: 6399 active+clean
1 active+clean+scrubbing+deep
io:
client: 336KiB/s rd, 391op/s rd, 0op/s wr

"ceph health detail" includes pool information "1 large objects found in pool 'default.rgw.meta'"
#==[ Command ]======================================#
# /usr/bin/ceph --connect-timeout=5 health detail
HEALTH_WARN 1 large omap objects
1 large objects found in pool 'default.rgw.meta'
Search the cluster log for 'Large omap object found' for more details.

egrep -i 'Large omap object found' /var/log/ceph/ceph.log
./ceph/log/ceph/ceph.log
2020-08-26 00:15:14.726736 osd.10 osd.10 10.1.1.8:6800/354953 9 : cluster [WRN] Large omap object found. Object: 3:caf94831:users.uid::USER.NAME.buckets:head PG: 3.8c129f53 (3.3) Key count: 510003 Size (bytes): 92776321
2020-08-26 01:29:08.129042 osd.10 osd.10 10.1.1.8:6800/354953 20 : cluster [WRN] Large omap object found. Object: 3:caf94831:users.uid::USER.NAME.buckets:head PG: 3.8c129f53 (3.3) Key count: 510003 Size (bytes): 92776321

Resolution

  • Delete unused buckets.
  • Create multipule users and spead the buckets evenly across all users. 
In v12.2.13 the default value for osd_deep_scrub_large_omap_object_key_threshold configuration parameter, which defines when it should start to warn about large omap object, has been changed from 2000000 to 200000 (decreased 10 times).

Currently they have 510003 keys:

2020-08-26 01:29:08.129042 osd.10 osd.10 172.19.121.58:6800/354953 20 : cluster [WRN] Large omap object found. Object: 3:caf94831:users.uid::DDOS.SIIRFE.buckets:head PG: 3.8c129f53 (3.3) Key count: 510003 Size (bytes): 92776321

That is why they do not observe the warning on production (510003 < 2000000) and observe it in the upgraded test environment (510003 > 200000).
Just for the reference, the release notes that mention the change [1] (search for "Lower the default value of osd_deep_scrub_large_omap_object_key_threshold") and the PR with the change [2].

[1] https://docs.ceph.com/docs/master/releases/luminous/
[2] https://github.com/ceph/ceph/pull/29175/files

The customer can change the setting in ceph.conf by using the 
"Adjusting ceph.conf with Custom Settings"
https://documentation.suse.com/ses/5.5/single-html/ses-admin/#ds-custom-cephconf

Create a osd.conf file. 
add:
osd_deep_scrub_large_omap_object_key_threshold = 2000000

A lower value is also acceptable as well:
osd_deep_scrub_large_omap_object_key_threshold = 1000000
osd_deep_scrub_large_omap_object_key_threshold = 750000
 
Then apply the configuration with the following two commands:
# salt 'Admin*' state.apply ceph.configuration.create
# salt '*' state.apply ceph.configuration

Where "Admin*" is the name of the Admin/Salt Master node.
or run stage.2, 3, 4.  

Cause

User indices are not sharded, ie. we store all the keys of names of buckets under one object. This can cause large objects to be found. The large object is only accessed in the List All Buckets S3/Swift API. Unlike bucket indices, the large object is not exactly in object IO path. Depending on the use case for so many buckets, the warning isn't dangerous as the large object is only used for List All Buckets API.

The error shows a user has 500K buckets causing the omap issue. Sharding does not occur at the user level. Bucket indexes are sharded but buckets per user is not (and usually the default max_bucket is 1000).

You can find the bucket stats:
$ radosgw-admin bucket stats > bucket-stats.txt

Number of buckets:
# grep '"bucket":' bucket-stats.txt | wc -l
510003

Number of bucket owners (replace USER.NAME with real user name):
# grep '"owner":' bucket-stats.txt | uniq
        "owner": "USER.NAME",

Number of buckets owned by "USER.NAME" (replace USER.NAME with real user name)
# grep '"owner": "USER.NAME"' bucket-stats.txt | wc -l
510003

Number of buckets containing data:
# grep '"size_actual":' bucket-stats.txt | wc -l
211159

Number of empty buckets: 
# grep '"usage": {}' bucket-stats.txt | wc -l
298844

The single owner called USER.NAME is the owner of all buckets in the cluster. There are 500K (510003) buckets owned by this owner, of which only 200K (211159)are actually containing objects.

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000019698
  • Creation Date: 10-Sep-2020
  • Modified Date:15-Sep-2020
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center