Cluster status reports MDSs behind on trimming

This document (000019740) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6

Situation

The cluster status shows the following health warnings:

HEALTH_WARN <x> MDSs behind on trimming
HEALTH_WARN x clients failing to respond to cache pressure

Resolution

Temporarily increase the following values:
 
ceph config set mds mds_cache_trim_threshold 512K
ceph config set mds mds_recall_max_caps 10000
ceph config set mds mds_recall_max_decay_rate 1.5
ceph config set mds mds_cache_memory_limit xxxxxxxxxx
 
Also see the Additional Information Section.

Cause

One possible cause is that clients are failing to respond to cache pressure in a timely manner and are not returning / releasing caps (fast enough).

Additional Information

Note that the increased settings when set as per the resolution section are not permanent and will revert back to default once a MDS is restarted. Specifically regarding the "mds_cache_memory_limit", this is dependent on the total amount of memory available on the server. If feasible, double the current setting.

Also see TID 000019591: When running "du" command on a cephfs mount, ceph -s reports 1 MDSs report oversized cache.

To get more details on the clients caps usage, the following commands can be useful:
 
ceph tell mds.<ins_mds_server_name> client ls
ceph daemonperf mds.<ins_mds_server_name> (needs to be executed on the MDS host)

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000019740
  • Creation Date: 23-Oct-2020
  • Modified Date:23-Oct-2020
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center