NFS access using CephFS FSAL fails with an input / output error after upgrading to the latest release

This document (000020812) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7.1

Situation

The cluster was initially deployed using SUSE Enterprise Storage (SES) 5.x, then upgraded to SES 6 and finally upgraded to SES 7.1. After upgrading the cluster to SES 7.1, NFS access from clients that worked normally before the upgrade fails with:

cat <path_to_nfs_mounted_data_file>
cat: <path_to_nfs_mounted_data_file> : Input/output error

Resolution

Set the application metadata for the configured CephFS data and metadata pools:

ceph osd pool application set <cephfs_metadata_pool> cephfs <cephfs_data_pool> cephfs

Cause

The Cephfs application metadata were not set for the Cephfs data pools.

Additional Information

On clusters that were initially deployed with SES 5.x, when "enable application cephfs" was executed for the cephfs data pools, the application metadata was actually never set, as it was not supported with this version.

It was not yet a problem with SES 6 because the osd caps for current users still allowed access, but after the upgrade to 7.1 (when NFS export creates / created a new user with more restricted osd caps) access stopped working.

To verify if the application metadata is set properly one can for example use:

ceph osd pool ls detail --format=json-pretty|grep -A 2 application_metadata
"application_metadata": {
"cephfs": {
"metadata": "cephfs"
---
"application_metadata": {
"cephfs": {
"data": "cephfs"

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.