Non-Kerberos NFS mounts experience delays or permissions problems if Kerberos is configured for other reasons

This document (000020779) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 15

Situation

Various situations might exist.  Some examples:

1.  A Linux NFS client is attempting to mount a v4 share.  The mount attempt may take 1 - 4 minutes to complete.  If the attempt is part of boot, it may delay the boot significantly, during which the slow mount attempt may abort rather than complete.

There may be many reasons for mount delays, but in this particular situation the NFS client machine has Kerberos configured for certain services, but not specifically for NFS use.

2.  A Linux NFS client has mounted an NFS share successfully, but afterwards "permission denied" errors come up even when filesystem permissions should be adequate.  For example, a "ls" command might show some permission denied errors, but still show a partial directory list.  It might show some lines of output with a lot of questions marks (????) present.  This behavior could signify a lot of different problems, but one potential cause is confusion about whether kerberos should be used.

Resolution

The NFS and RPC layers on a modern NFS Client will attempt to detect and negotiate what form a security should be used.  This is a bit different from the behavior years ago, when the nfs client would always assume "sec=sys" if no other value was specified.  (See "man nfs" for more information about sec=<value>.)

Kerbose usage (in one form or another) is becoming more frequently found on modern systems.  As a side effect, it is becoming more common for the NFS client code to mistakenly believe it should use kerberos, even when it is not the administrator's intention.  If the NFS client decides to use kerberos, but full support for NFS hosts or NFS users is not really present in the kerberos databases, malfunctions can occur.

1.  The first thing to try would be to explicitly set "sec=sys" as one of the NFS mount options.  This should avoid many possible points of confusion.  This can be placed in /etc/fstab or in autofs maps that might be in use.

2.  Another option is to forcibly block NFS's ability to query Kerberos.  This can be done with the commands:
systemctl mask rpc-gssd.service
systemctl mask rpc-svcgssd.service

With these masked, subsequent reboots will not start those services, and the NFS components will not be able to use Kerberos.

If a reboot cannot be done right away, these services can be stopped with "systemctl stop <service-name>".  Alternatively, rpc.gssd and rpc.svcgssd processes can be manually identified and killed.  Typically, only NFS makes use of these services for access into Kerberos.  Other things needing Kerberos do not use them.

3.  It can also help to track down and correct any Kerberos problems.  For example, in one case it was found that none of the KDCs known to the Linux NFS client machine were reachable over the network.  This caused timeout delays while KDC connections were retried several times.  For more details, see the "Cause" section.

Cause

NFSv4 clients try to use gss/krb5i (Kerberos) to negotiate initial state with the server.  If Kerberos is not available, NFS will quickly fall back to auth-sys.

In cases where Kerberos has not been set up for any reason, there will be no keytab file (/etc/krb5.keytab).  The absence of that file will correctly cause rpc-gssd and rpc-svcgssd services to abort (fail to start).  When Kerberos is not set up, this is the desired result.  This prevents NFS from making attempts to use Kerberos, and NFSv4 will quickly revert to auth-sys.

If rpc-gssd or rpc-svcgssd have aborted because there is no /etc/krb5.keytab file, "systemctl status <service-name>" against those services will show a message such as:

Condition: start condition failed at Wed 2022-09-21 10:46:25 MDT; 14min ago
           ConditionPathExists=/etc/krb5.keytab was not met

However, if /etc/krb5.keytab does exist, these services will successfully start and NFS can make queries about NFS credentials.  Once such queries are made, if positive results are found, the client will continue to try to use kerberos, even though that may not be the administrator's intention.  In contrast, if the queries are answered quickly saying that no NFS credentials exist (a negative response), NFS should quickly fall back to auth-sys.

On the other hand, if the queries are not answered at all, this means that Kerberos is malfunctioning, or its configuration is bad, or KDCs cannot be reached, etc.  In those cases, a series of several independent 30 second timeouts and retries will occur, adding up to very significant delays.

Additional Information

Services rpc-gssd and rpc-svcgssd are considered to be "static" (always started under certain conditions) rather than configurable as "enabled" or "disabled".  Thus, to prevent them from starting, they need to be masked, as shown above.  A service which is has been masked can be made available again with the "unmask" directive, i.e. "systemctl unmask rpc-gssd.service".

Be aware that these daemon names contain dots (such as rpc.gssd) whereas systemd treats dots differently, so for  systemd configuration naming, dots in daemon names are replaced with hyphens, resulting in designations such as:
rpc-gssd.service.

The service rpc-gssd is used by both NFS Servers and Clients.

The service rpc-svcgssd is used by NFS Servers.

The nfs-client package (built from nfs-utils source package) provides rpc.gssd and rpc.svcgssd.  These services could technically allow other RPC-based services to use Kerberos if those other services were written to take advantage of them.  However, this author does not know of any RPC service (other than NFS) which does so.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020779
  • Creation Date: 27-Apr-2023
  • Modified Date:27-Apr-2023
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center