Non-Kerberos NFS mount execution takes excess time if Kerberos is configured on the system for other reasons

This document (000020779) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 15

Situation

A Linux NFS client is attempting to mount a v4 share.  The mount attempt may take 1 - 4 minutes to complete.  If the attempt is part of boot, it may delay the boot significantly, during which the slow mount attempt may abort rather than complete.

There may be many reasons for mount delays, but in this particular situation the NFS client machine has Kerberos configured for certain services, but not specifically for NFS use.

Resolution

There may be something wrong with the Kerberos configuration or functionality, which can effect NFS v4 even when mounts are not explicitly configured to use Kerberos (See the "Cause" section of this document for more information). There are two ways to deal with this:

(1) Track down and correct any Kerberos problems.  For example, in one case it was found that none of the KDCs known to the Linux host (aka the NFS client machine) were reachable.  They may have been powered off or blocked by firewalls, etc.

or 

(2) Forcibly remove NFS's ability to query Kerberos.  This can be done with:

systemctl mask rpc-gssd.service
systemctl mask rpc-svcgssd.service

With these masked, subsequent reboots will not start those services, and NFS will not be able to attempt to use Kerberos.

If a reboot cannot be done right away, these services can be stopped with "systemctl stop <service-name>" or alternatively rpc.gssd and rpc.svcgssd can be killed.  Typically, only NFS uses these services for access into Kerberos.  Other things needing Kerberos do not use them.

Cause

The overall cause is usually a broken Kerberos environment.  Having no Kerberos configuration would be better than a broken one, in this case.

NFSv4 clients always try to use gss/krb5i (Kerberos) to negotiate initial state with the server.  If Kerberos is not available, NFS will quickly fall back to auth-sys.

In cases where Kerberos has not been set up for any reason, there should be no file at /etc/krb5.keytab.  The absence of that file will correctly cause rpc-gssd and rpc-svcgssd services to abort (fail to start).  When Kerberos is not set up, this is the desired result.  This prevents NFS from making any attempts to use Kerberos, and NFSv4 will quickly revert to auth-sys.

If rpc-gssd or rpc-svcgssd have aborted because there is no /etc/krb5.keytab file, "systemctl status <service-name>" against those services will show a message such as:

Condition: start condition failed at Wed 2022-09-21 10:46:25 MDT; 14min ago
           ConditionPathExists=/etc/krb5.keytab was not met

However, if /etc/krb5.keytab does exist, these services will successfully start and NFS can make queries about NFS credentials.  Once such queries are made, if they are answered quickly saying that no NFS credentials exist, NFS will quickly fall back to auth-sys and no delays will occur.  If the queries are not answered, this means that Kerberos is malfunctioning, or its configuration is bad, or KDCs cannot be reached, etc.  Negative responses should normally be returned quickly.  But if no answers (neither positive nor negative) can be obtained, a series of 30 second timeouts and retries will occur, causing significant delays.

In summary, if Kerberos has been setup for any reason, it needs to be functioning correctly.  Otherwise, NFS can experience initial delays while figuring out it Kerberos should be used.

Additional Information

Services rpc-gssd and rpc-svcgssd are considered to be "static" (always started during boot) rather than configurable as "enabled" or "disabled".  Thus, to avoid starting them at boot, they need to be masked, as shown above.  A service which is has been masked can be returned to service with the "unmask" directive, i.e. "systemctl unmask rpc-gssd.service".

Be aware that these daemon names contain dots (such as rpc.gssd) whereas systemd treats dots differently, so for  systemd configuration naming, dots in daemon names are replaced with hyphens, resulting in designations such as:
rpc-gssd.service.

The service rpc-gssd is used by both NFS Servers and Clients.

The service rpc-svcgssd is used by NFS Servers.

The nfs-client package (built from nfs-utils source package) provides rpc.gssd and rpc.svcgssd.  These services could technically allow other RPC-based services to use Kerberos if those other services were written to take advantage of them.  However, this author does not know of any RPC service (other than NFS) which does so.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020779
  • Creation Date: 21-Sep-2022
  • Modified Date:21-Sep-2022
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center