NFS4 mount shows all ownership as "nobody" or 4294967294

This document (7005060) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10

Situation

An NFS client is successfully mounting an NFS v4 file system.  However, upon executing "ls -al," most the files' user and group ownership are showing as "nobody" or as "4294967294".  However, when the same files are viewed directly at the NFS Server (not through an NFS client) they show different ownership, typically a more desired and expected ownership.

NOTE:  The main discussion in this document only applies to cases where ownership at an NFS client is being reported DIFFERENTLY that the native file system reports when viewed directly at the NFS Server machine.  For cases where *both* the NFS client and NFS server show "nobody" unexpectedly, see the "Additional Information" section below.

Resolution

When the NFS client shows ownership of "nobody" but the NFS Server shows a different (usually more desired) ownership, this means that NFS 4 "id mapping" (governed by idmapd, the identity mapping daemon) is being attempted, but is not set up correctly.  ID mapping is an alternative to the method of simply sending UIDs and GIDs between client and server.

The identity mapping daemon is not used in NFS v3.  See the "Cause" section for more details on the purpose of idmapd, or for discussion of NFS v3 versus v4 in this regard.

NOTE, however, that correcting the idmapd configuration may not be the most desired solution.  The need for ID mapping was mandatory when NFS4 first came about, but now it is considered optional.  Adding ID mapping to an NFS configuration is generally considered complicated, unnecessary, and without real benefits.  Therefore, if possible, it is usually desired to avoid the use of ID mapping.

1.  Avoiding the idmapping method

1.1  Avoiding idmapping on Linux NFS Clients

A Linux NFS client will attempt (or not attempt) to use idmapd based on the nfs module option:

nfs.nfs4_disable_idmapping=1

This can be set to 0 or 1, off or on.  Note that this is a "disable" option, so having it ON (1) means that the use of idmapping will be disabled (or off).  However, even with idmapping "disabled", if the NFS client is interacting with a NFS Server with a very old implementation, the NFS Server might insist that idmapping be used, and then the client will fall back to using idmapping anyway.

SLES 12 and 15 (or likely anything using kernel 3.3 or higher) default to having this option set to "1", so those systems acting as NFS clients will not attempt idmapping by default.
SLES 11 (using kernel 3.0.x) has this option set to 0 by default, so it will need to be set to 1 if desired.
SLES 10 (using kernel 2.x) probably did not have this option at all.

The current value can be checked with:
cat /sys/module/nfs/parameters/nfs4_disable_idmapping

The option can be set a variety of ways:

A.  It can be done via Yast --> System --> Boot loader, by adding the kernel command line option:
nfs.nfs4_disable_idmapping=1
 
B.  Alternatively, it can take effect slightly later during boot if the following has been done:
Edit or create /etc/modprobe.d/99-nfs.conf
and add (or modify) the line:
options nfs nfs4_disable_idmapping=1"
 
C.  It can also be set on-the-fly (in case immediate reboot is not possible) with:
 
echo 1 > /sys/module/nfs/parameters/nfs4_disable_idmapping

Note, however, that setting it on the fly will only impact mounts performed after it is set.  It will be necessary to umount and remount existing nfs4 mounts to put it into effect there.  It is best to umount all mounts first, before remounting any, because multiple mounts pointing to one server can share an environment.

1.2  Avoiding ID mapping on Linux NFS Servers

A Linux NFS Server using kernel 3.4 or higher (including SLES 12 and 15) will allow clients to submit requests with either method:  ID mapping, or passing plain UID/GIDs.  So having NFS Servers "avoid" idmapping is simply a matter of being on a relatively modern kernel and then insure the clients avoid asking for it as well.

NFS Servers on older kernels (such as used on SLES 10 or 11) will insist on using idmapping.  An NFS 4 client which attempts to use the UID/GID method will be told to use idmapping instead.

1.3  Avoiding ID Mapping on non-Linux NFS Servers or Clients.

Many non-Linux NFS Servers or client may not automatically avoid using idmapping.  They may have configuration options which control whether NFS 4 will insist on using idmapping or will allow the old UID/GID method from v3 to be used on v4 as well.  It can simplify NFS v4 usage greatly to configure such 3rd party devices to allow the v3 method (UID/GID) to be used with v4 also.

2.  If or when ID Mapping is MANDATED

If ID mapping MUST be used, it is usually because either the NFS client or NFS server code (or both) is quite old.  In some cases, it is mandated by administrative decision.  In those cases, if the "nobody" ownership symptom (the original title and subject of this document) is present, it is typically caused when the NFS v4 identity mapping daemon (idmapd) is not running, or it is misconfigured, or it is holding old information in it's cache.
 
2.1   If idmapd not running.
 
2.1.1  SLES NFS Servers

SLES NFS Servers (those which export or "share" their file systems via NFS) need to be running idmapd in case a client requests that method.  On Linux, check for an idmapd process with:
 
ps aux | grep idmapd
 
A process for "/usr/sbin/rpc.idmapd" should be found.
 
On SLES 10, you can restart idmapd with the command "rcidmapd restart".
 
On SLES 11, 12, and 15, idmapd is not separately started, it is part of the initialization of the nfsserver service.  It should start any time "nfsserver" service is started.  So "rcnfsserver restart" can be used on SLES 11, or "systemctl restart nfsserver" on SLES 12 or 15.

2.1.2 SLES NFS Client machines

2.1.2.1  "Recent" SLES NFS client systems (SLES 12 SP4 and higher) typically do not require idmapd to be loaded at the client.  They will use an upcall program called "nfsidmap" to accomplish the client functions of idmapping.

2.1.2.2   Earlier NFS 4 clients (older code found in SLES 10, 11, and early 12 support packs) required idmapd to be running at the client machine as well.  On those and other older Linux distributions, check for an idmapd process with:
 
ps aux | grep idmapd
 
A process for "/usr/sbin/rpc.idmapd" should be found.
 
On SLES 10, you can restart idmapd with the command "rcidmapd restart".
 
On SLES 11 and early 12 support packs, idmapd may start as part of the initialization of the nfs (client) service.  So "rcnfs restart" can be used on SLES 11, or "systemctl restart nfs" on SLES 12.
 
However, in some cases idmapd might not start.  For example, if the /etc/fstab does not contain any NFS v4 mount, it is possible that idmapd will not be started even when the "nfs" service starts.
 
2.2.  If idmapd is misconfigured
 
For correct idmapping between NFS clients and NFS servers, both must have good /etc/idmapd.conf settings.  Even when the same user accounts are known to both the servers and clients, idmapd configuration problems can prevent proper ownership from being displayed.
 
Check the /etc/idmapd.conf file.  The [General] section should have a Domain setting.  The domain is an arbitrary string but it must be set identically on NFS clients and their NFS servers.  This setting often reports "localdomain" by default, and that is usually adequate.  In many cases, administrators have set it to match the company's DNS domain name, but that is not required, and would technically be a coincidence, rather than accomplish anything meaningful.  The only requirement is that clients and servers use the idmapd domain as each other.

It can also be helpful for /etc/idmapd.conf to contain a [Translation] section which specifies the method of translating between names and IDs.  Typically, it is best to point to nsswitch methodology.
  
So, for example, a typical idmapd.conf file might look like the following.  If this file is changed, idmapd must be restarted.
 
[General]
Verbosity=7
Pipefs-Directory=/var/lib/nfs/rpc_pipefs
Domain=localdomain

[Mapping]
Nobody-User=nobody
Nobody-Group=nobody

[Translation]
Method=nsswitch
 
2.3.  If idmapd is caching old information.
 
Old information can be cached by idmapd.  This may temporarily prevent idmapd from learning new information.  Even the fact that a user has no mapping can be cached (still believed) beyond the point where newer information is created.  This is known as a "negative" cache.  The cache defaults to keeping entries for 10 minutes.  This is usually fine, because user identities do not change or get created very often.  However, idmapd may learn early in system boot that certain users were not found.  This might be due only to a delay in access to information.  For example, idmapd might start before sssd or other methods of obtaining user identities.  In such a case, negative cache might exist for the first 10 minutes after server boot.  To make the cache clear more quickly:
 
Stopping and starting idmapd will correct this, but that is not always desired.
 
On SLES 12 and higher, the command "nfsidmap -c" is available and should clear the cache and allow new information to be learned.
 
Another option is to lower the cache timer.  This is set inside /etc/sysctl.conf with:
 
fs.nfs.idmap_cache_timer = 60
(this value represents seconds, the default is 600)
 

Cause

When NFS 4.x was original conceived and implemented, a feature of ID mapping was created and mandated.  This concept had weaknesses both foreseen and unforeseen, so it has since become undesired and optional.  More detail of ID Mapping follows, for those less-common implementations which may still use it.  Note that this discussion may not fully apply to NFS over Kerberos.

If the ID mapping method is in use, the correct display of usernames depends upon the NFS v4 server having the same knowledge of user and group accounts (including their UIDs and GIDs) as the NFS v4 client is using, and both must be in the same idmapd domain.
 
When ID mapping is used, NFS v4 will pass identities between servers and clients in the form 'username@domainname'.
 
This is a major change from the traditional method of passing UID and GID numbers.  With the UID/GID method, an NFS server can store and report these UID values in the file system, even if it has no knowledge of the user accounts they belong to.  So essentially, a UID that was not known to one side or the other could still be handled in a valid way.  In contrast, ID mapping does not allow unknown identities to be preserved.  It will convert them to "nobody".
 
If users and groups are centrally managed, and all systems have access to the same identity store, idmapd's methods work fairly easily.  But it is crucial that NFS servers and NFS clients have access to identical account information and idmapd domain name settings, otherwise idmapd cannot properly do its job, and may display ownership as "nobody" or equivalent high values.
 
Put simply, in NFS v4 with idmapd, the same username@domainname must be recognized by both sides and represent the same UID, or it may be mapped to "nobody".

One of the original intentions of ID mapping (when it was conceived theoretically) was that the NFS client and NFS server would not need to get information from the same identity stores, and (for example) "userBOB@domainORG" would not need to be known as the same UID at both ends.  However, this theory did not keep in mind that the RPC layer (which all NFS depends upon) still focuses on UIDs and GIDs, so this goal is not currently obtainable.

Additional Information

The above concerns with idmapd CANNOT cause problems on NFS v3, because v3 does not use identity mapping.  Even so, when using NFS v3, sometimes users see ownership unexpectedly showing "nobody" and bring up similar questions.  Additionally, even on NFS v4, "nobody" can be seen unexpectedly for a reason not related to idmapd,.

However, in these non-idmapd variations of the symptom, "nobody" would appear simultaneously on both the client and the server.  In other words, the client and server would agree on the "nobody" ownership.

This is because of a separate common cause:  The concept of "root_squash".  By default, an NFS Server which gets a request from a client machine's root user will "squash" the request and treat it as if it came from user "nobody".  Therefore, after a NFS client's root user creates something, both the NFS client view and the NFS server view would agree that the entity is owned by "nobody".

For root squash concerns, the most common solutions are:

(a)  Make sure NFS client processes run as a non-root user.
or
(b)  Set the NFS server to export with the option "no_root_squash", so the NFS client's root user can be treated as the NFS server's root user -- though this is less secure.  See "man exports" for more information.
or
(c)  Set the NFS server to squash the root user to an ID other than nobody.  This is accomplished with "anonuid" and "anongid" export settings.  See "man exports" for more information.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7005060
  • Creation Date: 17-Dec-2009
  • Modified Date:06-Nov-2023
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center