How to enable NFS related debug logging

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10
 

A NFS Server or Client on SUSE Linux Enterprise is not working as expected. The logs don't provide any details about what goes wrong.

There is additional debug logging that can be turned on for NFS related functions.  Be warned, however, that this logging is very verbose and very cryptic.  It may not be helpful to anyone but a developer.  A better approach (than the one here) is probably to pursue analysis of the problem through network packet captures (i.e. tcpdump).  Of course, either of these methods (debug logs or packet captures) will require certain skill and experience.  If you are getting this debug information with the intention of bringing it to someone else to analyse, then you probably will want that person's input ahead of time, concerning what method of troubleshooting and data gathering to use for your particular symptoms.  You may just waste time if you gather data for someone without their input.

If you are going to gather debug logs of an NFS issue, here is the approach:

1.  Set up the environment so that the problematic behavior can be reproduced immediately. It is best to enable additional debugging when the issue is directly reproduced and no other distracting logs have to be read and analysed. Note, also, that the debug logging can be so intensive and verbose that it may slow down performance of the system.  It is not very desirable to leave these debug methods turned on for long periods of time.  So the basic steps should be:

a.  Enable debugging.
b.  Promptly reproduce the issue.
c.  Disable debugging.

2.  rpcdebug is a command-line tool that enables and disables debugging for various nfs related modules, and various categories of debug logging within those modules.  The "-m" flag identifies the module were debug is being activated, and the "-s" option identifies which debug flags will be set.

To generate debug logs for NFS Server functions:

rpcdebug -m nfsd -s all

To generate debug logs for NFS Client functions:

rpcdebug -m nfs -s all

[Note that the difference between nfs server and nfs client is "nfsd" vs "nfs" respectively.]

In many cases the RPC protocol needs to be debugged as well, and that can be achieved by using:

rpcdebug -m rpc -s all

The additional logging information will appear in /var/log/messages or can be viewed with the command "dmesg".

To disable the respective debugging options, use the same command but with the "-c" option to clear the flags in question.

rpcdebug -m nfsd -c all
rpcdebug -m nfs -c all
rpcdebug -m rpc -c all

In cases where you are pretty sure about where the issue resides (nfs server vs nfs client) you may not need to enable debugging at both ends.  But in some cases, it may be beneficial to enable rpc/nfs at the NFS client system, and rpc/nfsd at the NFS Server system.

3.  Alternative methods:

On some quite old Linux distributions, the "rpcdebug" command does not exist.  However, that command is really just a convenient front end to setting values in:

/proc/sys/sunrpc/nfsd_debug
/proc/sys/sunrpc/nfs_debug
/proc/sys/sunrpc/rpc_debug

Even without an "rpcdebug" command, you can accomplish the same thing with the sysctl command, or with the echo command.

For example, the sysctl version of the above 3 commands would be:

sysctl -w sunrpc.nfsd_debug=0x7fff
sysctl -w sunrpc.nfs_debug=0xffff
sysctl -w sunrpc.rpc_debug=0x7fff

And setting any of those to 0 or 0x0 would turn that module's debug off again.

Or the echo versions would be:

echo 0x7fff > /proc/sys/sunrpc/nfsd_debug
echo 0xffff > /proc/sys/sunrpc/nfs_debug
echo 0x7fff > /proc/sys/sunrpc/rpc_debug

And echoing 0 or 0x0 into those locations would turn them off again.