NFS mount attempts fail with "permission denied" or succeed initially but fail after 30 minutes
This document (000020618) is provided subject to the disclaimer at the end of this document.
SUSE Linux Enterprise Server 15 SP2
SUSE Linux Enterprise Server 15 SP1
SUSE Linux Enterprise Server 15 SP0
NFS v3 client mount attempts against a Linux may fail immediately, or may succeed but after 30 minutes stop working, with "permission denied".
There are, of course, many reasons an NFS Server could return "permission denied," but for this particular scenario, several unique factors and clues are present.
1. The possible difference in v3 behavior as described above.
2. When checking certain output at the Linux NFS Server machine, the following error may be seen:
conf_parse: last line non-terminated, ignored.
Places on the Linux NFS Server machine where this message have been noted include:
a. Output of "exportfs -v"
c. Output of "systemctl status nfs-server" (but not from "systemctl status nfsserver")
d. Output of "journalctl"
3. After "permission denied" is given at the client machine, issuing the following command at the NFS Server machine:
will show a line similar to:
nfsd 192.168.1.245 -no-domain-
It is the "-no-domain-" part of that line which is of unique interest here. This indicates that something has gone wrong with the way NFS clients are being trusted (allowed) to mount what the NFS Server is exporting.
4. When "permission denied" is received from the NFS Server, packet traces show that the RPC header (not NFS header) of the reply packet includes the following (as viewed from wireshark):
bad crediential (seal broken) (1)
1. To fix manually (recommended, even if code will also be updated):
The last line of certain NFS-related configuration files may be unterminated. In other words, the file does not end with a "new line" character. This is quite unusual, as most Linux based editors will automatically include a new line at the end of an ASCII file when it is typed up and saved, even if the user does not explicitly press <enter> after the final line. As such, it is probably best to correct this condition if it is found. Files where this condition may trigger this problem include:
/etc/nfs.conf.local #which may not exist, but (if it does) is the most likely source of the problem
Edit those files. If, while editing the file, it is already possible to move down to a final empty line, then nothing needs to be changed. However, if the file ends at the right hand side of a non-empty line (even in a comment line) this can trigger the problem. After the last character of the last line, insert a new line by pressing the <enter> key. If using "vi", be sure to be in "insert mode" first.
You could also check those files for new-line characters with the "od" command. For an example, see the "Additional Information" section below.
2. To fix this with updated code:
The "nfs-kernel-server" package needs to be updated. Incidentally, this package is built from a source code package named "nfs-utils," so it is best to update the "nfs-client" package at the same time, as both are built from the "nfs-utils" source package.
On SLES 15 SP1 / SP2 / SP3, the fix was first supplied in nfs-kernel-server 2.1.1-10.21.1. For SP3, this is available in general maintenance updates. However, for SP1 and SP2 it is available only with LTSS or ESPOS entitlement.
On SLES 15 without any support pack, the fix was first supplied in nfs-kernel-server 2.1.1-6.17.1 (available only with LTSS or ESPOS entitlement).
Upstream fix is
Commit 7fc4d064f951 ("mountd: Initialize logging early.")
In the upstream public project for nfs-utils, this fix first came in nfs-utils version 2.4.2. However, within SLES, it has been backported into the earlier versions listed above in the "Resolution" section.
# End of /etc/nfs.conf.local
Using this command to view it:
od -cb nfs.conf.local
0000000 # E n d o f / e t c / n f
043 040 105 156 144 040 157 146 040 057 145 164 143 057 156 146
0000020 s . c o n f . l o c a l \n # # #
163 056 143 157 156 146 056 154 157 143 141 154 012 043 043 043
0000040 # # # # # # # # # # # # # # # #
043 043 043 043 043 043 043 043 043 043 043 043 043 043 043 043
0000060 # # # # # # # # #
043 043 043 043 043 043 043 043 043
In the od output above, the end of the file's first line is designated with the "\n" symbol, aka "new line", aka value 012. That value may differ from 012 if other od options are used. The final line has no such character at the end, meaning it is missing the new line character and could trigger the problem.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000020618
- Creation Date: 16-Mar-2022
- Modified Date:16-Mar-2022
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: email@example.com