NFS performance drops dramatically after a SLES 10 support pack or SLES 11 (or higher) install.

This document (7003313) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10

Situation

After updating a SLES NFS Server (often to a newer support pack or newer major release), NFS performance may decrease dramatically, especially for operations writing a large number of small files. Even non-NFS-related operations may noticeably drop in performance.

Resolution

This typically occurs when a journaling file system (like reiser or ext3) is in use, and as part of the update, a disk driver is enhanced to support barrier settings. Journaling file systems want to first write the journal entry (reflecting what they are about to commit to the normal data portion of the file system). Once that is successful, they go ahead with the actual "commit-to-disk." That way, if power is lost during the commit action, corruption is avoided because the journal can be used to recover. However, many disks have hardware level features like caching or reordering of writes, to improve performance. As such, there is risk that the journal entry might not be 100% written to the journal before the commit-to-disk begins. This could nullify the safety of a journaling file system.

To avoid this possibility, "barriers" were created. Barriers insure that journal entries finish getting written to the journal before the commit takes place. Thus, barriers help insure data does not get corrupted, but they do so at a performance penalty. Depending on the number and nature of operations being done, the penalty can be extreme. Some Linux distributions don't turn on barriers by default because of this penalty.

SLES distributions have barriers enabled. However, some disk drivers do not support barriers, so the "fast-but-dangerous" behavior is still going on.

If a system's disk driver is updated from a version that did not support barriers to one which does, performance will likely drop. This is considered unfortunate but necessary for data integrity.

If the system is on a good UPS (Uninterruptible Power Supply) which ensures clean, constant power, then it would generally be considered safe to disable the barrier and enjoy the higher performance. To disable barriers, edit /etc/fstab (on the system which is acting as the NFS Server). Find the entry which represents the file system in question. NOTE: This refers to a file system on a local disk, not an NFS mount definition. Add the option to turn off the barrier:

For ext3: barrier=0 (default is 1)

For reiserfs: barrier=none (default is "flush")

For xfs: nobarrier (but see NOTE FOR XFS below in "Additional Information" section)

For example, before-and-after entries for an ext3 file system might look as follows:

before:

/dev/disk/by-id/ata-whatever /home ext3 acl,user_xattr 1 1

after:

/dev/disk/by-id/ata-whatever /home ext3 acl,user_xattr,barrier=0 1 1

Then either reboot or umount / mount the file system again. If this does not return the system to the previous performance, then the performance drop was for some other reason, and it may be best to consider turning the barrier back on.

Additional Information

A SUSE system with disk drivers that do NOT support barriers will typically give messages during boot, indicating that barriers can not be enabled. In such cases, if barriers are set to (or defaulting to) 1, the following types of messages may be seen:

JBD: barrier-based sync failed on hda2 - disabling barriers

reiserfs: disabling flush barriers on hdb1

NOTE FOR XFS:

Beginning in Linux kernel 4.19.x (and therefore effecting SLES 15 SP2 and beyond) the "nobarrier" option for XFS file systems has been completely deprecated. If used, it will cause the XFS file system to fail to mount.

Previous to that, beginning in Linux kernel 4.10.x (and therefore effecting SLES 12 SP4, SP5 and SLES 15 SP0, SP1) the "nobarrier" option is ignored. It's use will not come into effect, but it will not prevent the XFS file system from mounting.

For more information on this "nobarrier" deprecation for XFS, see:
https://www.suse.com/support/kb/doc/?id=000020240

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.