NFS mount randomly fails when going through Cisco device with Firewall Services Module

This document (7016980) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 3 (SLES 11 SP3)

Situation

An NFS server provides an export which is mounted on multiple NFS clients.
Randomly one NFS client will loose it's connection to that export while other NFS clients are still able to access it.

Resolution

Resolution #1

This will most directly addresses the issue with the TCP Sequence Number Randomization feature, and is Cisco's recommendation:

Since TCP Sequence Number Randomization is a legacy feature that was supposed to protect hosts that use predictable algorithms for initial TCP sequence number generation, it is does not provide much additional security on the modern TCP stacks. Hence, the feature can be selectively disabled to take full advantage of TCP SACK and achieve the maximum throughput on a single TCP flow. The best way to disable the randomization is to use Modular Policy Framework (MPF); you can also narrow the class down just to those trusted hosts that do the high-speed transfers:

class-map TCP

match port tcp range 1 65535

policy-map global_policy

class TCP

set connection random-sequence-number disable

service-policy global_policy global

Resolution #2
Another recommendation if the Cisco device cannot be configured properly, is to disable TCP SACK on the end point servers.

Here are some commands to help you with this change.

Display the endpoint servers current TCP SACK setting
sysctl -a | grep -i tcp_sack
# my test server returns - net.ipv4.tcp_sack = 1

Change the value for this session:
sysctl -w net.ipv4.tcp_sack=0
# my test server returns - net.ipv4.tcp_sack = 0
This will not survive a reboot.

Reload sysctl
sysctl -p
This command reloads the settings in the /etc/sysctl.conf file which means that you could simply add this line to the /etc/sysctl.conf file so that on reboot, or a sysctl -p it would be loaded:
net.ipv4.tcp_sack = 0

Turning off the TCP SACK option will affect all future TCP connections with any other device.
In other words, TCP SACK will no longer be seen as an available TCP option for all TCP connections.
That will effectively work around the Cisco devices TCP Sequence Number Randomization issue with the TCP SACK option.

Cause

NFS/TCP: TCP SACK handling is broken with seq randomization on Cisco Fire Wall Services Module
See this Cisco document for more information and recommendations:

https://supportforums.cisco.com/document/48551/single-tcp-flow-performance-firewall-services-module-fwsm
Specifically the section entitled: TCP Sequence Number Randomization and SACK

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.