Delayed outgoing packets causing NFS timeouts
This document (000019943) is provided subject to the disclaimer at the end of this document.
SUSE Linux Enterprise Server 12 SP5
nfs: server *HOSTNAME* not responding, still tryingand after several minutes
nfs: server *HOSTNAME* OKBecause of an in-kernel retransmit timer (or another packet being queued), the stuck packet will eventually be sent out, after a delay.
In tcpdump packet capture analysis, this problem can be identified by spurious resend attempts of the same packet (with equal TSVal) a long time apart.
SUSE has released kernel maintanance update that will mitigate this problem by disabling the lockless optimization on pfifo_fast qdisc (which is the only qdisc currently making use of this optimization) .
The issue is solved in the following kernel versions:
- SLES15 SP2: 5.3.18-24.61
- SLES12 SP5: 4.12.14-122.66
echo 'net.core.default_qdisc = fq_codel' >>/etc/sysctl.conf sysctl -w net.core.default_qdisc=fq_codel tc qdisc add dev $devname root handle 1: mq tc qdisc del dev $devname rootIn case the $devname above is not a multiqueue-capable device, the following commands have to be used instead:
echo 'net.core.default_qdisc = fq_codel' >>/etc/sysctl.conf sysctl -w net.core.default_qdisc=fq_codel tc qdisc add dev $devname root handle 1: fq_codel tc qdisc del dev $devname root
However, the lockless optimization has a design flaw which (under certain very specific circumstances) opens a window for a race condition that causes the "last" packet in the queue to be stuck (and not sent out to the wire) for a potentially unbound amount of time, causing network stalls.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000019943
- Creation Date: 20-Apr-2021
- Modified Date:20-Apr-2021
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com