How to improve DNS resolution performance inside a kubernetes cluster?
This document (000020020) is provided subject to the disclaimer at the end of this document.
This article gives a quick overview of DNS resolution inside a Kubernetes cluster. It also explains how performance can be improved.
- A working Kubernetes cluster with CoreDNS or kube-dns installed.
When a workload is deployed on a Kubernetes cluster, the nameserver (DNS server) is set to the Service IP address of CoreDNS, along with the search options.
# cat /etc/resolv.conf nameserver 10.43.0.10 search cattle-system.svc.cluster.local svc.cluster.local cluster.local members.linode.com options ndots:5
- The first search option is generally
- Second option is
- Third one being just
- The rest of the options are copied over from what's configured on the node. (options from
/etc/resolv.confon the node)
The above options are helpful in the following ways: - When a workload tries to connect to a service in the same namespace, it can simply reference it using the destination service name. Ex: hello-world-service - Similarly, when it tries to connect to a service in a different namespace, it's enough to append the namespace name after the service name. Example: database-service.finance-app
The above conveniences come at a cost, though! Suppose the workload tries to resolve a service name outside of the cluster. In that case, the DNS client first exhausts appending all the options to this name before finally resolving to the correct IP, this is known as search path expansion.
Say the workload needs to resolve
login.mycompany.com. The DNS client sends out a query for
login.mycompany.com.<namespace-where-workload-is-deployed>.svc.<cluster domain>. The DNS server responds correctly with a nonexistent domain message (NXDOMAIN) as it's an invalid domain.
Next, the DNS client tries to resolve
login.mycompany.com.svc.<cluster domain>, followed by
As you can see, there are a lot of lookups, on top of this there is an
A (IPv4) query, and a quad A
AAAA (IPv6) query sent out for each of the above.
All these unnecessary queries add to the network traffic and increase the load on the CoreDNS service.
To avoid DNS client cycling through each of the options specified in
/etc/resolv.conf, add a trailing dot
. to the name that's being resolved outside of the cluster, this ensures the DNS client performs an absolute lookup. Example:
Similarly, when possible, use the full service name (
As a further recommendation, some clusters can benefit from deploying NodeLocal DNS. This is particularly useful in a production cluster or a cluster with a high frequency of DNS queries.
There are a number of ways that NodeLocal DNS improves on the reliability and scalability of CoreDNS, please see the documentation for details.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000020020
- Creation Date: 06-May-2021
- Modified Date:06-May-2021
- SUSE Rancher
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com