How to improve DNS resolution performance inside a kubernetes cluster?

This document (000020020) is provided subject to the disclaimer at the end of this document.

Situation

Task

This article gives a quick overview of DNS resolution inside a Kubernetes cluster. It also explains how performance can be improved.

Pre-requisites

A working Kubernetes cluster with CoreDNS or kube-dns installed.

Overview

When a workload is deployed on a Kubernetes cluster, the nameserver (DNS server) is set to the Service IP address of CoreDNS, along with the search options.

Example:

# cat /etc/resolv.conf
nameserver 10.43.0.10
search cattle-system.svc.cluster.local svc.cluster.local cluster.local members.linode.com
options ndots:5

The first search option is generally <namespace-where-workload-is-deployed>.svc.<cluster domain>
Second option is svc.<cluster domain>
Third one being just <cluster domain>
The rest of the options are copied over from what's configured on the node. (options from /etc/resolv.conf on the node)

The above options are helpful in the following ways: - When a workload tries to connect to a service in the same namespace, it can simply reference it using the destination service name. Ex: hello-world-service - Similarly, when it tries to connect to a service in a different namespace, it's enough to append the namespace name after the service name. Example: database-service.finance-app

The above conveniences come at a cost, though! Suppose the workload tries to resolve an external domain name from outside of the cluster. In that case, the DNS client first exhausts appending all the search options to this name before finally resolving to the domain name, this is known as search path expansion.

Say the workload needs to resolve mycompany.com. The DNS client sends out a query for mycompany.com.<namespace>.svc.<cluster domain>. The DNS server responds correctly with a nonexistent domain message (NXDOMAIN) as it is not a known domain.

Next, the DNS client tries to resolve mycompany.com.svc.<cluster domain>, followed by mycompany.com.<cluster domain>.

As you can see, there are three additional queries before the search lookups added by Kubernetes are completed, on top of this there are A (IPv4) queries, as well as a AAAA (IPv6) queries sent out for each of the above.

All these unnecessary queries add to the network traffic and increase the time to get the final query.

Resolution

To avoid the DNS client iterating through each of the search options specified in /etc/resolv.conf, for external domain names it is helpful to add a trailing dot . to the name that's being resolved outside of the cluster, this ensures the DNS client performs an absolute lookup. Example: login.mycompany.com.

Similarly, when possible, use the full domain name for internal service names (<destination-service-name>.<destination-namespace>.svc.<cluster domain>). As a further recommendation, some clusters can benefit from deploying NodeLocal DNS. This is particularly useful in a production cluster or a cluster with a high frequency of DNS queries.

There are several ways that NodeLocal DNS improves the reliability and scalability of CoreDNS, please see the documentation for details.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.