How to improve DNS resolution performance inside a kubernetes cluster?

This document (000020020) is provided subject to the disclaimer at the end of this document.

Situation

Task

This article gives a quick overview of DNS resolution inside a Kubernetes cluster. It also explains how performance can be improved.

Pre-requisites

  • A working Kubernetes cluster with CoreDNS or kube-dns installed.

Overview

When a workload is deployed on a Kubernetes cluster, the nameserver (DNS server) is set to the Service IP address of CoreDNS, along with the search options.

Example:

# cat /etc/resolv.conf
nameserver 10.43.0.10
search cattle-system.svc.cluster.local svc.cluster.local cluster.local members.linode.com
options ndots:5
  • The first search option is generally <namespace-where-workload-is-deployed>.svc.<cluster domain>
  • Second option is svc.<cluster domain>
  • Third one being just <cluster domain>
  • The rest of the options are copied over from what's configured on the node. (options from /etc/resolv.conf on the node)

The above options are helpful in the following ways: - When a workload tries to connect to a service in the same namespace, it can simply reference it using the destination service name. Ex: hello-world-service - Similarly, when it tries to connect to a service in a different namespace, it's enough to append the namespace name after the service name. Example: database-service.finance-app

The above conveniences come at a cost, though! Suppose the workload tries to resolve a service name outside of the cluster. In that case, the DNS client first exhausts appending all the options to this name before finally resolving to the correct IP, this is known as search path expansion.

Say the workload needs to resolve login.mycompany.com. The DNS client sends out a query for login.mycompany.com.<namespace-where-workload-is-deployed>.svc.<cluster domain>. The DNS server responds correctly with a nonexistent domain message (NXDOMAIN) as it's an invalid domain.

Next, the DNS client tries to resolve login.mycompany.com.svc.<cluster domain>, followed by login.mycompany.com.<cluster domain>.

As you can see, there are a lot of lookups, on top of this there is an A (IPv4) query, and a quad A AAAA (IPv6) query sent out for each of the above.

All these unnecessary queries add to the network traffic and increase the load on the CoreDNS service.

Resolution

To avoid DNS client cycling through each of the options specified in /etc/resolv.conf, add a trailing dot . to the name that's being resolved outside of the cluster, this ensures the DNS client performs an absolute lookup. Example: login.mycompany.com.

Similarly, when possible, use the full service name (<destination-service-name>.<destination-namespace>.svc.<cluster domain>). As a further recommendation, some clusters can benefit from deploying NodeLocal DNS. This is particularly useful in a production cluster or a cluster with a high frequency of DNS queries.

There are a number of ways that NodeLocal DNS improves on the reliability and scalability of CoreDNS, please see the documentation for details.

Further reading

  • https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
  • https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020020
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center