Word on the Cloud: Keeping you up-to-date on cloud native. Short & sharp!

Troubleshooting Kubernetes Service Discovery – Part 1

This is the first of two articles I’m writing on the topic of troubleshooting Kubernetes service discovery. 

In this article, we’ll validate the server-side infrastructure. In the next article we’ll look at the client-side components and list some simple troubleshooting activities.

> Read my Demystifying Kubernetes service discovery article if you need to get up-to-speed on Kubernetes service discovery.

Kubernetes service discovery and DNS

The Domain Name System (DNS) is at the heart of the internet. It’s also at the heart of service registration and service discovery in Kubernetes. It consists of server components and client components. The main server component is a Kubernetes-native DNS server, and the main client component is the DNS resolver that runs in every container.

At a high level, application microservices talk to each other using names. However, these names need to be resolved to IP addresses. When an application initiates a connection using a name, the DNS resolver running on the application’s container sends queries to the DNS server asking it to resolve the the name to an IP address. If you like analogies, it’s similar to converting a contact’s name to a phone number in order to call them.

Anyway…. let’s look at how to inspect the DNS server components in a Kubernetes cluster.

Kubernetes DNS server Pods

Modern Kubernetes clusters (since about 1.12) use CoreDNS to provide a highly available DNS server within the cluster.

CoreDNS is deployed as a Kubernetes-native application that runs in the kube-system Namespace as a set of Pods managed by a Deployment object called “coredns”. The following commands list the Deployments and Pods that must be running for service discovery to work.

$ kubectl get deploy -n kube-system -l k8s-app=kube-dns


coredns   2/2          2                         2                     18h

$ kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME                                               READY   STATUS    RESTARTS   AGE

coredns-5644d7b6d9-b7nsk          1/1            Running   0                    18h

coredns-5644d7b6d9-zwmm5       1/1            Running   0                    18h

You can also use kubectl describe to get more detailed info and events.

If you delete the Pods, replacements will automatically be created in their place. There may be a short period when service discovery is unavailable but the registration database will remain in tact. In troubleshooting situations, such as a situation where one DNS Pod is returning errors, you can safely delete the faulty Pod and a new one will be started.

Kubernetes DNS Service object

The CoreDNS Pods sit behind a Service called `kube-dns`. Like all Services, this provides a stable name, IP address, and set of ports that clients can connect to.

$ kubectl get service -n kube-system -l k8s-app=kube-dns

NAME         TYPE          CLUSTER-IP        EXTERNAL-IP   PORT(S)

kube-dns   ClusterIP    <none>              53/UDP,53…

The kube-dns name is a hangover from previous versions of Kubernetes that used a technology called kube-dns instead of CoreDNS. The CLUSTER-IP value is the stable virtual IP that clients will connect to, and needs to match what is in each container’s /etc/resolv.conf file. Port 53 is the correct DNS port.

All of the above must be running and correctly configured for service registration and service discovery to work.

It’s possible to delete one, or all, of the coredns Pods and let the Deployment object restart them.

Kubernetes DNS logs

You can also inspect the logs of the DNS Pods with the following command. You’ll have to substitute the name of the Pods in your environment, but the log output should be similar.

$ kubectl logs coredns-5644d7b6d9-74pv7 -n kube-system

2020-02-19T21:31:01.456Z [INFO] plugin/reload: Running configuration…

2020-02-19T21:31:01.457Z [INFO] CoreDNS-1.6.2

2020-02-19T21:31:01.457Z [INFO] linux/amd64, go1.12.8, 795a3eb


linux/amd64, go1.12.8, 795a3eb

Next week we’ll explain the client-side configuration, plus some commands to test DNS service discovery.

If you liked this article, you’ll love my Kubernetes book and video courses. In fact, even if you didn’t like this article you’ll probably like my books and videos — they’re way better 😉

Kubernetes video training courses
The Kubernetes Book

Share this post



Special Editions

Word on the cloud: What's going on in cloud native

Nigel’s Keeping you up-to-date on cloud native. Short & sharp! #Docker #Kubernetes #WebAssembly #Wasm

© 2023 Nigel Poulton – All rights reserved


Looking for something specific?

Try the search facility.