Get Nigel’s weekly Kubernetes and Docker update direct into your inbox – Subscribe here

Troubleshooting Kubernetes Service Discovery – Part 1

This is the first of two articles I’m writing on the topic of troubleshooting Kubernetes service discovery. 

In this article, we’ll validate the server-side infrastructure. In the next article we’ll look at the client-side components and list some simple troubleshooting activities.

> Read my Demystifying Kubernetes service discovery article if you need to get up-to-speed on Kubernetes service discovery.

Kubernetes service discovery and DNS

The Domain Name System (DNS) is at the heart of the internet. It’s also at the heart of service registration and service discovery in Kubernetes. It consists of server components and client components. The main server component is a Kubernetes-native DNS server, and the main client component is the DNS resolver that runs in every container.

At a high level, application microservices talk to each other using names. However, these names need to be resolved to IP addresses. When an application initiates a connection using a name, the DNS resolver running on the application’s container sends queries to the DNS server asking it to resolve the the name to an IP address. If you like analogies, it’s similar to converting a contact’s name to a phone number in order to call them.

Anyway…. let’s look at how to inspect the DNS server components in a Kubernetes cluster.

Kubernetes DNS server Pods

Modern Kubernetes clusters (since about 1.12) use CoreDNS to provide a highly available DNS server within the cluster.

CoreDNS is deployed as a Kubernetes-native application that runs in the kube-system Namespace as a set of Pods managed by a Deployment object called “coredns”. The following commands list the Deployments and Pods that must be running for service discovery to work.

$ kubectl get deploy -n kube-system -l k8s-app=kube-dns

NAME      READY   UP-TO-DATE   AVAILABLE   AGE

coredns   2/2          2                         2                     18h

$ kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME                                               READY   STATUS    RESTARTS   AGE

coredns-5644d7b6d9-b7nsk          1/1            Running   0                    18h

coredns-5644d7b6d9-zwmm5       1/1            Running   0                    18h

You can also use kubectl describe to get more detailed info and events.

If you delete the Pods, replacements will automatically be created in their place. There may be a short period when service discovery is unavailable but the registration database will remain in tact. In troubleshooting situations, such as a situation where one DNS Pod is returning errors, you can safely delete the faulty Pod and a new one will be started.

Kubernetes DNS Service object

The CoreDNS Pods sit behind a Service called `kube-dns`. Like all Services, this provides a stable name, IP address, and set of ports that clients can connect to.

$ kubectl get service -n kube-system -l k8s-app=kube-dns

NAME         TYPE          CLUSTER-IP        EXTERNAL-IP   PORT(S)

kube-dns   ClusterIP    192.168.200.10    <none>              53/UDP,53…

The kube-dns name is a hangover from previous versions of Kubernetes that used a technology called kube-dns instead of CoreDNS. The CLUSTER-IP value is the stable virtual IP that clients will connect to, and needs to match what is in each container’s /etc/resolv.conf file. Port 53 is the correct DNS port.

All of the above must be running and correctly configured for service registration and service discovery to work.

It’s possible to delete one, or all, of the coredns Pods and let the Deployment object restart them.

Kubernetes DNS logs

You can also inspect the logs of the DNS Pods with the following command. You’ll have to substitute the name of the Pods in your environment, but the log output should be similar.

$ kubectl logs coredns-5644d7b6d9-74pv7 -n kube-system

2020-02-19T21:31:01.456Z [INFO] plugin/reload: Running configuration…

2020-02-19T21:31:01.457Z [INFO] CoreDNS-1.6.2

2020-02-19T21:31:01.457Z [INFO] linux/amd64, go1.12.8, 795a3eb

CoreDNS-1.6.2

linux/amd64, go1.12.8, 795a3eb

Next week we’ll explain the client-side configuration, plus some commands to test DNS service discovery.

If you liked this article, you’ll love my Kubernetes book and video courses. In fact, even if you didn’t like this article you’ll probably like my books and videos — they’re way better 😉

Kubernetes video training courses
The Kubernetes Book

Share this post

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn

Leave a Reply

Your email address will not be published. Required fields are marked *

Contact / Hire Nigel

Send a message if you have any questions. We will reply as soon as possible. 

Nigel's tech books

Want to learn, understand and apply Kubernetes or Docker in your day to day work. 

Contact
Subscribe

Get Nigel’s weekly K8s and Cloud-native tech update direct to your inbox. Tips, news, advice, announcements, videos and more.

© 2021 Nigel Poulton – All rights reserved

Search

Looking for something specific?

Try the search facility.