Troubleshooting Kubernetes Service Discovery – Part 1

This is the first of two articles I’m writing on the topic of troubleshooting Kubernetes service discovery.

In this article, we’ll validate the server-side infrastructure. In the next article we’ll look at the client-side components and list some simple troubleshooting activities.

> Read my Demystifying Kubernetes service discovery article if you need to get up-to-speed on Kubernetes service discovery.

Kubernetes service discovery and DNS

The Domain Name System (DNS) is at the heart of the internet. It’s also at the heart of service registration and service discovery in Kubernetes. It consists of server components and client components. The main server component is a Kubernetes-native DNS server, and the main client component is the DNS resolver that runs in every container.

At a high level, application microservices talk to each other using names. However, these names need to be resolved to IP addresses. When an application initiates a connection using a name, the DNS resolver running on the application’s container sends queries to the DNS server asking it to resolve the the name to an IP address. If you like analogies, it’s similar to converting a contact’s name to a phone number in order to call them.

Anyway…. let’s look at how to inspect the DNS server components in a Kubernetes cluster.

Kubernetes DNS server Pods

Modern Kubernetes clusters (since about 1.12) use CoreDNS to provide a highly available DNS server within the cluster.

CoreDNS is deployed as a Kubernetes-native application that runs in the kube-system Namespace as a set of Pods managed by a Deployment object called “coredns”. The following commands list the Deployments and Pods that must be running for service discovery to work.

$ kubectl get deploy -n kube-system -l k8s-app=kube-dns

NAME READY UP-TO-DATE AVAILABLE AGE

coredns 2/2 2 2 18h

$ kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME READY STATUS RESTARTS AGE

coredns-5644d7b6d9-b7nsk 1/1 Running 0 18h

coredns-5644d7b6d9-zwmm5 1/1 Running 0 18h

You can also use kubectl describe to get more detailed info and events.

If you delete the Pods, replacements will automatically be created in their place. There may be a short period when service discovery is unavailable but the registration database will remain in tact. In troubleshooting situations, such as a situation where one DNS Pod is returning errors, you can safely delete the faulty Pod and a new one will be started.

Kubernetes DNS Service object

The CoreDNS Pods sit behind a Service called `kube-dns`. Like all Services, this provides a stable name, IP address, and set of ports that clients can connect to.

$ kubectl get service -n kube-system -l k8s-app=kube-dns

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)

kube-dns ClusterIP 192.168.200.10 <none> 53/UDP,53…

The kube-dns name is a hangover from previous versions of Kubernetes that used a technology called kube-dns instead of CoreDNS. The CLUSTER-IP value is the stable virtual IP that clients will connect to, and needs to match what is in each container’s /etc/resolv.conf file. Port 53 is the correct DNS port.

All of the above must be running and correctly configured for service registration and service discovery to work.

It’s possible to delete one, or all, of the coredns Pods and let the Deployment object restart them.

Kubernetes DNS logs

You can also inspect the logs of the DNS Pods with the following command. You’ll have to substitute the name of the Pods in your environment, but the log output should be similar.

$ kubectl logs coredns-5644d7b6d9-74pv7 -n kube-system

2020-02-19T21:31:01.456Z [INFO] plugin/reload: Running configuration…

2020-02-19T21:31:01.457Z [INFO] CoreDNS-1.6.2

2020-02-19T21:31:01.457Z [INFO] linux/amd64, go1.12.8, 795a3eb

CoreDNS-1.6.2

linux/amd64, go1.12.8, 795a3eb

Next week we’ll explain the client-side configuration, plus some commands to test DNS service discovery.

If you liked this article, you’ll love my Kubernetes book and video courses. In fact, even if you didn’t like this article you’ll probably like my books and videos — they’re way better 😉