Get Nigel’s weekly Kubernetes and Docker update direct into your inbox – Subscribe here

Ask Nigel: How many Kubernetes Masters

I’ve been doing a series of live-stream labs and Q&A with Linode. Inevitably, there’s never enough time to answer all questions. This week I address the following question from Anil Dugar:

QUESTION: How many K8s masters should be present in production environment?

Here’s two really quick answers:

  1. You know your environment better than me, so you decide 😉
  2. Usually 3 or 5 is best

Let’s dig a bit deeper…

Quick background

Kubernetes masters run the cluster controlplane where the Kubernetes intelligence lives. As such, it’s important that it’s highly available (HA).

A well-designed HA control plane should always start with an odd number of masters, and 3 or 5 is usually the magic number. 

Why an odd number?

Consider the following scenario…

You’re running with 4 masters a network incident isolates masters 1 and 2 from masters 3 and 4. In this scenario, every master knows that an incident occurred, but none of them knows what happened to the two missing masters — they could be offline, or simply inaccessible via the network. 

This poses the following problem. Kubernetes masters need a majority quorum (more than 50%) to operate properly. However, in the above scenario, neither side has a majority. This forces the control plane into read-only mode where your apps continue to work but configuration changes are not allowed.

If you’d started with  an odd number of masters (see below) and the same network partition occurred, one side would have a majority and would continue cluster operations as normal. This is why 3 is better than 4, and 5 is better than 6.

Why is 7, 9, or 11 masters not the magic number?

This is all to do with achieving consensus and consistency as quickly as possible.

Basically, all cluster changes need pushing to all masters to maintain a consistent view of the cluster. More masters means the time to consistency is longer. It’s a bit like deciding where to eat dinner — if there’s only three in the group it won’t take long to decide, but if there’s 33 in the group you could spend half the night deciding.

This is why, generally speaking, 3 and 5 are usually good numbers, with anything above 7 approaching slowdown territory.

What happens if my masters fail?

If your masters fail, or there’s no clear majority, the control plane enters a majority failure state and goes into read-only mode. Applications will continue to work but changes are not permitted.

For more details, see my Getting Started with Kubernetes course (you might have to right-click the link and choose “open in a new tab”).

Wrap up

If you have a Docker or Kubernetes question, drop me a message via the contact form at the bottom of the home page and use AksNigel at the top of the message. I get a lot of questions, so can’t promise to answer them all 🙁

Finally, feel free to connect on the various socials where I’m happy to talk about technology all day long!

@nigelpoulton

Share this post

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn

Leave a Reply

Your email address will not be published. Required fields are marked *

Contact / Hire Nigel

Send a message if you have any questions. We will reply as soon as possible. 

Nigel's tech books

Want to learn, understand and apply Kubernetes or Docker in your day to day work. 

Contact
Subscribe

Get Nigel’s weekly K8s and Cloud-native tech update direct to your inbox. Tips, news, advice, announcements, videos and more.

© 2021 Nigel Poulton – All rights reserved

Search

Looking for something specific?

Try the search facility.