Word on the Cloud: Keeping you up-to-date on cloud native. Short & sharp!

Ask Nigel: How many Kubernetes Masters

I’ve been doing a series of live-stream labs and Q&A with Linode. Inevitably, there’s never enough time to answer all questions. This week I address the following question from Anil Dugar:

QUESTION: How many K8s masters should be present in production environment?

Here’s two really quick answers:

  1. You know your environment better than me, so you decide 😉
  2. Usually 3 or 5 is best

Let’s dig a bit deeper…

Quick background

Kubernetes masters run the cluster controlplane where the Kubernetes intelligence lives. As such, it’s important that it’s highly available (HA).

A well-designed HA control plane should always start with an odd number of masters, and 3 or 5 is usually the magic number. 

Why an odd number?

Consider the following scenario…

You’re running with 4 masters a network incident isolates masters 1 and 2 from masters 3 and 4. In this scenario, every master knows that an incident occurred, but none of them knows what happened to the two missing masters — they could be offline, or simply inaccessible via the network. 

This poses the following problem. Kubernetes masters need a majority quorum (more than 50%) to operate properly. However, in the above scenario, neither side has a majority. This forces the control plane into read-only mode where your apps continue to work but configuration changes are not allowed.

If you’d started with  an odd number of masters (see below) and the same network partition occurred, one side would have a majority and would continue cluster operations as normal. This is why 3 is better than 4, and 5 is better than 6.

Why is 7, 9, or 11 masters not the magic number?

This is all to do with achieving consensus and consistency as quickly as possible.

Basically, all cluster changes need pushing to all masters to maintain a consistent view of the cluster. More masters means the time to consistency is longer. It’s a bit like deciding where to eat dinner — if there’s only three in the group it won’t take long to decide, but if there’s 33 in the group you could spend half the night deciding.

This is why, generally speaking, 3 and 5 are usually good numbers, with anything above 7 approaching slowdown territory.

What happens if my masters fail?

If your masters fail, or there’s no clear majority, the control plane enters a majority failure state and goes into read-only mode. Applications will continue to work but changes are not permitted.

For more details, see my Getting Started with Kubernetes course (you might have to right-click the link and choose “open in a new tab”).

Wrap up

If you have a Docker or Kubernetes question, drop me a message via the contact form at the bottom of the home page and use AksNigel at the top of the message. I get a lot of questions, so can’t promise to answer them all 🙁

Finally, feel free to connect on the various socials where I’m happy to talk about technology all day long!


Share this post



Special Editions

Word on the cloud: What's going on in cloud native

Nigel’s Keeping you up-to-date on cloud native. Short & sharp! #Docker #Kubernetes #WebAssembly #Wasm

© 2023 Nigel Poulton – All rights reserved


Looking for something specific?

Try the search facility.