Once your organization has decided to implement Kubernetes, you have to make another incredibly important decision: Where is Kubernetes going to run?
In this blog post, we’ll discuss managing Kubernetes yourself in the cloud vs. using a Managed Kubernetes Service or KaaS offering.
What’s “Self-Hosted” Kubernetes?
Kubernetes by itself is like the foundation of a house. If you want more than foundation, such as networking, storage, and scalability, you have to add it yourself. This can be cumbersome for teams that aren’t ready for it (more on how to solve this next). However, if you do have the team to manage it, it can be a great asset.
Self-hosted Kubernetes means that you manage every part of Kubernetes, from the Control Plane (Etcd, Scheduler, API Server, Controller Manager) to every aspect of the Worker Nodes (Kubelet, Container Runtime, Kube-Proxy) yourself.
There are a few ways to do this:
- Bootstrap Kubernetes manually
- Bootstrap Kubernetes in an automated fashion
The “manual” option is what you’d see with something like Kubernetes-The-Hard-Way by Kelsey Hightower, a popular way to truly learn the ins and outs of Kubernetes. This is great from a learning perspective (and I do recommend everyone go through it at least once), but no engineers really implement it like this in production. There are engineering teams that are, for example, boostrapping Etcd on separate servers for scalability purposes, but at the time of writing this, it’s a unique case (not impossible, just unique).
The automated bootstrapping option is still self-hosted and you still must manage all of the components of the Control Plane and Worker Nodes, but it’s deployed in an automated way. The most popular boostrapper that’s production-ready is Kubeadm. In fact, a lot of Managed Kubernetes Services, which you’ll learn about next, use Kubeadm underneath the hood to bootstrap the Kubernetes Services. It’s a great way to automate a Kubernetes cluster deployment while still managing and keeping control of the underlying Kubernetes components.
What’s A Managed Kubernetes Services?
Managed Kubernetes Services or Kubernetes-as-a-Service (KaaS) is a literal service in the cloud specifically for Kubernetes. Three of the most popular are Azure Kubernetes Service (AKS), AWS’s Elastic Kubernetes Service (EKS), and the Google Kubernetes Engine (GKE).
The largest and most important difference between running Kubernetes yourself vs. using a managed solution is that the Control Plane is abstracted away, along with some Worker Node components like having to manually add a CNI plugin (although, you can manage your CNI if you want to. You just don’t have to). Scaling and managing Etcd, the API server, Scheduler, and the Controller Manager is no longer your concern. It’s all handled by “the cloud.” You do still have to manage updates though, as in, the Kubernetes API updates.
Having the Control Plane managed for you is great for teams that:
- Don’t have the engineering expertise to do all of it
- Don’t want to hire engineers to specifically do the infrastructure piece
- Want engineers to worry about other platforms vs managing the underlying infrastructure of Kubernetes. More Dev, less Ops.
You’ll still manage the Worker Nodes. They run in VMs within the cloud and have all of the nifty cloud features like auto-scaling so you don’t have to worry about it.
However, there is a new trend called “Serverless Kubernetes.” With Serverless Kubernetes, there’s no longer a need for Worker Nodes. You can run the entire Kubernetes cluster as a serverless service (say that ten times fast). AWS Fargate and GKE AutoPilot are two of the largest services for this purpose. With Fargate Profiles or AutoPilot enabled, depending on which cloud you’re in, the need for Worker Nodes running in VMs is completely removed. Everything from an underlying VM management perspective is abstracted away from you.
When To Choose One Over The Other
Regardless of where you run Kubernetes, it’s still Kubernetes. That said, Managed Kubernetes Services in the cloud will “alter” the way it’s installed. For example, with AKS out of the box, you get a bunch of Container Storage Interfaces (CSI) available for Azure. With EKS, you get a few that work in AWS. Of course, you won’t see AWS CSIs in an AKS deployment. The reason why this is possible is because Kubernetes is open-source and the installation of it can be modified.
On the flip side, with self-hosted, it’s a “choose your own adventure.” There are no CSIs configured, no CNIs, no authentication practices, nothing. It’s all up to you to decide, and that’s not a bad thing. Some organizations prefer having the customization ability over the “out of the box” experience.
Another aspect to think about is scalability. The cloud is scalable, but things happen and sometimes services go down. Around August/September, AKS had an issue that was cross-region. An Ubuntu update bug (and Ubuntu runs many of the underlying Worker Nodes for AKS) went out, which broke many new Deployments to the clusters. This issue was fixed in a day or two, but a day or two could be millions lost for an organization. With that, having a hybrid model like running a Managed Kubernetes offering and a self-hosted cluster for scalability purposes may not be such a bad idea.
So, when should you choose Managed Kubernetes over self-hosted?
Choose self-hosted if you want the 100% customization option along with not wanting to be tied to the Managed option so you can control everything yourself. For example, Mercedes (the car company) runs over 1,000 Kubernetes clusters on OpenStack. They chose not to go the cloud route because they wanted full control over the customization ability vs being locked into a specific configuration.
Choose the Managed Kubernetes option if you don’t want to have to manage the customization piece and you want something that’s more prepared out of the box. With Managed Kubernetes you get a cluster that’s 80% managed for you and ease of scalability.
Both options are great. It really just comes down to three things:
- Expertise to run self-managed Kubernetes
- The need to run self-managed vs having an out-of-the-box experience with Managed Kubernetes
- How much vendor lock-in you want to deal with
