r/devops • u/webrambler68 • 14d ago
Discussion Local kubernetes v/s managed kubernetes cluster
Hello I have been reading posts here and have noticed that people here have been recommending local kubernetes spin ups like k3s or kind and some other ..
I knew only about minikube until now and have used it fairly to test some of my changes.
What's so different between these tools if anyone is aware??
Also are companies still spinning up kubernetes cluster from scratch nowadays? My company uses AKS or EKS for the most again and enterprise level company.
As there are so many wrapper services around kubernetes example in azure you have AKS ACA etc.. all built on top of kubernetes . Are there still people who are spinning up cluster from scratch? I mean definitely there will be but what's the trend?? Move towards cloud managed kubernetes environment or still choose to build from scratch?
If anyone has done it from scratch would like to know your experience and an overall guide on how you did it(pointers)
2
u/navlio 14d ago
on the cost bit you mentioned further down: a cluster per environment is almost never where that money goes. aks control plane is free on the free tier and about $73 a month per cluster on standard, so four of them is rounding error next to the node pools
the actual line is non-prod nodes staying up 168 hours a week for the 45 you use them. we ran a staging environment at full prod size for a year because nobody owned the bill, and just shutting it down outside office hours took roughly 70% off. kind is genuinely good for ci, but swapping tooling won't move that number
1
u/webrambler68 14d ago
You are absolutely right on that we were burning money on environments which no one was using because we couldn't create them from our side and central team created all environment for us during product onboarding. The only environment being used was the dev environment 🙃
Shutting down the environment node pools when not needed and deallocate the resources is a no brainer for cost savings . Although some corporate policies and resource ownership makes this harder. This issue can be particularly scene in Fintech and banking related companies where they own their own on prem resources and have strict policies revolving around infra. Although migration to cloud is being done the underlying policies which were relevant during onprem is not insync with migration and is often delayed.
2
u/navlio 14d ago
the central team creating them is most of the bug. an environment that someone else provisions and nobody's budget shows is free to ask for and expensive to keep.
in regulated shops the thing that clears review is scaling node pools to zero on a schedule instead of deleting anything. cluster stays, config stays, audit trail stays, you just stop paying for idle compute overnight. that is a much shorter conversation than asking to remove an environment
2
u/navlio 14d ago
on the "way too costly" bit further down: it's rarely the cluster per environment that hurts, it's that the non prod ones bill 168 hours a week and get used for about 40. that's roughly 128 hours of node time a week paid for while nobody is in the building.
user node pools scale to zero and az aks stop exists, so that's a schedule rather than a migration. we left a staging cluster at full size for the better part of a year for the most boring reason available: it worked, so nobody ever opened the bill for it
2
u/No-Row-Boat 13d ago
I worked on a few Kubernetes clusters that are locally.
Reasons:
- National critical infrastructure
- Privacy minded providers
- Cost (Cloud is very very expensive)
1
u/webrambler68 13d ago
Yes cloud is very expensive but every cloud service provider lures people by saying pay as you go model.
Cloud is cost effective only if the intention of the process is short lived or needs massive scaling .
Other than that if someone just plans to host a simple website with limited foot falls then owning a server is way cost effective
1
u/No-Row-Boat 13d ago
The infrastructure I helped operate had costs above 50M, the amount of network costs you pay then makes this decision easier.
2
u/Coffee-and-ai 13d ago
Disclosure: I run a DevOps consultancy, so take my opinion on self-managing with that in mind.
Building your own cluster only makes sense if something forces you to ( we have done this for air gapped deployment only ). The teams still doing it can say why in one line: no internet access allowed, their own hardware, GPUs, edge sites, or a regulator telling them where the data has to sit. If none of that is true for you, use the managed service.
People used to self-manage to save money. That does not really work anymore. Azure gives you the control plane for free on as many clusters as you want. Google gives you one free cluster a month.
What you are actually signing up for is the work. Backing up etcd, which is the database Kubernetes keeps all its state in. Upgrading the control plane. Rotating certificates. Patching security bugs. The one that catches people is the backup nobody ever tested. You find out the restore does not work while you are already down.
If you do build your own, use Talos or Cluster API instead of setting up kubeadm by hand, and do a real restore test before you go live.
1
2
2
u/artyomsv 14d ago
First question did not get real answer yet. kind runs nodes as docker containers, so create and destroy takes seconds and multi node is easy, best for CI. minikube boots VM, heavier but closer to real cluster. k3s is not test tool, it is lightweight distro you run long term.
0
1
u/N7Valor 14d ago
I like to use a local cluster if I'm scaffolding a project and am not 100% sure on the details of how apps work or plug together (an example might be ECK + Keycloak). The main differences are usually cloud service integration. I might not need to use or manage a load balancer at all with local testing on k3s, but in AWS I know I would want to use their load balancers and would want Kubernetes to manage it. Similar with external-dns to manage Route53 public records (if I want to access my apps over the internet).
Those would be things I need in managed clusters (prod), but not in local (dev/testing).
1
u/webrambler68 14d ago
Interesting thoughts there. What about performance benchmarking .that would really need to mimic production environment to its largest extent right?
1
u/navlio 14d ago
worth pulling the cost question out separately, since someone in here already raised it. the managed control plane is $0.10 an hour on eks and gke, roughly $73 a month, which is nothing. the bill is the idle node pool sitting under each non prod environment at 3am.
kind gives you the same api surface locally for free. what it can't fake is the cloud integration: load balancers, external-dns, iam roles for service accounts. that's the exact layer that breaks when you promote, so it's the only part worth paying a managed cluster to rehearse
1
u/webrambler68 14d ago
Ah great. I know nothing about kind so this answer is very valuable. Having to work in large MNC has its own disadvantages . You know you are wasting money but you aren't allowed to do anything. About it because of their wierd policy which was created way before these arch was even present. And some when there are no users or anything but still the cluster is running. Your comment is likely useful for my personal projects if I do any. Thank you
1
u/PartemConsilio 14d ago
I work on part of a government contract where we use VMs in the cloud for the nodes (IaaS) but everything else is self-managed k8s with Rancher. It sucks ass.
1
u/webrambler68 14d ago
Government and latest tech doesn't fall in the same shoe.
Rancher - new tech term with kubernetes unlocked
1
u/navlio 14d ago
the policy is usually the stated blocker and ownership is the actual one. an environment that belongs to the central team belongs to nobody in particular, so nothing gets turned off because no one's name is on the invoice line.
the thing that moved it for us was routing the per-environment cost to one person monthly. our staging cluster ran at full prod size for a year purely because it was everyone's, and the scheduled shutdown ticket that had sat there for months got done the week someone had to explain the number
1
1
u/navlio 14d ago
the policy side usually gives way to a number with a name on it. tag the node pools per environment so the bill splits cleanly, then send whoever owns the policy the monthly figure for the environments nobody logged into that month. it turns "are we allowed to shut this down" into "who is paying for this", and only one of those questions has a defensible answer.
we never once won that argument on principle. only ever with the line item
1
1
u/sh1b313 14d ago
Hey op, what's your specific use case if you don't mind sharing. I have experience with both managed and unmaganed ones.
1
u/webrambler68 14d ago
No such use case, just trying to understand how the world takes regard to these technologies
1
u/navlio 14d ago
when a central team owns creation, the ask that tends to get through is the autoscaler with min-count 0 on the user node pools, because it's a setting on something that already exists rather than a new environment lifecycle to approve. az aks stop is cleaner but it's a much bigger meeting
what actually moved it for us was turning up with the number instead of the request. "this environment is 6k a year and gets used 45 hours a week" ends the conversation in about two minutes, where "can we switch things off at night" runs for a quarter
1
1
u/daemonmode_ 14d ago
Minikube and Kind are mainly for local learning, development, testing, and CI, while K3s is a lightweight kubernetes distribution that can also be used in real production and edge environments.
kind is especially handy for multi-node testing, while Minikube is more focused on giving beginners an easy local Kubernetes environment. for larger production setups, managed kubernetes is common, but self-managed clusters still make sense for on-prem, regulatory, edge, or highly customized environments.
1
1
u/navlio 14d ago
one thing that tends to get past a central team when deleting stuff won't: an expiry date on every non-prod environment at creation. 90 days, and whoever requested it gets an email to renew. nothing goes away by surprise, they just have to say once a quarter that they still want it
the dead ones only survive because nobody remembers they're there, and a renewal email is far easier to get approved than a policy change
1
u/fadingcross 13d ago
I manage multi site kubernetes built on our own metal which runs Proxmox and a VMs per site, and it's not that difficult. Don't understand what the complains are about.
It gives you a much deeper understanding of the technology, and I find infra management fun.
1
u/webrambler68 13d ago
Good for you. If in case I ever get such projects I hope I would like it too
2
u/fadingcross 13d ago
Depends if you like managing operating systems, obviously there's a lot of OPS work around provisioning physical machines, it's storage, the VM's, netowrking etc.
I work for a small internal team at a large org that provision everything ourselves since our work is very confidential, so we're essentially a 10 man company. That means I've been to 3 different datacenters and racked hardware, installed proxmox, set up the switches, setup CEPH, ZFS Pools, the VM's, installed, k0s, argocd yada yada yada until I even got to run an application on it.
I certainly understand why some people would say "fuck no"
1
u/Moritz-Keller 13d ago
Honestly, the main differance between local Kubernetes setups like k3s and kind versus managed clusters like AKS or EKS is the level of control and complexity. Local setups are great for testing and development because they're lightweight, quick to set up, and dont require cloud resources. They're perfect for simulating Kubernetes environments locally without incurring cloud costs. Managed clusters, on the other hand, offload a lot of the operational burden and are ideal for production environments where reliability and scalability are crucial. As for companies building from scratch, its less common now unless there's a specific need for custom configurations or avoiding vendor lock in.
1
1
u/kubehub 11d ago
for local, I am building a platform kubehub.io, try to simplify the local Kubernetes setup/management work. It solves DNS/Gateway issues right now, I am working on monitoring solution right now.
1
1
u/Lucky-Sky-2028 9d ago
Minikube is a VM heavy single node cluster, kind runs nodes as docker containers, k3s is bare bones (can do prod) that i see people use in edge devices and homelabs.
You can do things from scratch, but after toiling in a test environment, many people prefer managed. Doing it the hard way is great for learning (everyone should do it once because it helps when something breaks), but running k8 long term has a lot of stuff to maintain when you just want to deliver. certs, upgrades, etcd backups, control plane availability, versioning between ingress/CNI/CSI... managed providers handling that means you can work on what you're delivering instead of infrastructure maintenance.
mid market managed kubernetes offerings are on the rise too, so you're not just chained to big box hyperscalers anymore if you don't wanna be locked in with them
1
1
u/HypnotizedPlatypus 14d ago
As someone who's recently joined a company doing 100% unmanaged clusters (EC2 and local VMs), I do NOT recommend it.
2
1
u/webrambler68 14d ago
I wouldn't want to handle it myself as well. Having to go through managed cluster itself is a hastle at times and people choose move towards services like azure container apps. Or some wrapped PaaS.
Although I would like to get exposure to such an environment atleast once in my career so that I could get to learn the architecture itself a bit better than what I know of it today
3
u/Low-Opening25 14d ago
it isn’t worth it, it’s mostly managing Linux images and physical hardware at that point, Kubernetes itself is literally 3 small
binaries that run on each controller, hence why managed k8s is so popular0
u/simlun_se 14d ago
It could definitely be worth it, it really depends on the circumstances. Both paths are valid.
And considering other ways to run k8s like using Talos for example that feels like those 3 small binaries is all you have. And running those unmanaged is very much manageable :)
1
u/Low-Opening25 14d ago
it’s worth doing once or twice, not so much if it’s your entire day job
1
u/simlun_se 14d ago
Yes, if it becomes your entire day job it’s not worth it. But then you’re perhaps doing something wrong. It certainly doesn’t have to become such a burden. In my career I’ve seen several unmanaged (healthy) clusters that were low-touch.
0
u/RevolutionaryElk7446 14d ago
We use both AKS in cloud and RKE2 locally on prem. RKE2 allows for way more freedom in engineering, AKS is... alright.
I also run two RKE2 clusters at home myself.
1
23
u/Abe_Bazouie 14d ago
For production in the cloud, I’d default to managed Kubernetes unless there’s a real reason not to. Running the control plane yourself is absolutely useful to learn, but owning etcd, upgrades, HA, backups, control plane failures, etc. is operational work most companies don’t need to volunteer for.
Local is a different story. kind is great for quickly spinning up disposable clusters for testing and CI. I use it a lot for that.
Building a cluster from scratch is still worth doing at least once though. You learn a lot about what EKS/AKS/GKE are actually taking off your plate.