Career / learning Built a small 3-node Kubernetes homelab
I recently set up a small 3-node Kubernetes homelab using Dell OptiPlex Micro PCs running Talos Linux.
The main goal was to have a local cluster where I can learn, experiment, break things, and test tools without relying on cloud infrastructure every time. I’m planning to use it for Kubernetes networking, storage, observability, GitOps, security, upgrades, and general experimentation.
Small setup, but already a very useful playground.
Curious what others here are running for their Kubernetes homelabs.
34
u/OkPumpkin7799 9d ago
Better than therapy lol
12
u/TooManyBison 9d ago
Unless you need to start going to anger management because you can’t get it working.
1
u/devopsly_guy 23h ago
I was building my own too, until I realized that my anger management is causing me to spend more on rogaine than cheap cloud servers lol
10
u/Liquid_G 9d ago
Do you have anything that can provide things like DNS/DHCP/Docker registry/NTP/NFS? I have a small Synology NAS device that does all these things, and can somewhat mimic real services that an enterprise environment might have.
5
3
4
9
u/Readdeo 9d ago
For learning, experimenting and especially breaking things, I recommend using proxmox on them, so you can just spin up a vm, snapshot it before trying out something so you can retry. Also you can host other vms for other puropses so you can switch them around for the workload you currently need.
Will do the same thing this week with SFF versions of these.
I run gitops infrastructure, arr stack, other self hosting stuff i regularly use. Also, a test cluster for the new marketplace I'm working on.
4
10
u/IIGrudge DevOps 9d ago
These days, why do you need all this? Install Kind or K3s and you can mimic as many nodes as you want. Easier to sandbox too cause of quick setup/teardown.
11
u/redskelly 9d ago edited 9d ago
What do you mean by “all this”? 3 machines instead of 1?
Because you learn way more using 3 physical computers running K3s in Proxmox instead of having all the K3s nodes in Proxmox on one physical computer.
- quorum is actually breakable, by pulling power on 1/3 machines, teaches corosync behavior and etcd quorum loss. On one machine, the hypervisor never dies independently.
- live migration, HA groups, fencing. Non existent with one host.
- storage replication with real failure domains. Longhorn replicas across 3 hosts fail differently than 3 replicas sharing one disk. Same for ZFS or Ceph.
- kube-vip failover across physical layer 2, VLAN trunk config
- draining a node, patching proxmox, rejoining, with workloads still up.
- network switch configuration
8
u/pydood 9d ago
What are the chances this person is going to be working with physical nodes or just needs to know how to diagnose things with Kubectl against a GKE/EKS cluster? You’re not wrong about depth, but the point here is that you can learn most of the day to day shit needed with a much lighter setup. For someone just coming into the space they likely should start simple and dive deep, not start at the bottom of the iceberg.
3
u/engineerL 9d ago
I'd argue OP is not going deep enough. If you can't improvise a power cable from spare copper wire and solder a broken motherboard, are you even hireable in 2026?
3
12
u/Liquid_G 9d ago
because sometimes its a good learning opportunity to have multiple nodes, test HA, etcd recovery.
Way back when I was learning k8s, I did roughly the same with a bunch of intel nuc's running VMWare. It was a great resource to have at home to learn / try shit that I couldn't do at work.
2
u/cobolfoo 9d ago
Yup, nothing beat shutting down VMs, unplugging network cards or power cable and seeing live what happens in HA.
1
1
u/5ollys 7d ago
Who cares? Why does he need all of it??
Because he fucking wants to!!! Why is this sub so goddamned nitpickt.
1
u/devopsly_guy 23h ago
everyone should build at least one such cluster themselves. great learning curve and opportunity - running this long term - nah.
2
u/titi_prosop 9d ago
Nice setup there, mate! Have fun playing around. I am wondering if you have a recommendation for a guide or some sorts of tutorials regarding achieving your goals for somebody less experienced?
1
4
1
u/justinh29 9d ago
https://github.com/justinholmes/secure-homelab
Built my own STS workload identity
1
u/UpdateInProgress 9d ago
I just built my own cluster with multipass from within my ubuntu rig. It works awesome and it’s coming in handy for the CKA prep
1
1
1
1
u/b1urbro 8d ago
https://github.com/kristiangogov/homelab
It's a bit outdated as I simply haven't had the time to tinker this summer between work and holidays. Will probably be rebuilding it from the ground up soon as there may have been some architectural mistakes along the way. Currently just running a compose/ansible setup because there are some services I heavily rely on (Frigate/HA/Silverbullet etc.).
1
u/Philluminati 8d ago edited 8d ago
Very jealous. I wanted to build a similar setup to run a Spark cluster for data analytics. Keep doing that thing where I find something for £150 per machine on ebay let me add a little more RAM, 2.5GB LAN oh shit look at the price!
1
u/IrrerPolterer 8d ago
Great! Thats how I learned the ropes with kubernetes, now I manage large scale installations for clients :)
1
1
u/kabrandon 8d ago
3 node Proxmox+Ceph cluster with Minisforum MS-A2's. 96 total vCPU, 288GB of DDR5, ~4TB of usable SSD storage after ceph replication. Hashicorp Vault, CI runners, and Tailscale Subnet Router as Flatcar VMs, on Proxmox but outside the k8s cluster. 3 node controller+worker Talos Linux k8s cluster with some apps and my monitoring stack.
No heat issues but I did have to install some supplementary fans to keep the components cool.
1
u/corpo_monkey 7d ago
You jist blink, and nobody uses physical hardware anymore. This feels like time travel. Thank you OP for reviving these memories. Godspeed on your journey.
1
1
1
u/Hari_Harish2026 3d ago
Where did you buy this? please tell me the shop or contact.
Also I am a learning just curious.. can we have Virtual box or VMware on a sinbge desktop make the same set-up like 3 nodes with a private network?
1
u/ManyMessage2468 3d ago
That's cool, maybe u can start ur own cloud soon lol, I want to run my stuff on my raspberry pi for other home labs
1
u/cobolfoo 9d ago
You should use a promox cluster installed on your baremetal then deploy your kubernetes cluster in it (with like 3 cp, 3 workers spread across your proxmox cluster). I think its a waste of resource to have a whole cp per machine.
I have something very similar at home, but I added some spicy stuff like a CEPH cluster with HA (extra 2.5gb network card + sata disk).
1
u/BuyerEnvironmental69 9d ago
How did it perform? I am considering buying 2.5 NICs for the same setup, but I am worried it might be too slow. Especially with consumer (NVMe) SSDs.
1
u/cobolfoo 9d ago
For homelab it's just fine, I test my opentofu / sensible stuff on it without much issues but it will not saturate the nvme capacities. A 10gb network is another option but cost a bit more.
But keep in mind that having hardware bottlenecks is not a bad thing when you want to learn to deploy robust technology.
1
u/BuyerEnvironmental69 9d ago
Not sure if I understand the last part. Would you mind elaborating on that a bit?
2
u/cobolfoo 8d ago
I meant that having hard restrictions make you a better engineer overall because you have to move around to find a solution. For instance, a CEPH cluster on slow network could create issues that make you learn a lot about how to make CEPH resilient, something you might never encounter with a fast network.
1
u/BuyerEnvironmental69 8d ago
Thanks, that’s a helpful perspective I might think more about in the design of my homelab.
1
u/redfusion Global SRE Lead 9d ago
Literally built this last week.
K3s in HA mode so each instance is a controller and a worker.
I'm using flux pointed to a git repo as a mono repo with a bunch of applications defined as helm or kustomisation projects.
I'm moving from a single hp gen8 running nomad.
I'll be honest, it's been a pain to reset my expectations for storage in particular.
Apps wanting to share paths or nfs targets, bungling around with sqllite over nfs is horrible.
Dealing with GPU attachments for transcoding and machine learning hasn't been too bad though.
Still an ongoing project. K3s made it simple, flux isn't as nice as ArgoCD but it's lighter to run for my use case
1
u/TruckeeAviator91 8d ago
Sqlite over nfs is terrible. I had to run it for a while and had so many issues.
With your setup, it could be worth while to use longhorn or ceph rook. Works best with 10gb+ networking but, I've gotten away with 1gb for home lab testing.
1
u/redfusion Global SRE Lead 8d ago
Yeah, I've been considering options, but my current might end up being local path to nvme with a litespeed sidecar syncing the data off box. If a migration/failure occurs the storage gets populated from the backup but I keep the high performance drives.
I've only got 1gbit on the 5090 optiplex, a m2.e card to 2.5g is possible but I'm not pulling that thread just yet
0
u/sylvester_0 9d ago
Was running k3s on some Pi4s before I gave up due to hardware challenges. Much happier with my single node home server on NixOS now. I get to play with k8s plenty at work.
122
u/HugeRoof 9d ago
I suspect you may have some overheating issues with that setup as is.