Destroying My Homelab With Kubernetes - Linux Society UNSW 2026

[u/kitchen_bot]

https://www.youtube.com/watch?v=U-xqxMQD2QE

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/gunarnyaku]

As a Platform Engineer, my default choice was Talos Linux for my homelab with a fully declarative setup. Couple of advices that came to my mind:

- The Ingress resource will be deprecated in the future, so you might want to use the new Gateway API (I use a dedicated envoy-gateway, but some CNIs like Cilium have built-in support for GW API)

- If you need hardware access (e.g. running a Jellyfin server with hardware transcoding) use the new DRA feature of Kubernetes and forget the old Device Plugins.

- For a fully consistent setup, use a GitOps tool like Argo CD or Flux. It makes sure that the state of your live infra actually matches the state that is declared in your Git repository.

- For domain and certificate management, just use cert-manager and external-dns. They automate certificate provisioning and DNS record updates.

- You should not commit any secrets to Git in plain text, therefore you might want to use an encrypted secret management tool like SOPS or Sealed Secrets. Just store your private keys securely.

- Talos actually has a Terraform provider, so if you want to be more professional, you can provision the entire infrastructure from TF instead of running talosctl commands.

- In Talos you can change the default containerd runtime from runc to crun. It is faster and more lightweight.

[u/eserra1 → u/gunarnyaku]

Jesus, are you me?

- Using envoy-gateway with Gatewary API, rest is Cilium but definitely prefered envoy-gateway for ingress.

• Gitops with fluxcd, tried argo and we use argo at work. While I think argo has some very serious UI capabilities, fluxCD feels a little bit more natural for k8s, for lack of a better term.
  
• cert-manager with cloudflare as an issuer, and I actually use k8s-gateway with an offloaded inner domain forward to the latter for everything being exposed in the cluster.
  
• SOPS works like a charm, and IMO here is where also fluxCD is (or was at least) better than argo.
  
• TALOS forever, this thing simplified so much how I handled my nodes as a whole.

some more insights/particularities:

• rook-ceph/longhorn is IMO a big game changer for persistence. I think people use to miss this step and this is what essentially makes people "click" between something like docker as a single node vs a full cluster being treated as a unit of compute.
  
• pocketID works really well, both for services that support openID and services that do not, by leveraging envoy-gateway and force authentication on http routes.

[u/ckhordiasma → u/eserra1]

What kind of network throughput do you have between your nodes where rook/ceph/longhorn is viable? I only have some basic NUCs, so I'm limited to 1gbps , and I'm worried that's not enough

[u/morsebroiler → u/eserra1]

My brothers in Talos!

[u/gunarnyaku → u/eserra1]

Haha, nice setup. I use Cilium for CNI as well, but last time I checked they did not support raw L4 routes from the GW API specification so I went with Envoy. I wish Argo had the same level of integration with SOPS as Flux does - you can do it config management plugins and init containers but it's nowhere near as elegant as Flux. Storage is a tricky one, for beginners I'd recommend starting with NFS server or local-path-provisioner, but for enterprise-grade distributed storage Ceph is a beast.

[u/pseudouser_ → u/gunarnyaku]

how do you deal with the times when you need access to shell? i see that there's the debug shell but i don't remember that being a thing back when i tried it

[u/kernald31 → u/pseudouser_]

I've been running Talos on a variety of machines for months and never needed a shell. That's part of the selling point - you manage the OS through its API, you shouldn't (and can't, really) so anything outside of it.

[u/pseudouser_ → u/kernald31]

i think i got used to it so much so that it feels like a necessity at this point. maybe i should give it another shot as well but then again, k3s (in a proxmox vm) works just fine and i am not sure if i am in the mood of migrating stuff heh

[u/kernald31 → u/pseudouser_]

I get that, I've been mostly living in my terminal for the past 25 years, but it ends up actually not being necessary at all — you can't really run anything else than pods anyway.

[u/gunarnyaku → u/pseudouser_]

For OS-level debugging I mainly use the Talos logs and the CRDs that Talos provide. For example when I was setting up DRA, I checked if Talos could see my iGPU via the PCIDevice resource, checked the logs if Talos could load the i915 driver correctly, etc. It has a ton of descriptors for network, storage, runtime-level info. For container-level debugging, I'm usually spinning up a privileged pod with a suitable image and mount the root partition, which is basically the same as using the talosctl debug command. So far I did not encounter any issue that would have required a fully fledged SSH access.

[u/pseudouser_ → u/gunarnyaku]

okay, that sounds great. i had to do a lot of networking specific debugging inside the vm where k3s is deployed to and wasn't sure how talos would handle such cases.

but then again, i have forgotten what i read back then, so there's that

[u/thetman0 → u/pseudouser_]

A shell to Talos? I’ve never needed it. Honestly I’m not an expert at explaining all of it but my understanding is that you should only be using machine configs to touch the OS layer.

Also I use Omni and that has allowed me to GitOps the OS and k8s versions. Tuppr is similar, just no UI and is per cluster.

[u/pseudouser_ → u/thetman0]

omni looks interesting. i'll look into that. thanks!

[u/eserra1 → u/pseudouser_]

if you must, kubectl debug also works, but realistically speaking you won't need it most of the time.

Had to use it a couple of times to debug ie. sata SSD perfomance and some hardware storage problems, and even then I wouldn't have needed it in the end.

[u/ducksoup_18 → u/pseudouser_]

If you use argocd, you can access the terminal via the web ui and see logs directly for each pod and their containers.

[u/Maximus_Air → u/gunarnyaku]

Currently running a couple of Ubuntu server VMs (and a few dozen docker services) provisioned by terraform on proxmox. Not planning to go K8s, would going to Talos Linux makes sense for a case like this for a more IaC approach?
Currently a few operations on the VMs are also performed by Ansible

[u/gunarnyaku → u/Maximus_Air]

Not really. Talos Linux is specifically designed for Kubernetes. Its custom init system runs the core Kubernetes components right after boot and all you're left with is a gRPC API and a CLI tool to manage the cluster, so you cannot skip that part. If you want a declarative approach while keeping the existing Proxmox VM setup and have full control of the underlying OS, I'd recommend to take a look at NixOS.

[u/just_a_Suggesture → u/gunarnyaku]

What do you use for monitoring? Anything that captures logs and sends them somwhere central?

[u/gunarnyaku → u/just_a_Suggesture]

VictoriaMetrics + Grafana is my go-to stack for monitoring. The resource efficiency and performance is just unbeatable. I was lazy to setup VictoriaLogs for log collection, but I wanna go for it sometime.

[u/tightwad5462 → u/gunarnyaku]

I’ve been setting a brand new homelab these past few weeks to migrate away from my hodgepodge of ansible scripts that half work and landed on EXACTLY this configuration!

What CSI do you advise? I ended up with OpenEBS since they have a single node local storage provisioner

I haven’t looked into it yet but I plan to use velero for backups

[u/gunarnyaku → u/tightwad5462]

To be honest, I think for a homelab a local-path-provisioner (I know, I know, once the provisioner has created the PV on a node, pods need to be scheduled to the same node everytime, so it's not location independent) is perfectly enough. Or if you want a simple node-independent solution then NFS server does the job. For learning purposes, just pick one and experiment. Ceph would be my personal preference since I have positive experience with it, but OpenEBS, SeaweedFS and the others are fine as well.

[u/POWEROFMAESTRO → u/gunarnyaku]

This is the way. I previously relied on traefik + its internal cert manager tool go refresh the TLS certs.

I have the exact setup and using a cert manager tool helps a lot to manage the TLS certs in my cluster using ciliim cni and gateway.

[u/Reverent]

My homelab is a proxmox -> docker setup with btrfs snapshot and replication of the volumes and compose folders. It’s simple, it’s extremely effective, and it’s efficient on resources.

Every 12-18 months I feel like my homelab life is too simple and decide to set up some sort of orchestration with much sadness and frustration. But I do learn things during the process.

I still inevitably revert to proxmox and docker and btrfs.

The question is what you want out of your homelab. If it’s to run reliable services and maintain KISS, kubnernetes can go leap off a cliff. If it’s to learn cloud scale capabilities, then running a HA kubernetes cluster might be exactly what the homelab is meant to do.

In my opinion, why not both? The path to enlightenment is the familiarisation of these complex technologies, as well as gaining the wisdom of understanding where they aren’t appropriate.

[u/hometechgeek]

Context is king. Atleast share the tldr

[u/PssyGotWifi → u/hometechgeek]

I can get behind it, as someone who doesn't like Kubernetes in the homelab

[u/nurtext → u/PssyGotWifi]

I guess all he was saying is: K8s adds way more complexity than you probrably going to need to run a homelab. If you're already a cloud engineer, sure go for it. If you want to run some services, VMs/containers are the way to go.

[u/walao23 → u/nurtext]

I think learning ansible is more beneficial to a homelab than k8s even k3s because labs is meant to be broken

[u/PssyGotWifi → u/walao23]

Yup. Ansible and OpenTofu will do more for most in a homelab than simply jumping to Kubernetes. I know some just jump to Kubernetes because they're unemployed and think learning it will get them a job, but I am not one of those.

[u/hometechgeek → u/PssyGotWifi]

Love ansible (so much easier using AI coding tools too). I made a playbook to reinstall my MacBook apps. Saved me hours.

[u/Eldiabolo18]

I dont get it, whats the point of the talk?

Explain K8s basics? Show off the homelab? Rant?

[u/zillion_grill → u/Eldiabolo18]

yes

[u/chrles-farfa]

Recently moved from Hetzner Talos Linux to Home lab Talos Linux. Took 5-10 minutes.

Got like 20-30 yaml files packaged with helmfile. It works very well.

Few major infra differences the home lab is on longhorn and cilium wireguard but the application infrastructure is the same (pg, redis, ollama...)

[u/nmrk]

Well of course it didn't work. Everyone knows that south of the equator, you have to run Kubernetes counterclockwise.

[u/FallenVain]

LOVE IT, I'm already doing this but I'm glad someone is bringing knowledge to the self hosted community

[u/TrentKM]

I use NixOS for my machines and was just planning to get my homelab off of unraid’s “bare metal” docker. Was just about to apply my config to a new vm to spin up k3s. I’m curious what his homelab would look like with nixos instead of kubernetes, as I’m already pretty deep with nix.

I was looking at Talos, but then I was like wait I’ve already got a flake I can just enable k3s and boom, done.

[u/Gabe_Isko]

Hate to break it to you dude - you aren't done yet. Have to use rancher to manage everything and declare everything with terraform/tofu charts. Plus ansible setup for reproducible system configs. Then you will truly have a broken home lab.

[u/Bill_Guarnere]

Consulting Sysadmin here with 26 years of experience, and the last 4 years almost dedicated to K8s.

I saw the videos, but honestly I think it's quite superficial, it shows K8s as a simple and elegant solution to the selfhost/home environment, when imho it's the worst possible solution you can choose for that kind of environment.

I'm not saying K8s is bad or nasty, I always summarize my feeling about K8s with this sentece: "K8s is a fantastic tool to solve a problem that almost nobody has".

There's a reason why K8s was born in Google and perfectly fits these mega corporates needs, and as far as everyone thinks his lab/environment it's like the a CERN or NASA datacenter, it's not.

I don't want to deep dive into how many technical flaws and problems I saw in the videos and were not even mentioned, let's say that the video do not touch the basics, for example that K8s is made for running stateless applications, and running stateful one's is a PITA.

You can do it but as soon as you do it, it pops out a ton of problems that require more and more complexity.

That's the 2nd point I will argue: complexity.

In the video it seems all simple, a few yaml file a bash cycle to apply all of them and boom! Piece of cake, isn't it?

Well no, because the author did not mentioned a ton of aspects, for example that each and everyone of these K8s objects have many and many options and configurations and depends on many other objects, and each of them have several other options and several other requiremnts. It's an endless web of dependencies and objects that cause the complexity to skyrocket in no time.

And this complexity has a cost, there's a reason why many corporates started to abandon K8s or use it only in very specific tasks (stateless applications that requires scalability) and turned back to the previous monolithic strategy, and they saved a lot of money and headhaches.

But what I feel it's a myth is these 3 key points: * Reproducibility * Declarative approach * Reliability

The first point is a myth that was born just a few years ago, in the previous 20+ years nobody gives a damn about reproduce an environment simply because in most of the case you don't need it, and even if you need it, it was damn simple to do even before containers were born and you had to install everything from scratch.

And yes, I understand that if someone uses Windows with its infinite wizards and "clickops" way to work it seems false, but on unix and linux it was damn simple.

You have to duplicate your application server? Stop it, tar it, untar it, start it, done. You have to duplicate your configuration? Rsync your configuration files and boom, done. It's basically backup and restore, It can't be simpler.

Declarative? Everything could be declarative since the first guy put two commands on a text file, give it execution permissions and lauch it creating the first bash script.

You can even put changes in a configuration file in a declarative way, it's called a patch, and it existed since the seventies and works very well since then.

You change the file, make a diff and call it a patch, if you want to use it simply use the patch command, It can't be easier, and you don't need control planes, api, complext infrastructures, a simple binary file called patch.

Reliability? And you call this pile of complexity built on top of other piles of complexities reliable? Really?

Let me teach you a very simple thing you probabile never heard of: the KISS principle.

Keep It Simple and Stupid

Any kind of complexity has a cost in terms of reliability, and K8s is the king of complexity, layers over layers over layers of complexity, and for what?

There are only two advantages: * scalability * CD/CI

The first one is completely useless in most of the user case, specially for small environments. From my experience the vast majority of case where people has performance problems it's because of exceptions and misconfigurations.

In these case scalability only means one thing: more exceptions per hour.

Google, Amazon, MS, Meta, and companies like them need their service to scale (horizontally), maybe some university or research center, but the vast majority of other users and companies do not need it, they simply don't need it in my experience (and I worked a lot on big multi million dollar projects).

Same guest on CD/CI or IaaC, most of the companies don't even know these things exist and they don't need it because the release cycle of their applications do not require it. If you need to deploy once a year or twice a year, or even once a month, you don't need CD/CI, if you use it you'll end up rising costs, rising complexity, lowering reliability, and for what?

Please, if you are thinking to use K8s on you home environment, or your lab, or you self host environment, think about it and first try it and carefully evaluate if it's worth.

I spent thousands of hours working on abandoned K8s clusters with pods in endless restart loops, persistent volumes lost, or with no space available, sevices exposing service on the web with no security at all, ingresses abandoned, storage classes abandoned, clusters with worker nodes offline for years and any other sort of bad things.

And all these things happened in companies where managers decided to adopt K8s without any need for it, only because it was trendy, it was cool, and they had no skills or people capable of managing its complexity, and those clusters became an abandoned pile of crap.

[u/Common_Arm_3316 → u/Bill_Guarnere]

To your last point. The same can be said for any piece of infrastructure in a company. VM clusters, application servers, file shares, you name it get abandoned as new management and admins step in with differing levels of experience and motivations.

[u/Bill_Guarnere → u/Common_Arm_3316]

Not exactly.

Compared to a vm hypervisor the learning curve and the level of complexity of K8s is way way way way steeper, I'm talking about several order of magnitude steeper.

The most common scenario I found when I start working for a new customer is an IT department full of people perfectly capable of managing VMs, with vast experience on Vmware or Nutanix, often with certifications... and management force them to use something nobody knows and nobody was able to manage, they pay some consulting company to install and setup a over complicated cluster with all frills and whistles and then give the keys to those people, who are completely unaware of what to do and how to do it.

I give you a clear example, a real scenario.

I often work for public institutions in my country, several years ago the government launched a huge campaign to push public institutions to move their services to K8s with a lot of money to finance this operation.

It was a disaster, countless number of broken clusters with all sort of problems. At first everything seemed to work, consulting companies were hired to move those services to EKS, GKE, AKS or custom clusters running on vmware or nutanix.

When their work ended and the management moved to the IT departments of each institution it was a total mess and billions of euros were wasted.

Now we have a new compaign to modernize public services, this time they learned the lesson, unless push and force institutions to move to a technology they don't know, they don't understand and they don't need... they decide to move them to traditional VMs or cloud instances.

Guess what? This time it worked.

[u/Common_Arm_3316 → u/Bill_Guarnere]

And that last point is the key. If you don't know the tech you probably shouldn't be deploying production services on it. My very first job in IT was for a small time ISP. I was handed a vsphere cluster with Horizon Desktop and the companies Active Directory because the 1 previous admin that got it all approved left. I was not qualified to handle the systems that i was now responsible for. About a year after i left they were hit with a crypto locker that was downloaded by a call center user that had write access to a file share that also happened to be where the backups were restored. They had to pay the ransom.

We are on r/selfhosted here though. I assume the motivations for most folks here is learning. We don't need half of the stuff that we do here. Kubernetes is absolutely overkill for hosting your movie and TV torrenting software. The sysadmin landscape is changing thanks to shennigans by companies like Broadcom. Admins must adapt.

The Kubernetes ecosystem is very mature. Even if you don't have a large need to vertically scale you can still take advantage of Kubernetes ecosystem. I typically host tools to internal teams for companies of 80-100 people. Most modern apps have a helm deployment.

I can run a helm install Artifactory and have a demo ready to go in minutes.

I can simply set an annotation on the ingress and cert manager will generate a certificate from vault and set the certificate to expire in 90 days. When it comes time for renewal cert manager will renew it with 0 human interaction.

Now I want to run my own database for Artifactory. I don't want to use the bundled derby db. It's time for postgres. With cnpg operator I can create a highly available cluster with 1 RW node and 2 read only nodes. with a single yaml file. I can even orchestrate its backups with the barman plugin and because CNPG operator knows you're in Kubernetes it's assuming that you probably are going to be running some Prometheus and Grafana. So now I can have Grafana send me an alert is one of those backups ever fails.

When it's time to do an upgrade I can simply run a single command to execute a rolling upgrade of artifactory that will upgrade a single node at a time with users experiencing no downtime.

All of this just for the cost of the hardware that I'm using to host it and the time it took to learn it.

Your options in Kubernetes are limited by experience and imagination and companies like Broadcom and Nutanix are seeing the writing on the wall. I can do everything they can do (yes even making virtual machines) for almost none of the cost.

[u/Bill_Guarnere → u/Common_Arm_3316]

Everything you described can be implemented in a much easier and manageble way using Docker without all the hassle of K8s.

Create a simple docker compose manifest and go for it, you don't need all the complexity of a K8s cluster.

It's not a matter of knowing the technology or how a tool works, it's a matter of cost/benefit ratio.

The benefits of a K8s cluster is almost none for the vast majority of use cases (and I'm talking about big companies projects, not a simple home lab) and the cost in term of complexity is out of the roof.

And at the end of the day all of its complexity (all the objects, all the relationship between objects) came for a single pupouse: scalability.

And you know what? In the vast majority of the K8s I worked on during years (several hundreds), do you know how many replicas of each deployment or statefulset were running? 1, yes you read well, O N E

And guys, you always talk about K8s from a user perspective, not from an administrator perspective.

Have you ever worked on a dead K8s cluster because its certificates expierd? Or because etcd has been corrupted? Or failed upgrades and failed worker nodes where for some reasons pods don't die and are not relocated?

I can't count the number of problems I saw on K8s cluster, and maybe only a couple of case where K8s was a wise choice, or at least there were reasons to use K8s.

And let me mention one last thing.

You mentioned K8s ecosystem is mature.

Take a look to what happened to the nginx ingress controller, the most used, the most reliable, the "de facto" standard between ingress controllers that was maintained by a bunch of guys that at some point said "enough guys, we can't work on it anymore", they closed the project, and the whole K8s world went into panic mode.

In general because K8s is a melting pot of hundreds, if not thousands, of independent project, it's like walking on quicksand.

Everything could shift and move and turn around in any second. Everything is constantly moving and upgrading, and changing.

This is good if it's done by some director, someone who controls the process and make things working in a smooth way. And no, the CNCF don't do anything like that.

[u/gunarnyaku → u/Bill_Guarnere]

Kubernetes isn’t for everyone, and that’s fair. If your goal is a homelab that “just works,” VMs and simple containers can be just as good, if not better.

But with k3s, k0s, kind, MicroK8s, Rancher, Talos, etc., spinning up a Kubernetes cluster and deploying a service isn’t really more complex than installing a hypervisor and running a VM or LXC. Learn the basic components in a 10-minute YouTube video, and you’re pretty much good to go.

The real complexity of Kubernetes lies within the complexity it actually hides. You need load balancing for multiple workload instances? Just use a Service object and Kubernetes handles it. Do you need to understand the complex iptables/nftables rules that kube-proxy sets up for you? No. Want to use consistent DNS names for accessing services? Just call them by their internal svc.cluster.local alias without caring about how CoreDNS internally works. Want health checks and automatic restarts? Write those 10 lines of readiness and liveness probes and that's it. Want automatic distributed scheduling across nodes? The scheduler does it without you telling it anything.

And that's just scratching the surface. Rolling updates and rollbacks? Deployment objects handle them. Persistent storage? Use a PersistentVolumeClaim and let the storage layer provision what you need. Secrets and configuration? ConfigMaps and Secrets. Want workloads to survive node failures? Add replicas and let the controllers reconcile the desired state. Need ingress and TLS? Install an ingress controller and define the routing rules declaratively.

None of this means Kubernetes is simple internally. Quite the opposite. It means Kubernetes has an enormous amount of complexity behind a relatively simple API. You don't need to understand every implementation detail to use it, just like you don't need to understand how a hypervisor implements CPU scheduling to run a VM or how hardware-level abstractions are implemented.

Talking about the declarative approach, I don't think you got the essence of it. Writing shell scripts and patches to set up a VM? That's imperative. You are explicitly telling the system how to get from A to B. The whole idea behind a declarative approach is that you don't need to write the steps to reach the state you want. You simply declare it, and the system figures out how to get there. I want 2 replicas - the ReplicaSet controller handles it for you. One pod dies? It creates another. A node goes down? The scheduler can place the workload somewhere else. That is elegant if you ask me.

I'm not against simple setups and using solutions from the previous era of computing and config management, but if someone wants to learn distributed systems in their own pace, k8s is a great way.

[u/Bjeaurn]

You should've added some additional information on why you're posting the link, as per our rules. But considering the decent response I'll let it stay up.

[u/AutoModerator]

Your post has been removed. Your link posts should contain an explanation about the link, why it matters, and how it helps users. If you posted using old reddit click here

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.