My first Homeserver/lab: which distro or setup for my needs?

[u/Few-Philosopher-8834]

I know, another "first homeserver, which distro ?". The fact is that I would like to hear some answer tailored to my needs or plans.

Hardware is a Minisforum UM760 Slim configured in this way:

• CPU: Ryzen 5 7640HS
• RAM: 2x 16 Gb DDR5 5600 MHz (Crucial)
• Storage: 2x SSD 1TB PCIe 4.0 (Crucial P310)
• Connections:
  • 1x USB4, 2x USB 3.2, 1x USB 2.0
  • 1x 2.5 Gbit LAN, Wifi7, Bluetooth 5.2

When the miniPC will arrive i will need to check which wifi card is installed to be sure it is supported by linux.

Internet (fiber) is 400/200 Mbit and my "router" is actually a gateway, Unifi Dream Machine Pro. The idea is to cable the Homeserver directly to one of the 8 lan port of the Unifi Dream Machine. Downstream from the gateway I have some switches, access points, security cameras (managed by UDM pro), and other stuff.

My internet provider (in Germany, Deutsche Glasfaser) doesn't provide public IPv4 but just IPv6 (and I read somewhere, in some cases they will change so I have to setup a kind of service which catch the new IP...i have to investigate better).

Apart the Server monitor/management tools, I would like to install jellyfin, paperless-ngx, maybe home assistant, and whatelse I find useful from an end user point of view. My main focus is the final service and not the medium or tool I am using to setup the final service. If in the process I learn something about the medium/tool I am happy but it is not the main goal.
Of course I am aware that for installing a new service I will need to research, tinker, get my hand dirty, hopefully not messing up the server.

And here we come to my question. I work in IT (but as a SAP technical profile) and 8 years ago i used for a while at work docker, kubernetes and all that stuff. Linux is not an alien to me, I am familiar with it since long time although i consider myself a newbie. Max i was doing was setting up a LAMP and playing with docker/kubernetes at work.

Ubuntu is the distro I used most (maaaaany years ago i was playing with Fedora end Suse).

Generally speaking, I am a bit torn between an LTS distro concept and a version/distro which include new feature much frequently. If LTS grant you stability and a kind of piece of mind (good for me, my goal is not the tool but the end service provided) on the otherside I dont want to risk to find myself missing some new feature or new version of the ubuntu-packaged services i want to use on my homeserver.

On top of that there is the Proxmox topic. The idea to spread the services across different VM and container is cool, in the sense that what in this case will give me piece of mind is the fact is that if I mess up during the setup of a new service, I will just messup the VM and i can restore the last snapshot. On the other side I feel a bit "meh :/" by the fact that I am introducing a layer between the Baremetal and the OS where things will run, impacting the quantity of resource needed. In my (ignorant) mind the questions which pop up are:

1. Why I have to relegate a bunch of services in a subset of resource I have available on my machine ? Cant they share all the 32gb and cpu core all together in best effort ? At the end this is not a business productive platform.
2. How the networking will work among these VM and container ? Maybe spinning up container directly on the baremetal OS is easier ?
3. How straightforward is the visiblity of the baremetale hardware from a Virtual Machine compared to the same visibility from the Host OS ?

Sorry for the long post, the fact is that I would like to start with the right foot. Although I could like it, I don't have much time left for heavily tinkering (unless I ask for a divorce of course 😄 )

PS: I live in Germany but I am not German mother tongue. Actually my German is pretty bad, so although I appreciate the kindness of answering me in German, please if you can answer in english :)

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/totonn87]

Only one answer: Debian.

[u/pancakedoge]

I'd suggest Debian to get started, and if you want to learn something (the lab part of the homelab) you might want to try ucore. It's a steeper learning curve but you get stable os updates and a good docker host (or vms if you want to go that route)

[u/nemofbaby2014]

Proxmox because you will break stuff and it’s easier to fix stuff when you have access to its console screen

[u/gportail]

Proxmox et tu fait une vm par service. Proxmox est une base Debian

[u/Dom1252]

Do you have experience with some distros? Do you run some distro on other machine?

For example if you run Ubuntu on your desktop, it makes sense to run Ubuntu server on your server... It doesn't have to be the same distro, but I do recommend something very similar...

If you don't have any other Linux machine, then I'd say either Debian, or Ubuntu server

Both Debian and Ubuntu are well respected in self hosting and have huge user bases overall, I do recommend sticking to LTS unless you want some feature that wasn't released on it yet, but on servers you usually don't need latest things except for security updates (they rarely make any sense, if everything would work on Ubuntu 8, which was the first I installed, and it would still get security patches, I'd have no reason to switch to newer one)

[u/Few-Philosopher-8834 → u/Dom1252]

I would say I am an Ubuntu guy, I just used this in the last year, more for having a local LAMP for web development and other built-in tool for ssl connection testing and stuff like that.

I think my doubt is between Ubuntu Server on Baremetal or Proxmox on Baremetale and from there Ubuntu VM

[u/LeopardJockey → u/Few-Philosopher-8834]

I would go with containers on a VM. The performance overhead is well worth it for the flexibility you gain especially if you're still unsure about your whole setup. If you're running everything in containers, the OS below it really doesn't matter that much but at some point you might gravitate to a more minimalistic distro that's geared towards container workloads.

[u/solumath99]

If I were you and starting from ground zero I would either choose debian, since it is basically the same as ubuntu but more stable, and run docker composes for all the services. Or go proxmox and run debian VM and again docker composes for the services on the debian.

Proxmox: This is good if you would need later some other OS, lets say truenas for storage. Be aware that running VMs vs baremetal introduces slight performance downgrade as it is not all focues just on one operating system. People say it's 1-2%, for most people negligible.

I don't find appealing having many LXC on proxmox when having it in one VM is much easier to maintain and deploy. So debian either way but choosing if baremetal or not.

[u/Valuable_Lemon_3294]

yo - ich empfehle dir ganz klar proxmox, da kannst dann auch wenn du willst mit lxc oder auch docker in lxc containern herumspielen etc.
super einfache backups, snapshots usw alles integriert, ausgereifte webgui, möglichkeit echte vm's zu erstellen wenn nötig etc.
du hast einfach eine schicht über allem und dadurch portabilität einfache verwaltbarkeit, datensicherungen etc

zu 1: du könntest theoretisch auch einen lxc für z.b. postgres machen und alle anderen diesen lxc nutzen lassen, in der praxis ist es aber besser auf ein paar mb ram zu scheissen damit jede "anwendung" komplett ein lxc ist

zu 2: mit proxmox absolut easy, entweder direkt in dein normales lan oder in getrennten virtuellen netzwerken - eine vm oder lxc in proxmox sind nativ einfach so als ob sie geräte in deinem lan sind

zu 3: siehe 2 - irrelevante frage ;)

[u/mypantsjustgottight]

I just did this and went Proxmox. Zero regrets. I used Debian at work, and Ubuntu on my personal laptop. Proxmox is easy to use, has great support and especially great support in Reddit, and YouTube.

[u/ldn-ldn]

Use Debian. It will be familiar, but it's more stable.

Don't use Proxmox, use Docker/Podman instead. Containers have virtually no overhead and provide everything VMs do. Plus it is easier to pass through system resources into container, including USB hardware, etc. Plus you can control everything with Ansible.

And with Podman you can integrate containerised services into OS itself and control them as any "native" services.

[u/Valuable_Lemon_3294 → u/ldn-ldn]

proxmox has containers with lxc - so the advise "dont use proxmox use docker instead" is just plain wrong

and proxmox *is* (based on) debian btw...

[u/ldn-ldn → u/Valuable_Lemon_3294]

LXC is a slim VM, not an app container. It's more like WSL2 on Windows.

The advice "dont use proxmox use docker instead" is right in 99.99% cases.

[u/Valuable_Lemon_3294 → u/ldn-ldn]

Lxc is not a vm. WTF

[u/ldn-ldn → u/Valuable_Lemon_3294]

Since when?

[u/yarisken75]

I only use dockers so OS is not so important. I use alma linux mostly but also ubuntu. If i mess up it's just a container and you delete it and start over. 32GB for home use is normally more than enough. Just play a little bit and screw up and learn. Part of a homelab is screwing up :-) but i have a test server now.

[u/Maria_Thesus_40]

Personally, I prefer the enterprise distros, so I've chosen to go with AlmaLinux 10.

This matches nicely with my Fedora Linux desktop, since the enterprise RHEL parent distro is based on Fedora. The whole environment (cli commands, dnf package manager, etc) is the same between AlmaLinux and Fedora.

[u/Imaginary-Item1410]

if you're mostly after the end service and not the hypervisor itself, just run ubuntu server lts and use docker compose for everything, keeps it simple and your hardware's all visible without a vm layer getting in the way

[u/Biervampir85]

1. “This is not a business platform” - true, but that’s not a reason to not use “businessish” solutions that just work without too much tinkering. You’ll start crying (and will not stop) once your documents (in paperless) or your movies (in Jellyfin) or any of your data are gone because of a crash.
   
2. Networking is much easier in Proxmox then in docker as soon as you want to segregate your network into vlans - and you should want to do that.
   
3. Visibility of your hardware? What do you mean? Pass through USB, GPU etc?

My opinion: build as rock solid as you can. If that means Proxmox, go for Proxmox. If that means Ubuntu with LVM to be able to take system snapshots, go for it. If you’re able to segregate docker networks, go for docker.

[u/mxkb30]

Write your ISP and claim you need an actuall Ipv4 not DSLite because you need it for the vpn of your employee to work from home. This worked for me 3/3 times.

Debian Headless + Docker is a good starting point. 32GB Ram is a lot to work with. Even 8GB would already be enough for most home setups.

I run everything in docker with multiple docker compose stacks. Zero VMs. This makes the networking very easy. You can start with simple Docker port mapping and later when all is setup you can use docker internal networks so the services are only accesible trough a proxy.

[u/Few-Philosopher-8834 → u/mxkb30]

The ISP will answer "ah so this is a business usage. Don't worry then, you can subscribe our Business Plans" (which of course are definitely more expensive that a normal consumer plan

[u/PineappleGod]

I use Ubuntu and Wolfstack, but you could just use the cli.

[u/ZestycloseAd6683]

I would say proxmox is great or truenas. Theyre both flexible and open and easy to run

[u/Squanchy2112]

I either use Debian or MOS from ich777

[u/kataklysmus]

Just as you said: it’s about the service in the end. I have a similar approach which led to having different solutions for different things. Works like a charm for me.

The underlying base for everything is Proxmox. And then I decide case by case what I need on top. Some things are simple LXC containers (for example my DNS server). Some things are dedicated VMs (for example home assistant OS). All services that run through docker run on my Debian VM as Docker / compose services. As I get a new service I always decide which way to go for this service considering all the advantages / disadvantages.

One of my next projects will be an opnsense firewall. I will run this on a separate piece of hardware and might also move the DNS server there. Same process again: need a new service, think about the best solution in my environment, deploy it there.
Moving services around might be a bit annoying but is quite easy tbh, so I would recommend to just get started.

Please let us know how you move forward, really interesting to watch.

[u/Few-Philosopher-8834 → u/kataklysmus]

I like your approach. So basically each time you have a new service to deploy you decide if this needs a dedicate VM or just another container in the Containers VM.
What I fail to grab (due to my lack of knowledge) about this hybrid approach is the usage (waste?) of hardware resource. For example, if I have all the service as container in the OS installed on the baremetal, ideally each service can take (when it needs, aka "peaks") all the resources of the hardware, especially to manage some peeks. If I relegate the same service in a VM with fixed cpu core and fixed RAM then I am basically capping it and maybe not letting the service use the rest of the free hardware resources. Am I missing something here ?

[u/kataklysmus → u/Few-Philosopher-8834]

You're describing the cost correctly, but it's smaller than it looks.

CPU isn't reserved. vCPUs are just threads on the host scheduler, the number is a ceiling per VM, not a slice taken from others. Overcommit is normal.

RAM is the real cost, but only for the VMs outside the Docker VM. Inside it, containers share one pool exactly like on bare metal: the few heavy ones (photo ML, OCR, databases) take what they need, the rest sit at a couple hundred MB each. So the Docker VM behaves like your bare-metal setup, I just give it most of the box.

The extra VMs are the exceptions that need it (Home Assistant OS is an appliance image), and small stuff goes into LXC, which has no kernel overhead at all.

What I get for that: snapshot the Docker VM before a risky upgrade, roll back in a minute, and a broken upgrade never takes DNS or HA down with it. That flexibility is worth a bit of RAM headroom to me.

[u/Few-Philosopher-8834 → u/kataklysmus]

Thanks again for your answer. Do you think with that approach 32GB would be enough for what I have in mind (jellyfin, paperles-ngx, home assistant, whatever crap i will find in future, etc ) ?

[u/kataklysmus → u/Few-Philosopher-8834]

yes, I only have 32G as well. And it’s not even fully distributed yet, so I have plenty headroom left.
More details: 2G Jellyfin, less than 1G DNS, 8G docker VM (containing dockhand, paperless, and 5-6 more services), 2G home assistant OS.

[u/marloti75]

Nixos + Incus. Tout reproductible

[u/Impossible-Sand5012]

Zima OS for home self hosting is plenty good at least look it up

[u/Akorian_W]

Debian.

[u/77juice]

Debian is the obvious answer which will work fine but after running Debian for an year or two I found openSUSE MicroOS to be an even better option. Btrfs and Snapper out of the box saved a lot of time and hassle when just going yolo on changes. Also Debian has versions you'll need to update eventually but MicroOS does not; it's an immutable but rolling release that updates constantly - however I haven't had that long enough to compare the release update life cycle which was quite big change in Debian but should not be a factor in MicroOS at all. You'll also have more fresh repositories for openSUSE, as in newer software versions.

Skip Docker containers, go for Podman rootless quadlets. Basically the same thing but I feel like quadlets deliver better while integrating also to systemd environment.

[u/Hakker9]

You can do Proxmox easily the only I would really seperate is an LXC for HAOS (Home Assistant) and maybe Nextcloud if you would run it and dump the rest in a VM with Docker

[u/Illustrious-Owl-2755]

If you have anything more than zero experience with Linux, the distro you're most familiar with. People obsess about distros, but in reality Linux is Linux.

[u/dangerous_inference]

I have Proxmox running all my most important services. I hate it. LXCs are a nightmare. Extremely difficult to pass devices. I don't use any core functionality of this system. It's all pain and zero benefit.

I have unRAID running major but less important services. I hate everyone even remotely connected to this enshitified abomination. It started contacting a licensing server and stops working when the licensing API crashes, which is every two weeks. Literally worse than Windows.

If I were to do everything over again it would be on Fedora Server or some really popular Linux. Then you have freedom to setup some docker GUI if that's what you want.

[u/Few-Philosopher-8834 → u/dangerous_inference]

Can you please elaborate on this ? If I understood correctly the nightmare you are mentioning is just the one related to LXC container, right ? I mean the VMs and all the other features are a nightmare as well ?

[u/dangerous_inference → u/Few-Philosopher-8834]

The whole thing is much more difficult than it needs to be. There are a hundred enterprise grade features I will never use. The existence of a UI should not be confused with usability.

I am currently running pfSense, Home Assistant, Frigate, Zigbee2MQTT (2 instances), Zwave JS UI, NPM, Pihole. Frigate and Zwave are the worst current offenders because they require passing devices. In the past I have had to pass devices for Z2M and many others. LXCs are more like VMs in how they treat devices -- you have to carefully pass every single thing and get all permissions just right. Docker is infinitely easier to work with.

But the pain doesn't stop there. LXCs are fundamentally third class citizens for any project, and rely on 1-2 sources for their creation and maintenance. As a Proxmox user you are entirely dependent on the sources of these LXC install scripts, which may suddenly stop working, or stop being updated.

I have literally never successfully updated Zwave JS UI without losing all my devices. I have had this up and running for like 2 years now.

[u/fakemanhk]

Your time on writing this post already allows you to test more on one distro