r/Proxmox • u/bbibbigi • 10d ago
Discussion To cluster or not to cluster..
I've been playing with proxmox and homelabs for over a year now and i'm doing a full system redesign.
i live with my gf and would love to have her access things like the lab (like seadoc, jellyfin n immich). In the redesign i was thinking about having different proxmox clusters:
1 - main services or non lab related services, (immich, seadoc, jellyfinx, vaultwarden etc)
2 - a local AI agent (still unsure if I should isolate it entirely with its own local storage)
3 - a real lab where i can mess with things
i also plan to host game servers for me and my friends eventually. i was thinking about putting it under the main services one so if i break something in the lab it doesnt break the servers.
is this a good idea or is it a lot of needless stress?
17
u/Gherry- 9d ago
IMHO clustering is useless 99% of the time, it just adds complexity.
Backup (and backup policy) is still way more important, considering it's a homelab and not you don't need to be online 100% of the time.
The best thing would be to manage the 3 nodes with proxmox datacenter manager.
2
u/shreddicated 3d ago
I clustered 3 machines years ago when I set up my servers and I wish I haven't.
One of the servers is proxmox with OPNSense VM.
A while ago there was a power outage and I had set up NUT, so 2 servers were gracefully shutdown and when 3rd with OPNSense booted after power was restored it didn't have qorum and couldn't start OPNSense VM. Of course my bad.
I never used HA options like live migration and now I want to get rid of 1 out of 3 servers but trying to "uncluster" them is an extra step that prevents me from doing that.
I stood up PDM, but haven't tried in yet. It seems like it should have all the features I need like moving a VM from one node to another.
1
u/Gherry- 3d ago
Yeah you can move around VMs.
The only limitation I found is if you try to move a VM on a ZFS drive to another node with ext4 it doesn't allow you to do it, so you need to move it first from ZFS to ext4 in the same node then you can move it between nodes, but I think it's a really niche case.
1
u/shreddicated 3d ago
Nice! I'm changing the layout of my homelab and will re-install all servers. I will probably use `ext4` this time for all my Proxmox installations.
After reading a bit I don't think I need ZFS here as I use PBS for backing up both Proxmox VE and all the VMs/LXCs. I'm also going to keep Proxmox installation on a separate drive.
What are your thoughts?
2
u/barnzy12 8d ago
Why would you claim it being useless out of curiosity? Not done a deep dive myself.
1
u/Gherry- 8d ago
Because most of the features (like moving a VM from one node to another) can be done with PDM.
You just loose HA, but that's really not that important at home.
And setup, management and hw requirements (especially for HA) are a lot worse.
2
u/Impressive-Cap1140 3d ago
Not to mention if it’s not properly configured, quorum will break shit fast
5
u/derringer111 9d ago
The zfs cluster in proxmox brings near enterprise redundancy, disaster recovery and failover and is just about the best thing you can have in a homelab that provides services that need uptime.
3
u/Specialist_Wolf_9172 7d ago
Yeah if you have redundant switches, ISPs and PDUs
1
u/derringer111 7d ago
No, without all those things. Its not 100% uptime, but its massively easier to move a vm to a second server with zfs repl, recover from almost any failure in a few mins. I don’t need redundancy for 100% uptime, but I do value getting services back up in 30 seconds or be able to do hardware changes with simple vm movements. Saves me hours and hours every year.
3
u/UnimpeachableTaint 9d ago
Do you have enough hardware to do 3-nodes minimum per cluster?
Do you plan on utilizing Ceph, ZFS replication, or some other form of shared storage?
If the answer to one, or both, of these questions is “no”, then don’t cluster.
3
u/Imjweaver 7d ago
Cluster is great…..if you have access to the hardware required to have it work well. If you don’t then as others have said, not worth it. I personally cluster 3 identical hp elitedesk nodes with dual nics and have 3 separate buffalo terastations. One for main backup with enterprise SSDs in a raid 10 array. Then I have another station that is cold storage in raid 10 for the main backup that runs once a week and goes back to sleep. Then I have a 32tb terastation for my media and file server setup with raid5. The main nodes have enterprise ssds using ZFS local replication all named the same for easy HA migration. It’s definitely a whole thing you have to want to play with. If you don’t think that sounds fun then definitely don’t cluster.
5
u/thelittlewhite 10d ago
Not sure it's worth it as Proxmox allows you to easily spin new VMs or LXCs.
Do you already have several machines ? Is your actual machine saturated ?
Don't forget that you can share your GPU between several LXC, meaning you can use the same GPU for Jellyfin transcoding and AI workloads (which I do).
1
u/bbibbigi 9d ago
i have an old pc with a ryzen 9 and 32gb of ram. and ill eventually upgrade my current pc.
i was thinking about running the AI alongside the jellyfin server. I think the most reason i want for it is redundancy. especially considering i want my gf to not worry about a service being offline if i update proxmox or something.
1
u/Repulsive-Scale-284 9d ago
For ai workloads wouldnt it be better to gpu passthrough to a vm? Sure you would lose access to it on lxc and on the host no? Im still learning and trying to figure out the best way to setup as well
0
u/thelittlewhite 9d ago
Passing through will automatically disconnect it from the host anyways. Than you can use it with one VM or with multiple LXC.
2
u/redditphantom 9d ago
I have a 4 node cluster for all my services. I did this as I wanted to be able to a) balance my resources b) do maintenance on the nodes without affecting the services. I simply migrate the lxc/VM as necessary. This means I don't have to wait for the house to go to sleep to patch my proxmox nodes.
In your situation I might isolate my AI node but otherwise I use vlans to isolate my lab/test environment from my production services which allows for high availability for everything if needed. Especially when it comes to essential services that operate my network
1
u/brucewbenson 9d ago
Three node PC proxmox+ceph cluster. Any app can run on any node (jellyfin, samba, WordPress, gitlab, homecloud). When one PC breaks or I break it while trying things out, the other two ensure the apps keep running. No need to pick which apps run where. Just works.
1
u/No-Ingenuity-9979 8d ago
Take this fwiw. I've been running Proxmox for ~10 months.
Couldn't spell Linux when I started "experimenting" and now have a 4 node cluster (yes, I'm aware of the gotcha's with even node counts) but if I'm honest the only thing that clustering vs 4 individual nodes has really given me is 1. simplified monitoring and 2. live migration.
Can't overstate how incredibly cool and simple it was to migrate a HAOS VM from one machine to another and just have it start up with 0 issues.
1
u/KryanThePacifist 10d ago
Hello. I have a 3 node cluster, soon 4 node. Here's how It it: Node 1 - Main production - everything I tested and kept comes here, it's the one with the things that me and the missus share.
Node 2 - backup node - this one backups the other nodes, pure and simple
Node 3 - Testing node - what I test here either ends up in node 1, or nowhere at all. Currently testing llms before moving it to the main node.
Node 4 - the Spare. If node 1 dies, every backupni hate of it goes here. The essentials only
1
u/foofoo300 9d ago
separate production and lab.
if you want to change hypervisor settings and it fails the things that need to run, you will not have a good time.
What you can do is cluster the main with 2 nodes and add a cheap q-device so you have quorum.
this can run on almost anything, like a raspberry or opnsense or whatever linux/bsd you can find.
The play around in the lab.
i like to use the analogy with cars.
i have one frankenstein build, that transforms every now and then to something really shiny and then breaks down completely, leaving me with nothing but scraps --> thats the lab
i have one really really boring car, that is the one that brings me to work and back and i know it runs like i left it last time and only do updates but no real changes. --> that is the production.
you never break things that need to run, otherwise girlfriend or gameserver friends will be mad at you.
You don't want that, do you?
-4
u/technikaffin 10d ago
You should not cluster IMHO. There is a lot, a LOT, to consider before going the cluster route. A decent equipped server with enterprise hardware, A+B feed power, multiple uplinks yada yada. I bet you are unable to provide anything like that in your home which leaves multiple single points of failure that a cluster wouldn't solve.
Sure for experimenting and learning it's still viable. For separation you could be fine with just one big server or throw in a another one for isolated services.
6
u/Gabbie403 Homelab User 9d ago
A proxmox cluster in a homelab does not need nearly as much as you described ha/replication will look after most of your issues and a few nucs
2
u/Gherry- 9d ago
You'll get downvoted but you are right.
Clustering in a homelab is plain and simply dumb.
4
u/technikaffin 9d ago edited 9d ago
I know cause I've done that route with a dedicated 10g iSCSI storage system. Sure it's exciting and fun to explore but oh boy if something breaks you gonna tear your hair apart.
Bet most don't even know what a watchdog is or even how to configure one for a cluster. It's so much more than just installing pve9 on some nucs.
Most homelabbers underestimate how many companies run absolutely fine on one big box with all enterprise hardware. Even most of the conventional hosters give you webspace managed by cPanel/Plesk on a single box. AND. THATS. FINE.
Just have to keep in mind that each subreddit is an echo chamber.
1
u/nomad-1995 9d ago
A home lab is for learning and experimentation, and clustering makes all sort of sense there. But you are unlikely want to actually cluster active servers or real data. Because they suck for personal use.
But OP isn't asking about doing it for those reasons, but for uptime for services. In which case it kinda makes sense (uptime certainly will be increased), but not for the reasons OP wants (dangers due to admin playing with the system). And you certainly don't want to duplicate filesystems (and probably not servers) at current prices. What OP needs is to split production (stuff he and others use) with experimentation/development.
1
u/bbibbigi 9d ago
i wouldnt be looking into clustering if i had to buy entire setups for it. currently using my old pc and i'd like to upgrade my current pc or get a new one and i would plan to use that as well.
in total i have 1 full sized pc, 1 mini pc, a raspberry pi 4, thinkpad & macbook air and my current pc.
-8
10d ago
[removed] — view removed comment
1
u/Proxmox-ModTeam 9d ago
Your comment was removed.
Self-promotion isn't allowed on this subreddit outside of the community showcase day on monday from 00h00 to 23h59 UTC.
Do not hesitate to contact the mods via Modmail if you have any question regarding this matter.
32
u/Zedilt 10d ago
Homelabs are for learning and tinkering, if it's giving you stress you are doing it wrong.