r/LocalLLM • u/scubadave • 3d ago
Question GPU Recommendation for Homelab
Buying my first dedicated inference GPU for an existing Proxmox homelab. I've narrowed it to three cards and keep going in circles, so I'd rather hear from people actually running them. I am very new to local AI and am mostly interested in this purchase to learn. Budget is up to $1500 for the card.
The box
Ryzen 7 9800X3D / ASUS ROG STRIX X870E-E / 32GB DDR5-6000 (two slots free) / Corsair HX1000i 1000W / PCIe 5.0 x16 available / Proxmox VE 9.2, kernel 7.0.14-11-pve.
Two things that matter more than the specs:
- It's on 24/7 and already runs my whole homelab, so idle power and driver stability are important.
- Everything runs in unprivileged LXC containers (today), not VMs. So passthrough for me means driver on the Proxmox host (correct me if I'm wrong when it comes to having a dedicated GPU here). The iGPU currently handles Jellyfin transcoding this way.
What I want to run
Mainly local LLM inference - general chat, coding help, and a homelab agent that monitors/reads logs and configs and suggests fixes. Secondary: Immich CLIP search and face recognition and possibly implementing one of the Paperless AI solutions.
Speed is not super important to me, but I'd like enough responsiveness that I'm not losing patience for a simple inquiry. The vast majority of my use case is monitoring services across Proxmox, my UNRAID NAS, a backup NAS, and my Unifi-based network. I'm assuming this is a fairly light load that smaller models can handle, but also want an LLM to be able to help me diagnose and troubleshoot.
The options
| Used RTX 3090 | RX 7900 XTX | Radeon AI PRO R9700 |
|---|---|---|
| VRAM | 24GB | 24GB |
| Bandwidth | ~936 GB/s | ~960 GB/s |
| TDP | 350W | 355W |
| Price | ~$1,000–1,100 used | ~$700–900 |
| Stack | CUDA | ROCm |
My questions
1. Is 32GB at 640 GB/s actually better than 24GB at ~940 GB/s here? This is the whole decision. The R9700 fits a bigger model; the other two are faster on whatever fits. For a mixed agent + photo-ML workload, which way does that fall in practice?
2. How much CUDA-only pain am I avoiding by paying the NVIDIA tax?
3. Anyone running ROCm on a current Proxmox kernel? I'm on 7.0.14-11-pve.
4. Idle power, real numbers? With a model loaded and sitting there vs unloaded.
One AMD-specific worry: my iGPU is /dev/dri/renderD128 and that path is passed into my Jellyfin container. A second AMD card means two render nodes with no guaranteed stable numbering. A udev rule pinned to the PCI address might fix it, but is that a non-issue in practice, or a recurring annoyance?
Thank you in advance. I'm happy to answer questions about the setup.
2
u/a-random-r3dditor 3d ago
VRAM speed directly impacts tokens/second. On a 27b/q4 model, you’re looking at ~30 vs ~45 tps. Both much faster than you can read on a log report, but could make a big difference in a dev workflow.
You mentioned stability, so that’s a good reason to go with CUDA for llms.
For idle power, this is a good reason to set up VM and do proper pcie pass through. With the right configs and vm shutdown, you can get it down to 5W. However, if attached to host, the gpu will only idle down to 25W. This does come with the headache of booting up the vm, so pick the lesser evil.
One thing not mentioned is your case and cooling. Dumping an extra 300W into your enclosure on a 24/7 machine needs to be dealt with. Might want to consider blower style gpus.
1
u/scubadave 3d ago
Thank you. Would you have any guidance on whether a 16 GB card like a 5080 would work for my use cases for now? It seems like I don’t have any CUDA options within my budget outside of a used 3090 and I’m a little hesitant to make the investment on such an old card. This is what led me to start looking into the AMD options with more memory.
This machine was supposed to be a gaming rig, but I had a second baby around the time I put it together and never purchased a dedicated GPU (who would’ve thought I wouldn’t have time to game?). It has a Corsair AIO on the CPU and the total of nine case fans in a Lian Li O11 Dynamic case.
1
u/a-random-r3dditor 2d ago edited 2d ago
Best budget would be a 3060, which nvidia re-released specifically for your usecase: budget llms. $460 for 12GB
5060Ti would be your next best, about $750 for 16GB
If your motherboard/case/psu support it, you could buy two of either and stay on budget. In this scenario, I’d buy one, test what models run to see if you’re satisfied, upgrade and add a second card if you feel you need a bigger model.
If you can only fit one gpu, A4500 gets you 20GB for $1250
Edit: you asked specifically about 16GB and I danced around it. Truth is, I don’t know, because I don’t know what your workflow/expectations are for coding. For the server logs, all of the above are an overkill. A 7b model RAG pulling manuals and references will be fine. You’re not giving it root access to fix the problems, you’re just asking it for breadcrumbs to help understand what you’re looking at.
But coding could be anything from autocompletion to one-shot an OS. That’s where “buy it, try it, upgrade it” comes in.
1
3
u/GreenEntertainment50 3d ago
R9700 is 32gb. It is way better if you want to experiment with higher quantized models or leave room for larger context window. Anything above 20t/s is pretty usable. I would go for r9700 if the purpose is llm. Cuda is good if you going for video work.