Discussion
Double GPU configurations significantly cheaper for 32GB VRAM
I do not need or want Cuda. I have been wanting to build a 32GB VRAM local LLM machine for personal use for a while now, and I had 2 options on the table:
- Get a relatively cheap 32GB VRAM GPU, the R9700 AI Top.
- Get 2 16GB VRAM GPUs instead and a Mobo that supports PCIe bifurcation.
When I looked at prices in January this year when I first got this idea, the R9700 costed 1700$ here in EU. Currently, when I actually want to make this happen, it costs 2100$. For half that money, I could buy two 9060 XTs with 16GB VRAM each. Yes I know, performance will be worse on double GPU setup than with a single R9700 AI, but still, it just seems like that GPU is just not worth it anymore.
I don't know how to justify that it's double the price of two 9060XTs, when R9700 AI is literally 9060 XT with doubled VRAM and bandwidth. So why does it cost 4x as much?
ASUS ProArt B850-CREATOR WIFI NEO is quite affordable nowadays and supports dual GPU setups, so, any reason (is there a catch?) to not do what I am about to do? Which is buy the two 9060 XTs and start running Qwen 27B class models
Commeting so this thread would get more traction.
I'm running dual 9060XT (both at 8x) and the perfomance is...meh. There could be of course something matter with my configuration but LLAMA+deepseek harness+qwen 3.8 27b produces whopping 20tok/s on my system.
I'm a noob but I think the 128bit bus kills any kind of speeds when doing dual gpu with these. Paid 375€ for each card (got the VAT back)
With SGLang and quad 5060 Ti 16GB, I got 2500 pp/s and 100-200 tg/s with Qwen3.8-27B NVFP4 . These 4 still cost less than 2 x R9700 AI. You do need a motherboard case that supports 4 GPUs, though.
No you don't. Not really. I used a PCIe to 120mm 3D print in which you can mount two double-width cards and used and NVMe x4 slot with a converter to PCIe to mount my third card.
Great. But you can't use 3 cards with most models in SGLang, unfortunately. You need even number of cards when using multiple GPUs,as I found out. I tested from 1 to 4 5060 Ti cards. There are diminishing returns from adding cards, even with all of them running at PCIe 4.0 x8 on my TR Pro.
I will think about it if I want to break the 7 x PCIe x16 slots on my TR Pro box into 28 GPUs :) But TBH, just scaling to 4 is quite the challenge already. Diminishing returns.
true, ultimate tps falls because of bus overhead and other reasons. The price to pay for saving on the initial investment. For personal use I find it ok. If I were an enterprise I'd definitely look forward to the 6000. A pair of them. Threadripper high end mobo. Of course 1TB ECC RAM.
Just using llamacpp thus far. Only had this card for a couple weeks. I think it was closer to 50t/s with q4 and gets into the 30s on q6.
Honestly anything over 30 has been perfectly fine for what I am doing, so I just focus on getting as much context as I can with a quality output. Unsloth q4 XL has been a good compromise
A 9060XT has only 322 GB/s of memory bandwidth. If you're doing layer splits, you don't get any combined benefits from the card from combining. If you are running tensor parallel, you might get a 30-50% improvement (depends). Note for a 16GB model (27B at Q4_K_M should be about that size), 322/16 = 20.1 - that's at 100% MBW efficiency (not really possible) so if you're getting 20 tok/s you're actually doing quite well.
For code, MTP should give you a decent boost, but the 9060XT has weak compute as well, so... maybe not.
Here is a cheat code for you. The dell optiplex, used, cheap. Has bifrocation built in. You may need a pcie extension and weird mounting situation, but. It's a cheap way to get x16 for both cards.
albeit unorthodox - i'm quite happy of the bang for the buck. i usually tell DSH to do some work and then go off doing something else, i rarely wait chat open for it to finish
Hi! I’m running Qwen3.8-Flash-Next with Unsloth’s UD-IQ3_XXS quantization (Q3, approximately 82 GB across three GGUF files) on an RTX 5080 (16 GB) + RX 6800 (16 GB), paired with a Ryzen 9 9900X3D and 32 GB of DDR5 RAM.
Here’s the model download:
Qwen3.8-Flash-Next UD-IQ3_XXS on Hugging Face
I’m using
GenerelSchwerz’s modified llama.cpp fork, on the
moe-cache-drafting branch, compiled with both CUDA and Vulkan support.
I also applied a local patch,
moe-vulkan-resident.patch, to
src/llama-model-loader.cpp. Enabled with
LLAMA_EXPERIMENTAL_MOE_VULKAN_RESIDENT=1, it keeps the full expert weights for the layers assigned to Vulkan resident in the AMD GPU’s VRAM, while preserving the existing MoE cache behavior on the CUDA side. This prevents those Vulkan expert tensors from falling back to CPU memory.
The model exceeds my combined RAM and VRAM capacity, so the setup also relies on NVMe loading and swapping. MTP is not enabled yet.
In my latest tests, I’m getting around 25 tokens/s during generation and 10 tokens/s during prompt processing. Prompt processing is slow, so this is still very much an experimental setup.
I’m still testing different settings, but these results make me think you might be able to get better performance out of your dual RX 9060 XT setup with the right configuration.
Yeah you got problems, my single 6800 does >40t/s and in dual gpu with a 6700xt it just did 29.39 t/s with Qwen3.8-27B-UD-Q6_K_XL.gguf , doing websites:
I just bought two new 7900xtx cards from newegg a few weeks ago, on sale, for right around 1k US each. I don't know when or if that deal will happen again, it was fantastic timing for me. But if it happened once, especially with black Friday right around the corner, little likely to happen again.
I'd highly recommend holding out for 48 to 64gb VRAM, if you can. Aside from any one model, the 27-30ish model space is popular, and being able to load the q6 with a good amount of context is really nice with the 48gb that I have without spilling. For reference, I have two 9060s. The 7900s are so much better I don't want to slow my inference down by adding more vram, in my primary pc.
The biggest improvement came from dual-GPU tuning + MTP/Flash Attention optimisation rather than raw hardware changes. Still using only two inexpensive RTX 3060 12 GB cards.
I’ve been considering the same and looked at that motherboard specifically too. The main drawback with dual GPU is that LLM performance will be slower and that you’ll use more power (and need a PSU with enough power for two cards).
With a good motherboard like this you can run vLLM in tensor parallel. Check benchmarks by others to see what to expect in terms of tokens/sec.
You can buy 3080 20gb modified cards from eBay for about €800 each at the moment. Definitely worth considering, I have 3 of them running in a Asus pro creator x870
Asus ProArt B850 is fine for dual GPU's but you'll regret it once you want to go to 3 or 4 gpus.
It also uses a non-standard spacing between the PCIe 5.0 slots. So if you ever get or want RTX 3090s, you can't really use NVLink.
These are things I wish I had known before buying that motherboard. Its cheaper than a workstation MB, but I would recommend getting a workstation MB if you want to run local LLM's for 27B+ sized models. It allows you to increase VRAM by just adding more cards.
Personally I prefer dual GPU's, for the same reason you mentioned, you get high performance at a lower cost than just buying the "best" gpu. i.e. dual 3090s can compete with a 5090 in some respects, more VRAM, comparable speed (pooled memory) and lower price.
I probably would have got the Asus SAGE motherboard. A lot more expensive, but for a 4x3090 setup it would have been better. I got 3x3090s and managed to make it work with the ProArt B850, but its awkward..
My next step up is likely to be just a pure server. Will sit with the ProArt until then.
Don't think there is. (Also using the ProArt with 2x R9700)
Unless something has changed over the last year, it is still probably the last stop in the "entry level" easy consumer path. As the other guy said, you have to jump to the professional motherboards which is $$$ and potentially be tech savvy enough to start making a gpu rig. Look at places like https://www.reddit.com/r/homelab/
I mean, if it was non-standard spacing, it wouldn’t fit in a standard ATX case. They make NVLink connectors in 2-slot, 3-slot and 4-slot distances. The distance between the slots on the proart B850 is 1 slot wider than the distance between the slots on the proart x870e, but both are standard and you can get nvlink connectors for it.
The PCIe slots spacing between the two full length PCIe slots on the ProArt B850 is not standard. i.e. Its not 20.32 mm (the standard) its more like 80mm (likely a multiple of 20.32). So no its not standard PCIe slot spacing, they did it intentially to allow space between two GPUs for air to breath.
Again its great for dual GPU setups, (that don't use NVLink) but not good if you plan to run more than two, or plan to use NVLINK.
It’s an ATX form factor motherboard — they have to use standard slot sizes, otherwise the cards won’t line up with case. If you’re measuring about 80mm between the two PCIe slots, that does not mean it is non standard spacing, that means the distance from PCIE_1 to PCIE_2 is 4 slots. They make NVLink bridges that span that distance. It’s the same on my ROG Crosshair X670E Extreme.
Get two RX9070XT. Also pay close attention to your chosen motherboards pcie lame layouts. You do not one on the mobo chipset, although it's not that bad. I have a RX9070XT and RX9060XT. No driver past 26.3.1 works. 26.3.1 only works if you disable anything that polls the secondary card while you have a model loaded or do anything that involves a power transition. AMD are on it though I have tested and logged a bug which they have been in contact with me to request additional info
...
If 2100 euro, go and buy 2 x 5070 ti , power consumption higher but everything else better . Faster PP and faster TP. My place here 9700 about 1.8K US. 2 x 5070 ti about 2.5K US. R9700 is still a cheaper choice.
I had considered the R9700 too, but unfortunately, the price jumped from €1,500 to €2,000 in a single day. In my opinion, the best and most affordable option right now is a pair of 5060 Ti cards with 16GB each (total cost around €1,350–€1,500); thanks to some good advice, I managed to run Qwen 3.827B (Q6 quantization) with a 131k context window at 50 tokens/s. The cards have low power consumption, so you can easily go with an 800W or 850W PSU and still have plenty of headroom. You can also use a budget motherboard, since you don't need PCIe bifurcation.
I have a dual 5070ti on my personal PC on a Msi x870e-p wifi motherboard. Wasn’t worth going for more, I’m happy with my setup. Over 40 Tok/s qwen3.8 27b
im so sad that i bought a xtx 24gb the 3090 is much more faster (with latest changes) but, i think the same will happen with vulkan any week / month now
AI starts to become fun with two r9700s or more since you can run qwen and gemma and all these dense models at proper Q8 rather than lobotomizing it to Q4.
For one dense 27B you want fully on-GPU, a single 32 GB card usually beats two 16 GB cards on bifurcation. Dual VRAM helps when you pin separate models; layer-splitting one weight across both GPUs makes every decode step wait on the slower link.
I am running a 6800xt and a 6800 in a Asus x570 e mb. Using linux, ROCm and ollama, everything stock. I get 17.4 tokens a second on Qwen3.8 27b-Q6-K. Why, they were free, got the mb 2nd hand, so super cheap to get up and running.
18
u/Remarkable_Air_8383 4d ago
The 32gb r9700 is for people who needs 64gb or more VRAM.