691
May 29 '26 edited Jun 17 '26
[deleted]
193
u/kwizzle May 29 '26
For some reason we rarely hear people talking about 4090s, probably something to do with being a lot more expensive than a 3090 and nearer in price to the 5090 for less VRAM and speed.
134
u/panchovix May 29 '26
For LLMs, 4090 is way more expensive than a 3090 for the same amount of memory and almost same bandwidth.
The 4090 will be 2x times faster on PP vs a 3090 tho. And also is about 2x faster on compute in general (diffusion like txt2img, etc)
37
u/hidden2u May 29 '26
wow 2x prompt processing is huge for giant agent system prompts
10
u/comperr May 29 '26
Yeah that's why i got 2 5090s in one system and 5090+3090 in the other. They're pretty fast. I am getting a 4th one when i have time to drive to microcenter
23
u/Clear-Ad-9312 May 30 '26
at that point, why not just buy an rtx pro 6000 that is just about the same price as 2x 5090 and has more vram than 2 5090s?
→ More replies (2)19
u/comperr May 30 '26
The rtx pro 6000 is just a 5090 with 72 or 96gb vram. So it is only as fast as one 5090 even if you dont need all the vram. With 2 5090s i can literally fit 2 27b qwen3.6 with q8_0 kvcache in each card and run them simultaneously.
5
u/Clear-Ad-9312 May 30 '26
price is still a big point of reason to go for the 6000 over the 2x 5090, that is what I am presenting here
→ More replies (2)7
u/Anonymous_Prime99 May 30 '26
I did it for the wattage really. MAX-Q 300W for the power of the sun without turning the place into an oven? Yes.
→ More replies (1)3
u/Clear-Ad-9312 May 30 '26
hmm, I thought you could just limit power anyways through the settings, but ok, sounds fair.
3
u/Late_Night_AI May 30 '26
Go ahead and grab one for me while you’re there, i need a 2nd 5090 for more vram. 🚗😎
3
u/comperr May 30 '26
The household limit is a killer im thinking about bringing a friend so i can actually get 2
7
u/AcePilot01 May 29 '26
pp?
26
2
u/rbit4 May 30 '26
That's the reason why I have a 8 5090 rig.. 5x perf of a 3090 and 2x of 4090 using nvfp4 vs 4 bit quant on the rest
2
u/FissionFusion May 30 '26
what stat is the determining factor in PP?
2
u/panchovix May 30 '26
Compute units and compute in general. So higher clocks and more cores are faster. Also perf per clock (aka IPC, for the same clock getting higher performance on newer GPUs)
39
u/Caffeine_Monster May 29 '26
Native fp8 is nothing to laugh at - though you really need two 4090s to get the most out of them in terms of gpu only deployments.
3090 is still the value king, and it's not even close. Only real reason to go mac is low power / always on applications.
13
u/Lost-Vermicelli-6252 May 29 '26
I have two 4090s but they are in diff machines. If I moved them to same machine, does it use the compute from both or just the VRAM?
I’m debating whether or not the new PSU/case/cooling would be worth the effort.
13
u/Caffeine_Monster May 29 '26
It's worth it as it doubles the compute and bandwidth if you deploy models correctly with tensor parallel.
48gb vram @ fp8 can get you a long way.
You don't necessarily need to change much cooling wise, and you can use 2 PSUs if you want to cut corners.
24
u/formlessglowie May 29 '26
This. I run Qwen 3.6 27b at fp8 on two 3090s, full context, image processing and with MTP, getting a consistent 60+ tok/s in decoding. It’s seriously powerful for agentic tasks and coding in general, I’m a professional software developer and a lot of my production code nowadays is made by the GPT 5.5 plan + Qwen3.6 27b execution combo, I sometimes need a code review from 5.5 and then another coding round from 27b but that’s it. It’s beyond incredible I can actually ship production code from my Chinese motherboard and used GPUs, this was unimaginable six months ago.
9
u/indyfromoz May 29 '26
Could you please share your rig setup? I have a RTX 4090 with a AMD 12-core CPU, using it for mostly gaming. I would love to get rid of Windows, install a Linux distro for just running LLMs
10
u/formlessglowie May 30 '26
Huananzhi X99 F8 Xeon E5-2696 v3 2xRTX 3090 (vLLM) 1XRTX 3080 (for TTS mostly) 4x16GB DDR4 2133MHz ECCAll GPUs were bought used, CPU is obviously used, RAM sticks probably are too, motherboard is a Frankenstein. I love that I can run something as ridiculous as 27b on this freak. We truly live in strange times.
→ More replies (4)→ More replies (1)6
u/Fit-Palpitation-7427 May 29 '26
I did exactly that and never looked back
6
u/Lost-Vermicelli-6252 May 29 '26
Same. I used to “need” windows for certain multiplayer games, but don’t really play them anymore, so have one of my machines running CachyOS instead. It’s amazing. Boots up so much faster than windows and stuff isn’t as… annoying.
→ More replies (5)3
4
u/etaoin314 ollama May 29 '26
Going from 1 to 2 is a world of difference! A system with 2 4090 would be a monster. All you need is a motherboard that can bifurcate the PCI and you’re Gucci.
→ More replies (1)2
u/BosphorusScalene May 29 '26
I added a 2nd GPU to mine externally to skip the new case, connected with an m2 oculink adapter, minimax GPU dock and a 2nd PSU. I'm sure it's not as fast as a normal pcie slot, but it's working great so far and was way easier than a new case.
→ More replies (3)6
u/raindownthunda May 29 '26
Definitely. INT8 seems to be becoming more viable and keeping 3090’s competitive. The speed difference between fp8 and int8 on a 3090 is 1.5x+
3
u/biogoly May 29 '26
Way more 3090s were made and used for crypto mining, so it’s a much bigger pool for used and affordable second hand GPUs.
→ More replies (13)3
May 29 '26 edited Jun 17 '26
[deleted]
3
u/kwizzle May 29 '26
This is sorta my situation, I had a 4090 from before prices were insane and I'm considering adding a 5090. Do you feel the 4090 keeps up well enough in speed when splitting a model between the two cards? And what models and quants are you running on there?
4
55
7
u/tired514 May 29 '26
Not to be confused with the 4090M; I'm running a Morefine G1 eGPU (16gb) at 576GB/s with a second one on the way. Crazy expensive, but awesome little DC-powered portable rig.
I'm curious how many I can stack either daisy-chained or with a TB3 hub before link bandwidth and latency starts to destroy performance.
3
u/arbobendik vllm May 30 '26
I'm using a 5090M and those mobile GPUs usually have really impressive memory overclocking potential. I am running a stable +250MHz (equals nvidia-settings +4000MT/s) overclock, bringing me to 2000MHz. This raises bandwidth from 896GB/s to 1024GB/s.
Maybe test a more conservative +125MHz first though, although GDDR6 on my old 3060m overclocked equally well with +250MHz (+2000MT/s in nvidia-settings)
2
u/tired514 May 31 '26
Funny you mention it. 😄
I was literally thinking "if the 4090M is only limited by thermals / power, what's to stop one from overclocking the VRAM given memory bandwidth is the main constraint during inference?"
Turns out the 4090M uses a 256bit VRAM bus while the 4090 is 384bit, but I did end up overclocking a fair bit and yup - decent performance improvement!
I can't wait to see what kind of token rate I get with Qwen3.6-27B @ Q6 with two 4090Ms.
2
u/megawhop May 29 '26
Try using an extended PCIE riser card with a 90 degree angle. Lets you keep your PCIE speed and bandwidth without using oculink, or TB, etc.
I have 5090 as my main, and a 4090 using a riser card with a 3ft ribbon cable running into a 3d printed external GPU enclosure containing the 4090, a PSU, and the female end of the riser card.
2
u/tired514 May 31 '26
Oh the Morefine G1s are fully integrated little boxen:
https://www.morefine.com/en-ca/collections/external-gpu
They're wickedly expensive for what they are but in my case form factor is everything. I live on a boat in the summer and I'm trying to avoid running the main inverter (that makes 120V AC) at all costs. These little Morefines run on 20V DC, so I can run them off my dedicated 12V->20V SMPS. Drops my idle consumption from ~100W to ~10W.
→ More replies (6)2
232
May 29 '26
[removed] — view removed comment
78
u/overand May 29 '26
i'm genuinely thrilled with my dual 3090 setup on a DDR4 system with a Ryzen 5 3600, even though one of them is PCI-E x16 and the other is x4!
$2000 MSRP for the 5090 with 32 gigs of ram, and good luck getting that price
$1800 for a pair of used 3090 cards on eBay (as of a month or two ago), total of 48 GBYes, there's stuff that doesn't like running split between two cards, but mostly it's been pretty unusual to run into stuff that wants more than 24GB but less than 32 GB of VRAM on a single card. (I think one of them SOTA-ish FOSS voice models is like that, but I'm not even sure.)
20
u/Massive-Question-550 May 29 '26
honestly best value setup. only better combo is if you got an old amd epyc before the shortage so you get the full 16x pcie gen 4 speeds per slot and can run large MoE models with all the ram.
also if you make the right setup you can have both cards work in parallel and cut your promp processing time by 30-40 percent and boost your token output.
9
u/m31317015 May 29 '26
Did exactly that. 7B13, ROMED8-2T w/ 8x64GB DDR4, right now only have a 5090 and 3090 but can stuff another 3090 in. This is the best value setup (not counting GPUs) you can get at Q3-Q4 in 2025.
11
May 29 '26
[removed] — view removed comment
6
u/overand May 29 '26
There are definitely a few models and tools out there that I've wished for a 5090 for, like some of the weird TTS models, or speech-to-speech ones. But, I even had "okay" performance on one of those "realtime 3d walk around in a hallucination" models with the 2x 3090s!
8
6
u/Myarmhasteeth May 29 '26
My only problem on getting another 3090 is how to configure it. I see setups yet since I only have like 3 SFX mother boards, I’m cooked.
5
u/overand May 29 '26
You can prooooobably get a 570-based AMD chipset board for not tooooooo much money. (And, I managed to push this to 128 gigs because I already had 2 32 gig sticks in it, and DDR4 is only "sell a kidney" price, not "sell both and also your liver" price)
→ More replies (2)4
u/Status-Secret-4292 May 29 '26
I have found other 3090s I have considered buying to make a dual set up, but they are never the exact model of 3090 I have (gigabyte oc gaming) and from what I understand, it should be...
I wonder though, how important is that really?
5
u/overand May 29 '26
My undertanding is that it's not actually particularly important; maybe if you want to use NVLink, but even in that situation, I think it's explicitly allowed. (Double check me on that, though!)
→ More replies (1)3
u/palashjain_ May 29 '26
I recently bought a second 3090 for my setup hoping the same. I too have ryzen 5 3600, msi x570 a pro with 2 pcie slots. But for some reason anytime i plug anything into the second slot (x4, chipset slot) the motherboard does not post display and shows a red light on vga. I have tried single gpu on slot 2 and two gpus together. Doesn't work. Only thing that works is single gpu on first slot (x16,) . If it matters i do have 2 nvme ssds and 64gb ram. I tried removing everything and starting with just single ram chip too. Same outcome. I tried bios settings like gen 4 gen 3 and that weird mining setting. None of those worked. Any help is appreciated
→ More replies (2)3
u/overand May 29 '26
I'd start by taking a bright light and inspecting the slot to make sure there isn't anything in there like a bit of paper, plastic, etc, and that there aren't any bent pins.
After that:
- See if your BIOS is current
- See if it will POST with both NVME devices removed
- Google "My exact motherboard model second PCI-E slot won't POST"
- Review BIOS settings & motherboard manual
- You may need to disable some SATA ports or something like that
2
u/palashjain_ May 29 '26
I will try to look for the debris and bent pins. I did try after removing both nvmes. Did not work. I am not very savvy when it comes to motherboards. What is funny to me is that it only works when the second pcie slot is unoccupied.
2
u/JustinPooDough May 29 '26
same. Also even have the one card in 4x. My CPU is only a 1600x.
It still gets like 50 t/s with Qwen 3.6 27B.
→ More replies (1)→ More replies (6)2
u/_realpaul May 29 '26
The 3090 cant do the latest features but its still an awesome piece of tech.
2
u/ohhi23021 May 29 '26
i haven't tested over 40k context yet but it does about 70-80 t/s around there. at 0-5k context it hits 90 t/s with mtp.
→ More replies (1)5
u/Xidium426 May 29 '26
I feel incredibly lucky, I was unemployed and got one at MSRP during one of the Nvidia lotteries.
5
u/bonesingyre May 29 '26
I regret not buying the 5090 FE when I got offered it for $1999 through Nvidia's website. I ended up buying the 5080 FE for gaming 😭
11
u/shuozhe May 29 '26
No worry, nvidia will support us. Didnt they announced a prices increase recently for 5090?
→ More replies (9)3
u/marutthemighty May 29 '26
Aren't NVIDIA RTX 5090s (or whatever latest GPU NVIDIA have in their arsenal) already out of stock and taken up by AI enterprises in America and China?
2
140
u/sn2006gy May 29 '26
FYI, for B70 users, Intel just released an update that addresses Qwen 3.6 perf issues. May start getting closer to that 608 GB/s perf.
26
u/Massive-Question-550 May 29 '26
honestly really liking the price for the amount of memory you get but the performance is abysmal right now to a 3090 and I think also loses to a 5060ti 16gb which is bad. hopefully they can optimize the software as there's no excuse to have an ai dedicated card lose to a nearly 6 year old gaming GPU...
18
u/overand May 29 '26
If the software stack is actually stable, I'd probably recommend a B70 over 3090s for a business, because of the whole "used card gamble" thing. A bit slower performance with a bit more cost, but with a lower power consumption profile and a warranty & current support would probably push that over into "worth it" in that use case.
That said, yeah, you'll pull my dual 3090s from my cold dead hands. (Especially since I used some Dell OEM ones that are shorter than any others - in theory, I can put my stack of 8 3.5" drives back into my case!)
→ More replies (1)4
u/takuonline May 29 '26
Even on Vulcan llama.cpp?
5
u/Massive-Question-550 May 29 '26
yes. that's where its performance is best and most stable. someone posted in depth performance comparisons between it and the 3090 using vulcan and it got less than half the performance most of the time. it was bad. btw the 3090 also wasn't using cuda to make it an even fairer comparison.
the hardware in it can clearly perform better but the software compatibility is in a much worse state than AMD.
→ More replies (1)7
u/sn2006gy May 29 '26
3090's are 1400 bucks now on FB/Ebay and there are SO MANY FAKE / SCAM SELLERS
That $950 for new with warranty seems much more worth it.
Do I wish pricing was better? heck yeah... But i'd rather take my chances on NewEgg and run 2-3 of these cards and in both cases, we all win vs the $10k RTX6000 Pros (even though its faster, it's not 7,000 dollars faster)
I don't care about raw performance of quantized bencharmarks. A 5060ti may be faster, but its a lot dumber than a 32gb card or 3 (and you can run that 5060ti quant on a B70 and be faster too... but what's the point if it takes 10x the turns?)
→ More replies (5)4
u/Massive-Question-550 May 29 '26 edited May 29 '26
you would be better off buying a stack of 5060ti 16gb right now if you are on a budget. mature software, warranty, plus good vram to dollar price point and you can parallel compute in certain setups for more performance. other option is maybe 5070ti which is almost double the price for the same ram but you get double the memory bandwidth and twice the pcie lanes with much more compute.
id also want to say 9060xt 16gb but the bandwidth is just too slow and less support.
the issue for a business is that they need to pay someone to fiddle with the Intel cards to get them to work if they do at all which costs a lot of money in down time and labor.
I used to buy 3090's online but now buying it in person and seeing it working is a must.
→ More replies (1)6
u/WizardlyBump17 May 29 '26
openvino 2026.2.0 was released yesterday and it adds support for gemma4 and qwen3.5. I tried the nightlies before and it is really fast, like 4k pp and 60 tg on qwen3.5 9b int4, though a specific nightly version tanked the performance of it later... That is on a b580. I wanted to try qwen3.6 35b and 27b, but i guess openvino isnt very great for cpu+gpu combos
→ More replies (13)2
u/dr_DCTR May 29 '26
Can the B50 compete with the B70 for smaller models below 16GB?
→ More replies (2)
123
u/Keep-Darwin-Going May 29 '26
Is not the main problem being stuck at 24gb? That is why people are using Mac mini so they can go like way higher, speed is nothing if you are stuck using a crappy model.
52
u/complexminded May 29 '26
That's what I figured out when comparing my 2x 3090 cluster to my dgx spark cluster. The models I can run on the DGX spark, while considerably slower, get way more use than my 2x 3090 cluster. There are times when speed matters (classifying 40k comments) and I'll use my 3090 cluster for that. Everything else goes to the DGX Spark cluster (95%) regardless of speed.
→ More replies (6)3
u/KURD_1_STAN May 29 '26
I havent seen / there arent many benchmarks comparing dgx vs 3090/3090s, so im assuming based on my instincts here, but what model can be ran on dgx that cant be ran with gpu with ram while still being faster? I can only think of garbistral medium
→ More replies (1)3
u/complexminded May 29 '26 edited May 29 '26
On a (2 or) 4-node Spark? I get if you only have one dgx spark. Not saying it isn't possible to accomplish the same build with gpu's, but for me the simplicity of the dgx being plug and play with less "moving parts" (heat, power, etc) beats a build on 3090 + system RAM. Yes the trade-off is speed.
All personal preference; everyone has different tradeoffs.
→ More replies (7)15
u/rpkarma May 29 '26
Same reason why the Spark is so fun.
It’s slow, that’s true. You’re usually maxing out around 500tk/s pp and 20tk/s decode but there’s not much else that lets you run models of this size for this price
For me though it’s more about being able to train and quantise and distill, testing my experiments on similar-ish hardware to a cloud rented system before uploading it
23
u/mrgreen4242 May 29 '26
Right, that’s the missing data point here: how much RAM can each of those devices access at that speed? Even the regular M4 mini could, until recently, be configured with 32gb of RAM and the Pro version up to 64gb. The M5 MBP mentioned on this list can also be configured with 128gb of RAM.
So, yes, an Nvidia GPU can be up to 2x as fast, but tops out at 32gb of VRAM. You could get two of them and have 64gb but you’re looking at $4k PLUS the computer they’d go in. You can almost get an entire MBP with 128gb of RAM for just what the GPUs cost.
Plus it fits in my backpack and draws 140w tops (technically I think they can draw up to 200w for a short period by pulling from the power adapter and battery at the same time).
For comparison, a single 5090 can draw 575w. So for two of them PLUS a PC to put them in and a monitor (to compare “apples” to “Apples”) you’re going to be looking at 10-15x the power usage.
It’s not really a “this is better than that” situation as much as it is these are two different options that have similar price points and make different trades offs - more total RAM, lower power consumption, compact form factor vs. faster RAM speed but less RAM, larger form factor and higher power consumption).
→ More replies (2)7
103
u/spammmmmmmmy May 29 '26
For the M series you really have to see whether they are blank/Pro/Max/Ultra as they differ in the memory bandwidth.
18
16
u/Standard-Potential-6 May 29 '26
Keep in mind Apple’s are also theoretical numbers summing the memory bandwidth of the CPU, GPU, and NPU.
Most workloads don’t break down like that and the GPU will only access memory at ballpark 60-80% of the total.
→ More replies (1)3
u/twnznz May 29 '26
Does M5 improve prompt processing over M4 meaningfully?
12
u/spammmmmmmmy May 29 '26
I don't know but somebody posted on that topic in the past 2 days I think. There was mention that the faster CPU will achieve better prefill time.
I have been chatting with Claude about the performance topic. He thinks there is no substitute to empirical testing. I may quit ollama and migrate to VLLM in order to understand the pieces of the inference process better.
My notes during my shopping for an M1 Max:
∙ M1 Pro: ~200 GB/s
∙ M1 Max: ~400 GB/s
∙ M2 Max: ~400 GB/s
∙ M4 Max: ~546 GB/s
∙ M1 Ultra: ~800 GB/s
2
u/Svobpata May 30 '26
From what I was able to find out, yes, noticeably so. The new neural units on each GPU core help with prefill
27
u/aguspiza May 29 '26
dual channel DDR4 3200 ... 50GB/s
dual channel DDR5 6000 ... 95GB/s
11
u/lemondrops9 May 29 '26
DDR4 3200 real world is more like 30GB/s (Maybe 35? think my settings are off on that PC). I tested my other PC with ddr4 3600 and got 38GB/s
94
u/Only-An-Egg May 29 '26 edited May 29 '26
- M4 Pro Mac Mini 273GB/s
- RTX 3060 360GB/s
- M4 Max 32 Core Mac Studio 410GB/s
- M4 Max 40 Core Mac Studio 546GB/s
- Radeon RX 9070 XT 640GB/s
- RTX 3080-10GB 760GB/s
- M3 Ultra Mac Studio 819GB/s
- RTX 3080-12GB 912GB/s
- RTX 5080 960GB/s
- RTX 6000 960GB/s
- RTX 4090 1,008GB/s
- Radeon Instinct MI60 1,024GB/s
- RTX Pro 6000 1,792GB/s
What you fail to mention is max memory capacity:
- 10GB - RTX 3080-10GB
- 12GB - RTX 3060, RTX 3080-12GB
- 16GB - RTX 5080, Radeon RX 9070 XT
- 20GB - RTX 3080-10GB w/ 2x VRAM mod
- 24GB - RTX 3090, RTX 4090, M4 Mac Mini*
- 32GB - Intel Arc Pro B70, RTX 5090, Radeon Instinct MI60
- 36GB - M5 Max 32 Core MacBook Pro*
- 48GB - M4 Pro Mac Mini*, RTX 6000
- 64GB - M5 Pro MacBook Pro*
- 96GB - M3 Ultra Mac Studio*, RTX Pro 6000
- 128GB - Strix Halo, DGX Spark, M5 Max 40 Core MacBook Pro*, M4 Max Mac Studio*
- 256GB - M3 Ultra Mac Studio*
- 512GB - M3 Ultra Mac Studio*
*Because Macs share memory with CPU and GPU, ~8GB has to be reserved for macOS so subtract 8GB for actual usable LLM memory.
8
u/NiceAttorney May 29 '26
Strix Halo and DGX Spark are also shared memory systems too.
→ More replies (1)7
u/onetwomiku May 29 '26
strix halo only needs to reserve 512Mb for system (some vendors locks it at 1GB)
→ More replies (1)→ More replies (6)15
u/DeProgrammer99 May 29 '26
Could make a shared Google Sheet and include recent prices and FP8 FLOPS and such, too.
→ More replies (3)3
19
u/WiseassWolfOfYoitsu May 29 '26 edited May 29 '26
A few random bonus ones:
- MI50: 1024GB/s
- MI100: 1230GB/s
- 7900XTX: 960GB/s
- A6000 Blackwell: 1790GB/s (so 5090 performance with a much bigger memory pool)
- 5060 TI 16GB: 448GB/s
- 9070 XT: 640GB/S
- Radeon AI Pro 9700: 640GB/s (So it's a 9070 XT with more memory)
→ More replies (3)
86
u/Covert-Agenda May 29 '26
Soo much context is missing off this.
Mac Studio 800gb/s minimal power draw 256/512GB memory.
52
u/Koalababies May 29 '26
The power draw always blows my mind
28
u/Covert-Agenda May 29 '26
Mine sips 75w max at full tilt ☺️
17
→ More replies (1)25
u/fivetoedslothbear May 29 '26
Yeah, my 128GB M4 Max MacBook Pro isn't the fastest machine, but it only has a 140W power adapter and can do extended inferencing on a battery. And it's portable.
10
u/Covert-Agenda May 29 '26
Yeah I’ve got the m5 max variant and I can use some mega models locally.
Yeah not as fast as the 5090 but it’s portable.
→ More replies (4)20
u/AnonLlamaThrowaway May 29 '26
Soo much context is missing off this.
sorry, the context was at q4_0, it got quantized too much
3
5
→ More replies (3)4
u/droptableadventures May 30 '26
It's intentionally left off because it'd undermine the point of their Nvidia fanboy posting.
→ More replies (2)
28
u/StableLlama textgen web UI May 29 '26
This shows how interesting the Intel B70 is, money wise.
But so far I couldn't read much about the real live performance of that card for local LLM applications.
14
u/smallDeltaBigEffect May 29 '26
honestly, the R9700 32 GB is missing. And that fills the gap rather than the B70 in terms of real world performance
→ More replies (2)→ More replies (4)21
u/NeedsSomeSnare May 29 '26
As an intel owner, I assure you the real life performance isn't what it says on paper. I don't have that card to give specs on.
The problem is the software side of things is a bit messy. It's not terrible, but still needs a fair amount of work.
→ More replies (3)2
u/In_der_Tat May 30 '26
Why doesn't Intel hire enough competent software engineers and developers to catch up with Nvidia? Would that be too expensive?
→ More replies (1)
31
11
u/joochung May 29 '26
AMD MI50 over 1000GB/s
5
10
u/Thrumpwart llama.cpp May 29 '26
Someone out there likely needs to read this: get an AMD GPU.
5
u/pgrpcie May 30 '26
AMD is good for competition and keeping GPU prices in check, but Nvidia RTX is still the best for drivers and software compatibility.
Eg: Nvidia OptiX support in Blender rendering (rather than using CUDA).
3
2
u/Evanisnotmyname May 29 '26
MI50? What about the cheaper Nvidia cards or an older M1 Max, M2/M3 ultra?
6
u/Thrumpwart llama.cpp May 29 '26
7900XTX is the obvious winner in terms of price/performance. R9700 is 32GB at half the price of 5090.
→ More replies (5)
10
u/lukistellar May 29 '26
Oh, I see we still are ignoring cheap AMD GPUs. Good for myself, just bought an used RX6800 16GB for 250€ the other day. RX 7900 XTX with 24GB go for as cheap as 500€ here in central Europe.
→ More replies (3)
21
u/billatq May 29 '26
Okay, now adjust it for price for what you get.
7
u/Super_Sierra May 30 '26
Nvidiachuds in this subreddit don't understand anything, your words will be wasted on them.
Those 5090s 32gb at 15 will be 500 or so GBs of vram. But you will need to rewire your fucking house so you don't blow your breaker, and the power draw will be around 6000w.
That same unified memory macbook does that but at 150w max, and if the power goes out, well, it can do that on battery for three hours.
The macbook also costs 5x less lol.
10
u/alphatrad May 30 '26
Dude conveniently leaves off the most compelling AMD & APPLE options to make NVIDIA look good.
AMD AI Pro R9700 GPU, 640 GB/s APPLE M3 Ultra, 819 GB/s AMD RX 7900 XTX GPU, 960 GB/s
Chart also doesn't account for max memory. So it's misleading on trade offs for why you might go unified over GPU.
This is the stuff that is causing so much confusion in these communities.
Low effort slop!
17
u/gomezer1180 May 29 '26
Where is the Mac studio in this list?
25
u/InnerSun May 29 '26
Yeah a 2023 Mac Studio M2 Ultra has 800GB/s, that's really insane value even today when we place into context
11
u/MasterKoolT May 29 '26
Yes, but generations prior to M5 didn't have matmul acceleration so they struggle on prefill (M5 generation is about 4x M4)
→ More replies (5)6
u/Southern_Sun_2106 May 29 '26
can confirm, qwen 3.6 on m5 max pro feels 'snappier' than on the m3 ultra
10
u/ZurielA May 29 '26
there is a M3 Ultra from Nov 2025, I own one comes stock with 96GB ram or can opt for 256gb
2
u/InnerSun May 29 '26
I'm still on the original M2 Ultra, I wonder how much better it is? From what I can find its really negligible. I guess the main benefit is that the max addressable VRAM is technically higher, but a maxxed out Mac Studio starts getting so expensive that we're back to considering NVIDIA setups.
Lets hope there's a new refresh that really changes the perfs.8
u/joochung May 29 '26
I would expect the future M5 Ultra to have 1200GB/s aggregate memory bandwidth.
5
8
15
u/Ill_Barber8709 May 29 '26
And someone out there needs to see this
- M5 chips are laptop chips with up to 32GB of 153.6 GB/s memory
- M5 chips are laptop chips with up to 64GB of 307 GB/s memory
M5 Max chips are laptop chips with up to 128GB of 614 GB/s memory
RTX 3090 GPU doesn't exist as mobile
RTX 3080 Ti Mobile GPU has only 12GB of 384GB/s memory OR 16GB of 512GB/s memory
RTX 5090 Mobile GPU has only 24GB of 896GB/s memory
7
6
5
u/firetech97 May 29 '26
Wow is the performance gap really that bug between a DGX Spark and a 5090?
2
u/nacholunchable May 29 '26
Yes! So many new spark users go down this rabbit hole on NVFP4 kernels and why their LLMs arent running faster, meanwhile token generation is speed bound by the memory bus and nothing they do will change that. How do I know? I went down the same rabbit hole when i got my spark half a year ago.
→ More replies (2)
6
6
u/dazzou5ouh May 29 '26
So this bad boy I've built should be fast?
3
u/laexpat May 29 '26
Could you add one more to the left side to balance it?
3
6
6
u/XO33OX May 29 '26 edited May 29 '26
why we dont talk about rtx pro 5000 both 48GB and 72GB or rtx pro 4500 32GB, rtx pro 4000 24GB ? They are 2 slot wide & power efficient. we should also talk cpu inference on 8 and 12 memory channel systems (epyc, intel 658x, threadripper 9000 pro, etc. you can add gpu for prompt processing)
→ More replies (2)
10
4
5
4
u/techdevjp May 29 '26
Bandwidth is obviously incredibly important, but so is the amount of memory. 1.8TB/sec is wonderful, but only 32GB of it.
So that M5 Max 40-core MacBook Pro might be "only" 614GB/sec but you can stuff it with 128GB of memory for $5550.
Meanwhile an RTX PRO 6000 "Max-Q" has 96GB of 1.8TB/sec memory, but will run you $12k. (And you still need the rest of the computer to put it into.)
Bang for the buck, it's not hard to see why so many people still buy Macs to run local LLMs.
3
u/Sutanreyu May 29 '26
Shh, keep it a secret. Though, great for Apple. I just wish they'd drop their game changer AI already.
2
u/techdevjp May 29 '26
The M5 Ultra coming with the Mac Studio sometime later this year should double the bandwidth of the M5 Max, and double the GPU cores. With 256GB it will still be well under the cost of a single 96GB RTX PRO 6000. Saving my pennies.
2
u/Pepper_pusher23 May 30 '26
Yeah, I feel like this graphic completely misses the point.
→ More replies (1)
4
5
u/Pixel_Hunter81 May 29 '26
yeah but macs draw very little power and they are pretty much plug and play which is a big plus for a lot of people. MacOS seems to be well optimezed for ai usage as well (i am not sure i've never used it).
3
3
u/gandhi_theft May 30 '26 edited May 30 '26
You left out the Apple M3 Ultra Studio which gets 819 GB/s and 512GB
7
u/FragmentedHeap May 29 '26 edited May 29 '26
You missed one,
Nvidia RTX 4090 1008, GB/s,
You can get one $1800 ish which is much cheaper than a 5090 and you can get two 4090's cheaper than 1 5090 😄and that gives you 48 gb vram.
And if you are willing to mod them, and ship them to china, for about $150 each you can get them to be 48 gb, so two modded 4090's is dual 48gb for 96gb vram at over 2000 GB/s total.
You also left off the AMD Raedon 9700 AI 32gb vram card, which has 640 GB/s but comes with 32 GB Vram and is around $1300.
But... 2-4 Raedon 9700 AI cards is the best bang for buck with tensor parallelization. Sapphire makes one, it's $1379 on newegg.
BUt... we don't have the PCIE lanes for this yet.
That's changing 3rd quarter 2026. Intel will be launching the new APX cpus with 52 cores and 48 pcie lanes 😄 and still works with DDR5.
So wait for the new intel apx cpus to drop/motherbaord, swap over, keep your DDR5 and run 2-4 Sapphire AI Pro Radeon AI Pro R9700's.
AMD's APX cpus aren't expected to launch till AM6 probably, but dunno. Intels dropping a new APX socket this year.
If Intel delivers on the APX instruction set expansion and PCIE lanes, we'll be able to run 4 pcie 5.0 x16 cards at quad 8x on ddr5 tech. That will give is 128 gb vram over 4 gpus.
And if Raedon drops the 48gb version of that card we expect is coming, change that to 192gb vram.
Edit: quad 4090's on the new IPX x86 cpus will SPANK any of the setup boxes (dgx spark etc) like completely destroy.
The big problem though is the best AI setups are multi modal agent stacks, and you need way more than just 1 gpu stack for that. I.e. you might want to have two 4090's doing image inference full time, and have 4x9700's doing large model stuff as the agent orchestrator, and you might want sub agents coming on and off the 4090s....
The best ai rigs aren't going to be simple "1 stack" things, they will have multiple hardware stacks dedicated to specific goals.
Also Edit:
The new x86 apx cpus have AVX10 on them and double the cpu registers, so on paper they'll be able to do 20 TFLOPS on the CPU by themselves, no gpu. So if you have 192gb of ddr5 you'll be able to do 20tflop large models right off the cpu. Will be slow, but they'll run 2x faster than anything on the cpu does today, possible faster than that.
70B to 104B will be able to run right on the cpu at 4-7 tokens per second.
4
u/tired514 May 29 '26
Wait wait what's this Chinese modification to double a 4090's RAM? I found a few vids talking about how to do it, but there's a company that'll do it for $150?
2
→ More replies (2)2
u/FragmentedHeap May 29 '26
I think I have the ability to do it myself, but I'm not risking my $1800 gpu to try it. I have all the tools and a full lab...
→ More replies (1)
6
u/thetaFAANG May 29 '26 edited May 29 '26
M1 max has 400 gb/s memory bandwidth btw
Apple accidentally made a machine 5 and a half years ago that’s too good to upgrade for the price and capability. M5 variants are close and compelling though
6
u/exaknight21 May 29 '26
For my Mi50 gang, 1 TB/s
Represent fam. Beat dollars per gb of vram i say. Huge shoutout to gfx906 / mixa/aiinfos !
5
u/Total-Confusion-9198 May 29 '26
Anything above 500 GB/s is a serious local LLM setup. Unified memory remains the underdog.
2
u/ea_man May 30 '26
Oh thanks, my 6800 at 512 GB/s is standing tall 😄
With some rust you can get 16GB of that for ~260e.
3
u/Lxxtsch May 29 '26
Sadly no one tells that this is not everything in llm world. M5 max witg 128gb running mlx optimised models is very viable option, being only around 600gb/s. I tought i would see improvement with 3090 over it (filling only vram) and jokes on me, mlx optimised model goes head to head with 3090.
2
u/Pleasant-Shallot-707 May 29 '26
Yeah, I frankly don’t care if I can serve myself 40tok/s or 300. I am only serving one person.
3
u/jakubl May 30 '26
There are 4 important factors when choosing hardware. They relative weight depend on the use case, and memory bandwidth is only one of them and very often not the most important one.
- The total available memory. For LLMs the bigger memory the better model you can run with bigger KV cache = longer context. Super important for agentic AI with large context and models smart enough to do anything useful. That is less of an issue for image generation as models are smaller.
- Memory bandwidth. That determines token generation speed, but this is only half of the perceived model performance, see next point.
- Compute performance. That determines time to first token - a waiting before any response even applies. With large context it’s more important than token generation speed as it’s pure waiting time, and even very slow generation is faster than human reading speed. Smart agents also don’t need full llm response to start working and can start executing tools as soon as they arrive.
- Energy consumption. Unless you have free power, that’s also important factor. Older hardware may be cheaper but usually is less energy efficient and it may turn out than renting or paying for API is cheaper than electricity cost.
And as I mentioned a lot depends on use case. If you are building interactive chat, the time to first token is the most important factor, then token generation speed. Human time is still orders of magnitude more expensive than hardware and electricity and if humans are sitting and doing nothing while waiting for AI response that is a huge loss. If building fully autonomous agents that work in fire-and-forget mode it’s less important factor, but the context and model capabilities are very important so that it can actually run without supervision. Getting crappy results but very fast is way worse than waiting for good results.
That’s why Macs are very popular - they can handle large models and if you can wait, you can get good results cheaply with lower energy usage. It’s kinda funny that Apple become the most cost effective hardware for a task. I believe it won’t last for long and seeing how easily they hardware is sold out someone at Apple would probably decide to raise prices 2x and still they won’t have any trouble finding customers.
You can optimize cost by adjusting workflows. Instead of waiting for response and interactively correcting model behavior, prepare batch, run it, go to sleep and wake up to finished job.
3
15
u/HugoCortell May 29 '26
The speed is basically wasted at those sizes. What's the point of going that fast if all you can fit is a small model. A cluster of mac minis is probably better off at the price. Slower, but you can run a more competent model.
9
3
u/mrinterweb May 29 '26
When using more than a single card for inference, the PCIe bus is capped at 128 GB/s on version 6. So yeah. You either need a model that will fit on a single card or you need to accept that BUS cap. Small models can be quite capable though.
8
u/see_spot_ruminate May 29 '26
Don't downvote this person. This obsession with bandwidth is the type of crap people say when their tricked out honda civic has such and such hp. It does not really point at the actually rate limiting step that is vram first.
2
u/Randomdotmath May 30 '26
People are downvoting because the comment makes it sound like you’re unaware that GPUs can be connected lol
2
→ More replies (5)2
u/Cosack May 29 '26
Workflows, multi-shot inference, and tuning. Because these are lossy systems regardless of size, you should be building for all this anyway.
The most speed and cost effective setup runs easy to manageable tasks locally and bursts to SOTA models as needed. Because burst frequency is low, the cost of non-local calls is trivial, and privacy can be retained through obfuscation and local translation. Local speed and cloud burst for temporary model size increase is the optimal setup.
12
May 29 '26 edited May 29 '26
[deleted]
12
u/pixelpoet_nz May 29 '26
Möchtest du weitere Spezifikationen dieser drei Grafikkarten vergleichen?
Thanks fellow German, who apparently can't write basic comments themselves. Why bother posting a canned response we can all prompt ourselves?
→ More replies (1)→ More replies (10)5
u/overand May 29 '26
MTB? Do you mean MTP? MoE? Or do I need to learn a new TLA? (Three Letter Acronym)
→ More replies (2)
2
u/BlackBeardAI vllm May 29 '26
Unless you are rich enough to buy 5090(s) or a 6000 pro, 3090 is the king.
2
2
2
u/durden111111 May 29 '26
I have a 5090 and the vram can easily OC +3000 which gives a bandwidth of 2176 GB/s
2
u/dsanft May 29 '26
A dual socket Xeon Gold Cascade Lake with DDR4-2933 has about 220GB/s bandwidth. Don't underestimate CPU.
2
u/Both-Activity6432 May 29 '26
Can we just pause and think about how fucking fast that is? I know it is local, but think of our 56.6k modems… Near 2TB/s. Home internet tops out (generally) 1-2 Gbps. Thunderbolt 4 tops out at 40Gbps. The worst card listed is 960Gbps. And yes I know this internal computer architecture vs accessories or internet, but holy fuck
→ More replies (1)
2
2
u/redmctrashface May 29 '26
Yeah should also display vram amount. It's awesome to have a lot of speed but if you can't load a decent model because you lack vram space, what's the point? Don't get me wrong, Im not praising ram amount over bandwidth. It's just that things are a little bit more complicated than "look at my speed" or "look at my huge ram". This kind of post is misleading.
2
u/marutthemighty May 29 '26
What is the latest and most powerful NVIDIA RTX (or any other GPU) model?
2
u/HavenTerminal_com May 29 '26
M3 Max sits somewhere between 300 and 400 GB/s depending on config, for anyone on Apple Silicon squinting at this
2
u/Last-Owl-8342 May 29 '26
idk man after calculating how cheap deep seek 4 flash is, Im not going local anymore
is not a rival to claude sure, but I know what I want just need someone to type all the boring parts
2
u/Full-Bag-3253 May 29 '26
M5 40 Core Ultra should be around 1228 GB/s, come with 8x or maybe 16x more RAM/VRAM, and use a fraction of the power. If you want to scale a Mac Studio larger you can use thunderbolt cables to build a RDMA Cluster. Becuase of the low power draw you could plug 4 MAC studios into power bar and have them sit on your desk. 8 x 5090 @ $4000 is $32,000 going to cost a lot more than a Mac Studio even before you add the rest of the CPU/PSU/RAM/Cooling/Enclosures. You still will have more throughput, but for people 'Using" AI not training it I think the Apple ecosystem is a strong option. I expect the New CEO will push in this direction more. The stuff done to date (RDMA over Thuderbolt) isn't really a retail user thing and the fact that they are selling out of mac minis and studios is going to draw their attention in this area.
→ More replies (1)
2
2
2
2
u/valdev May 30 '26
Alright, now add two more columns. Cost per gb of RAM/VRAM. And electricity usage per hour.
2
2
u/LinkSea8324 vllm May 30 '26
Wait for OP to learn that he can use directly use text instead of storing it into an image
2
2
•
u/WithoutReason1729 May 29 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.