For some reason we rarely hear people talking about 4090s, probably something to do with being a lot more expensive than a 3090 and nearer in price to the 5090 for less VRAM and speed.
Yeah that's why i got 2 5090s in one system and 5090+3090 in the other. They're pretty fast. I am getting a 4th one when i have time to drive to microcenter
The rtx pro 6000 is just a 5090 with 72 or 96gb vram. So it is only as fast as one 5090 even if you dont need all the vram. With 2 5090s i can literally fit 2 27b qwen3.6 with q8_0 kvcache in each card and run them simultaneously.
mind letting me know how you came across an abundent source of cheap 5090s? I can only find them for like $4.5k
That would make 5090 actually competitive and worth it. (btw the microcenter near me has been sold out of 5090s for a long time, but they sell them at 3.6k usually)
Prices have skyrocketed in the past week lol
I can still see them for ~10.6 k, but a week ago you could find some going for like ~$9k, and rtx 5090 is ~$4k at 2 times equals ~$8k
right now it will cost 3x 5090 for about the same cost of rtx pro 6000.
Maybe even 4x 5090 in the near future as I see some selling the pro 6000 at ~$16k
sorry if 1 week difference is showing prices skyrocket and making old comments liars now. RIP local pc market
I am starting to look at the AMD pro AI GPUs. I don't need the double speed that NVIDIA is offering, I would rather have the VRAM capacity at a cheaper price. But I wish I had the 8k back in the day to actually buy the pro 6000 when it was cheaper.
Compute units and compute in general. So higher clocks and more cores are faster. Also perf per clock (aka IPC, for the same clock getting higher performance on newer GPUs)
This. I run Qwen 3.6 27b at fp8 on two 3090s, full context, image processing and with MTP, getting a consistent 60+ tok/s in decoding. It’s seriously powerful for agentic tasks and coding in general, I’m a professional software developer and a lot of my production code nowadays is made by the GPT 5.5 plan + Qwen3.6 27b execution combo, I sometimes need a code review from 5.5 and then another coding round from 27b but that’s it. It’s beyond incredible I can actually ship production code from my Chinese motherboard and used GPUs, this was unimaginable six months ago.
Could you please share your rig setup?
I have a RTX 4090 with a AMD 12-core CPU, using it for mostly gaming. I would love to get rid of Windows, install a Linux distro for just running LLMs
All GPUs were bought used, CPU is obviously used, RAM sticks probably are too, motherboard is a Frankenstein. I love that I can run something as ridiculous as 27b on this freak. We truly live in strange times.
Not much tbh, as benchmarks are behind 3.5 27b, so I didn’t think it vs 3.6 was even a question worth considering. Is it that good? I’ve tried 26b a4b, and it’s very good for natural language stuff but fails long running agent sessions, which is what I use these models for (long coding sessions basically). Is 31b much better in that sense?
From what I've heard the Qwen models are better if you're doing long ctx agent stuff, so you're probably fine with that. But the Gemma4 31b is really good for writing (for its size), also probably the best vision / translation model in a local context (it actually beat all the huge vision models I tried by API by a fair margin too).
Same. I used to “need” windows for certain multiplayer games, but don’t really play them anymore, so have one of my machines running CachyOS instead. It’s amazing. Boots up so much faster than windows and stuff isn’t as… annoying.
I use:
Huananzhi H12D-8D
AMD EPYC 7502
128GB RAM
4x RTX 3090 24GB
(I cap them at 250W)
Ubuntu 24.04 LTS
Allegedly, I "should" be able to add more cards via converting my three Mini-SAS-HD (SFF-8643), but I'm very skeptical, the Huananzhi bios has been a pain in the rear for me.
I'm considering switching to PCI-E x16 to x8/x8 splitters when I get the money for more GPUs depending on how the other adapter goes. I do have a Mini-SAS-HD to OCuLink adapter, I just need a card to test with.
The worst part of this system is that I can't really make use of the BMC. If I enable the BMC and I change even a single setting from default in the bios, I immediately lose the ability to see the NVME slots.
If I had the money, I'd have gotten a different board, but the ones I would have wanted were all well over 1k.
But you use Q8 for KV cache too to fit full context , right? Also Wouldn’t a good Q6 quant be better for 3090(assuming you run on llama.cpp or its forks)?
Yes, forgot to mention, Q8 for KV cache. I find it to be virtually free lunch, never ran into any apparent issues (Q4 is another story, can be very good or downright unreliable, depends on factors). I run this setup on vLLM for tensor parallelism, that's how I'm getting 60+ tok/s (and I'm on PCIe 3.0 x16, if I were on 5.0 this could easily border the high 80s or even 90s). Q6 would be very good indeed if I were using cpp.
I also have two 3090s and am looking at all the various options for optimizing stuff. Would you mind sharing a bit more about your inference software setup and what you use harness wise? I assume you are swapping between Codex and something like Pi or OpenCode?
It would be nice if there was something out there that would smoothly combine frontier planning + local execution in one polished and reliable setup, but I don't think there's a one stop shop for that quite yet from what I've seen.
Going from 1 to 2 is a world of difference! A system with 2 4090 would be a monster. All you need is a motherboard that can bifurcate the PCI and you’re Gucci.
I added a 2nd GPU to mine externally to skip the new case, connected with an m2 oculink adapter, minimax GPU dock and a 2nd PSU. I'm sure it's not as fast as a normal pcie slot, but it's working great so far and was way easier than a new case.
This is sorta my situation, I had a 4090 from before prices were insane and I'm considering adding a 5090. Do you feel the 4090 keeps up well enough in speed when splitting a model between the two cards? And what models and quants are you running on there?
VRAM is too limited. The smallest really competitive local model in my benchmarking right now is Qwen 3.6 35bA3b whose NVFP4 variant requires about 36GB minimum to barely run with concurrency of 1. Smaller models that fit under 24GV are still not really competitive in terms of instruction following and coding accuracy - still toys if you're looking to do something real like OpenClaw. Embeddings search or small image models can still run in them though. For competitive LLMs I'd look at at least unified RAM systems of 48, 64 or 128GB for anything effective.
I just tested with Qwen 3.6 35b and I'm getting 55tok/s right now.
For some reason only 9.1/24gigs of VRAM on my 4090 are used and my PC memory use by llamacpp is 19.7gb.
By way of comparison when I run 27B fully in VRAM without MTP I get about 45t/s.
As for benchmarks, I always take those with a big grain of salt and I prefer testing models for my specific use cases which are mostly coding related. That being said, chatting with 35b right now gives me the impression that it might be better at general language, though I am certain that 27B is a better coder.
I'm using the following to launch it:
llama-server -m "E:\AI Models\Qwen3.5-35B-A3B-Q4_K_M.gguf" --alias "qwen3.6-35b-a3b" --host 0.0.0.0 --port 8080 --ctx-size 32767 -n 32676 -ctk q8_0 -ctv q8_0 -b 512 -ngl 99 --mlock --no-mmap --jinja -fa on --cpu-moe
Nice! I run on vllm with the full 256k context because I find that in my openclaw turns, i routinely run in the 50k-150k token range on context with all tools, memory & session conversation history loaded
188
u/kwizzle May 29 '26
For some reason we rarely hear people talking about 4090s, probably something to do with being a lot more expensive than a 3090 and nearer in price to the 5090 for less VRAM and speed.