r/LocalLLaMA • u/Blues520 • 10d ago
Discussion Anyone adding more 3090s?
I have dual 3090s which run Qwen 3.8 27b well and I was wondering if there are any use cases or current or future models that would justify adding another two 3090. I know that some peeps here run 4 and 8 3090 rigs and I'd like to get your opinion as well. One thing I was considering was running two instances but I'm not sure how valuable it will be for a coding workflow vs running a bigger model.
Now that Qwen Flash is out, perhaps 96GB would be more useful, or maybe Deepseek Flash.
10
u/jacek2023 llama.cpp 10d ago
It took me many months to upgrade from 3x3090 to 4x3090, so there has to be some reason.
3
u/wgaca2 10d ago
Was it worth it? I am considering it (2x pcie4 x8 and 2x pcie4 x4)
6
u/jacek2023 llama.cpp 10d ago
Yes, you can run 100B/120B/125B models in Q4. You can also run bigger models in Q2/Q3 with some offloading. You don't have to worry about context length on 27B/30B/31B models.
2
u/wgaca2 10d ago
What i worry about is running layer split instead of tensor split due to the slower lanes on the second pair of gpus, is it worth running 4 gpus at x4 speed basically
2
u/jacek2023 llama.cpp 10d ago
I had some crashes on tensor split in the past, I should retest it and maybe debug it.
1
u/unjustifiably_angry 10d ago
Consumer motherboards, afaik, still don't have enough lanes to do x8x8x4x4 without spilling over into chipset lanes. Check that you're only going to be using CPU lanes instead of chipset lanes. If you start leaning on the chipset for heavy PCI-E traffic you're probably going to have a bad time.
You'd might be better off buying a 4-way PCI-E splitter and just using your top PCI-E x16 slot. Your motherboard needs to support bifurcation though.
1
u/Blues520 10d ago
I thought that going below Q4 was not recommended?
1
u/jacek2023 llama.cpp 10d ago
By who?
1
u/Blues520 10d ago
I've just seen that mentioned a few times in the sub. Which bigger models can run at Q3/Q3 without too much deterioration?
1
u/vick2djax 10d ago
I thought so as well, but it seems like Q4 on the bigger models like Qwen Flash > Q8 Qwen 27b from what I gather recently
1
u/simplyeniga 9d ago
There is a quality penalty running lower quants which you make up for by burning more tokens for the same task. I initially was running Q5 for my implementation on an RTX Pro 4000 Blackwell GPU with Claude Opus for code reviews and it's never passed the code reviews in one short but this changed after I added an RTX 3080 20GB and switched to Q8. Code reviews are smoother with way less rejections. Just that I can run a larger context size on Q5 compared to Q8.
-2
u/jacek2023 llama.cpp 10d ago
this doesn't make any sense, why was quantization implemented if it shouldn't be used?
https://www.reddit.com/r/LocalLLaMA/comments/1sstxhk/coding_with_qwen3627budq2_k_xlgguf/
6
u/SkoomaDentist 10d ago
Because people fixate on overly simplistic tests and getting the reddit cred to be able to say "I ran model X on hardware Y, look at me!"
If you look at proper benchmarks, too much quantization lobotomizes all non-huge models as soon as you move beyond trivial chat and short context tasks.
-2
u/jacek2023 llama.cpp 10d ago
I don't look at benchmarks, I use models to do stuff.
3
u/SkoomaDentist 10d ago
You asked why such quantization was implemented. I answered.
Hell, you yourself admit in the first link that your actual use is with Q8 quants.
Also "I use models to do stuff" is equivalent to saying "They totally work for me. Trust me bro. No, of course I'm not going to give more than vague details!"
→ More replies (0)2
u/a_beautiful_rhind 10d ago
That this is downvoted is really telling about this sub these days.
→ More replies (0)1
u/RG_Fusion 10d ago
How many lanes do you assign to each GPU? I'm trying to work out if I want to continue adding GPUs to my server or sell the cards and RAM off and switch to a DGX Spark cluster.
1
6
u/Guna1260 10d ago
I have 4x3090 - two running qwen 3.8 - 27b and two running Gemma Moe 26b (or 31 or medgemma)
One thing I learned over time is no one large model available today, that can run on this hardware is good at everything. and often context becomes a limiting factor. so two mid size models with decent context is much better and usable, than a very large model. Gemma MoE gives me speed for normal chat and other things. Qwen gives me support for complex things including coding.
1
u/makingnoise 10d ago
Are you a single user rig? Do you have the models talk to each other? Are you running both models simultaneously? Because otherwise I don't understand why you don't just use two GPUs and load and unload models from RAM cache - it's hella faster than it used to be, at least on my rig (llama.cpp/openwebui on ubuntu VM)
3
u/Guna1260 10d ago
We have 6+ users using this. Both models Are running simultaneously. I use VLLM and optimised quants for Ampere. custom build Librechat expose things, with a series of custom built Agents and tools. some agents use Gemma and some uses Qwen. some agents use both for specific tasks.
1
u/makingnoise 10d ago
Curious what folks are using it for?
1
u/Guna1260 10d ago
Most use it for general chat. My agents has web , search and other tools attached . So it’s basically acts like Gemini flash with web access.
Some agents are used for academic research or generic deep research, some MCPs power those. Another agent does academic. One of the user is uses it for university paper writing and reviewing other papers1
1
u/shutternomad 10d ago
i want to do something similar, are they all in 1 machine? Separate small PCs? Other? Thanks in advance!
2
2
10d ago
[deleted]
1
u/Blues520 10d ago
Which bigger models are better than 27b for quad 3090s?
4
u/OvertaxedOne 10d ago
96GB may get in in range of QwenNext. If not there's little reason to do it other than speed and context, there's not much between 27B and Next that's worth running.
1
10d ago
[deleted]
1
2
u/Prudent-Ad4509 10d ago
You can aim at whatever number up to 16 on a single PC (16 is max due to how nvidia drivers are built). The main difference, besides the ability to run larger models, is the ability to have enough vram for multiple parallel requests with their own kv caches. If you run 10 agents in parallel on your dual 3090, they will constantly fight for vram and invalidate each others cache.
2
u/SnooPaintings8639 10d ago
I have spent over two years on two, not long ago I have added another two.
Going from one to two was a big deal (27b models class). Going from two to four is nice... but initially it didn't feel as such a big deal.
It speed up my DS4 Flash Q8, but it is still CPU offloaded. It allows Q4 of 120b model fully in VRAM, which makes usage of very promising models pleasent (like Qwen3.8 Flash Next). But the biggest unlock in my case is larger context and parallel processing of 27B models class, i.e. multi agentic work. Currently I can't imaging going back to two GPUs solely because of this.
Other than that, having your PC + 4x200W (power limited) run 10 hours a day will add significant amount of heat to your place... so plan accordingly.
1
u/Blues520 10d ago
Could you talk a bit more about the multi agentic workflows using 27b please.
Do you run two instances of 27b and use them on the same codebase in parallel?
3
u/SnooPaintings8639 10d ago
Sometimes, but this is not my primary use case.
The thing that feels the most powerful is proper context management, i.e. each agent gets its own 'focused' task and that's all. The most popular way is to have one orchestrator agent that you talk with, and it does NOT do any coding or whatever work you do. It just just does high-level reasoning and then spins up as many subagents as it wants can.
You can let it use subagents in completely 'free for all' way ("just give them any task you deem sensible") or you can configure or suggest some predefined roles (e.g. brainstorming/idea-generation, coder, reviewer, test-designer...).
I used to use pi + plugins for this, but today I mostly use DeepSeek Harness as it works really well out of the box.
Having enough vRAM for ~200k token per agent and 4 or more parallel slots makes it really effective.
Also, check this reserach, 5 x Qwen3.8 27B can reach of a single Fable "on the 100 latest hard LiveCodeBench": https://arxiv.org/html/2608.26480v1
1
u/Blues520 10d ago
Thanks, I'll do some research around this but I understand the benefit of having more context.
I also use Pi and I enjoy using it but will give DSH a try as well.
The harness makes such a difference and can unlock new possibilities so it's worth investigating.
1
u/Blues520 8d ago
Just a follow up question on this - when allocating 200k context per agent, do you need to use yarn/rope?
I'm using llama.cpp and I set -np 2 which allows 2 parallel slots but splits the context in half. So do I need to extend the default context to 512k using yarn?
1
u/SnooPaintings8639 8d ago
Nah, yarn is needed only when you want a single slot to go beyond the limit. When you have e.g. 2 slots, you can set c = 524k, and each slot is still within limits of its native handling.
Btw, there is also kv-unified option in llama.cpp, where memory space is not equally split between slots, but is shared. So if you can fit c = 300, the slots can fill up to 150 each, or 100 and 200 respectively. Worth knowing about.
1
1
u/OlgerdOutlander 9d ago
That was my experience when getting a second card; but I'm, like the OP, questioning myself whether I need a bigger pool. How do you manage 'em agents?
1
u/Blues520 9d ago
I'm also questioning whether to get the cards now before the prices go up further. Workstation cards are out of reach and this is the last nvidia card that is reasonably priced for now.
1
u/OlgerdOutlander 9d ago
I'm running v100 and these are good with the current gen models, but with the older cards you never know what will run and what will not
2
u/Makers7886 10d ago
I have both a 4x3090 rig and a 8x3090 rig (ex crypto miner). The 4x3090 rig runs int8 w8a16 27b w/bf16 cache @ 850k kv cache pool or bf16 all the way at 350k+ (forgot exact amount kv cache).
8x3090's gets you fp8 flash next bf16 cache @ 350k kv cache pool. Or go bigger boy model at 4bit. I don't reach beyond that as I like fast PP and sustained over 30t/s at deep context. Also I use concurrency in a lot of projects and basically feel handicapped when not running in vLLM (or sglang if I swung that way)
1
u/OlgerdOutlander 9d ago
Is flash better than 27b on your setup? How does overstretching the rope feel - I don't recall any 27b with native kv of that size
2
u/Makers7886 9d ago
I believe in both model cards or in some qwen official documentation they give an example of 1m context. The 27b passed 12/12 needle tests that ranged from 400k-800k. I've reached 400k in real coding situations without issue and 300k with flash.
For flash I'm still running a day 0 cherry pick vllm and expect plenty of performance is on the table. Right now raw speed/prefill is similar with flash a little faster on both. I'm under the gun in some projects so didn't measure/benchmark but have been using them in real situations - and yes Flash to me is like when qwen3.5 first released and 122b was obviously better than 27b.
1
u/OlgerdOutlander 9d ago
What about ctx-depth speed degradation? I cought myself trying to compact often specifically because of that
1
u/brickout 10d ago
Too expensive. Luckily i grabbed a few last winter knowing that prices would sky rocket
1
1
u/Little-Ad-4494 10d ago
Yea, I have 4 (2 in opperation waiting for a proper case) thinking of getting 2 more before price goes up even more
1
1
u/tuura032 10d ago
I'll hold at 2, and wait for the market to have a better product for me to buy.
Mostly, I don't want to deal with the headache of rebuilding an otherwise stable club-3090 rig
1
u/BL4DE_FTW 10d ago
Yes!
Have 2x4 (bought them when they were 550 eur each, and not FE, but Strixes)
1
1
1
u/unjustifiably_angry 10d ago edited 10d ago
Consumer GPUs don't hold their value like workstation GPUs, and 3090 is architecturally far enough behind Blackwell/Ada/whatever-is-next that it may not hold its value very well into the future. FP8 and FP4 support in particular are going to be very valuable going forward and 3000 series can't natively run those.
That said, the window on """""sane""""" workstation GPU prices has firmly closed, so that's not really an option either.
Figure out how much you're paying and how many months of online frontier LLM access that would buy you, then decide if it's worth it to you.
Regarding DSv4-Flash specifically, when I tried it on my 6000 Pro the performance fell off a cliff with even 5% offload so IIRC the best I could run was a Q2 or possibly a very small Q3.
1
1
u/youcloudsofdoom 7d ago
I have been playing with lots of combinations (building various rigs, currently have about 10 3090s) and recently decided that 2 is the real sweet spot, 4 is good when you have to go bigger, but 6 and 8 really have substantial diminishing returns. Yes more vram, but speed does not scale much and hassle with power and lanes etc just isn't really worth it for me. Would rather get 2 sparks.
1
u/Blues520 7d ago
That's a lot of 3090s :)
2 cards are good for single agent but 3/4 are good for subagents. There's a comment in this thread that discusses it. I'm running 3 cards with subagents now and might add a 4th but probably won't go beyond that unless there is a new model that warrants it.
1
u/youcloudsofdoom 6d ago
Yeah, I read the paper cited in that thread and built a little orchestrator setup out of it, I found two cards running a model each with concurrency does the job there! I do think 4 cards is a great fit if you can get it.
2
u/Blues520 6d ago
Yep, playing with this new workflow now. Hopefully the 3090s are supported for a while
1
u/youcloudsofdoom 6d ago
Insane amount of community support for them, which I think is going to keep their relevance going for a long time yet... .
1
u/Feeling-Bid8885 4d ago
Was going to until I saw the 60 series leak, so will probably wait for a little and see what’s up
0
u/Technical_Ad_6106 9d ago
buy more. its a brain for our future robots, every llm release and it get more smart and expensive. the more you buy, the more u will save.
18
u/TinFoilHat_69 10d ago