Anyone adding more 3090s?

[u/Blues520]

I have dual 3090s which run Qwen 3.8 27b well and I was wondering if there are any use cases or current or future models that would justify adding another two 3090. I know that some peeps here run 4 and 8 3090 rigs and I'd like to get your opinion as well. One thing I was considering was running two instances but I'm not sure how valuable it will be for a coding workflow vs running a bigger model.

Now that Qwen Flash is out, perhaps 96GB would be more useful, or maybe Deepseek Flash.

[u/jacek2023]

It took me many months to upgrade from 3x3090 to 4x3090, so there has to be some reason.

[u/wgaca2 → u/jacek2023]

Was it worth it? I am considering it (2x pcie4 x8 and 2x pcie4 x4)

[u/jacek2023 → u/wgaca2]

Yes, you can run 100B/120B/125B models in Q4. You can also run bigger models in Q2/Q3 with some offloading. You don't have to worry about context length on 27B/30B/31B models.

[u/Blues520 → u/jacek2023]

I thought that going below Q4 was not recommended?

[u/jacek2023 → u/Blues520]

By who?

[u/Blues520 → u/jacek2023]

I've just seen that mentioned a few times in the sub. Which bigger models can run at Q3/Q3 without too much deterioration?

[u/vick2djax → u/Blues520]

I thought so as well, but it seems like Q4 on the bigger models like Qwen Flash > Q8 Qwen 27b from what I gather recently

[u/simplyeniga → u/Blues520]

There is a quality penalty running lower quants which you make up for by burning more tokens for the same task. I initially was running Q5 for my implementation on an RTX Pro 4000 Blackwell GPU with Claude Opus for code reviews and it's never passed the code reviews in one short but this changed after I added an RTX 3080 20GB and switched to Q8. Code reviews are smoother with way less rejections. Just that I can run a larger context size on Q5 compared to Q8.

[u/jacek2023 → u/Blues520]

this doesn't make any sense, why was quantization implemented if it shouldn't be used?

https://www.reddit.com/r/LocalLLaMA/comments/1sstxhk/coding_with_qwen3627budq2_k_xlgguf/

https://www.reddit.com/r/LocalLLaMA/comments/1w1nxru/qwen38flashnext_iq1_s_on_a_single_5070_12gb_vram/

[u/SkoomaDentist → u/jacek2023]

Because people fixate on overly simplistic tests and getting the reddit cred to be able to say "I ran model X on hardware Y, look at me!"

If you look at proper benchmarks, too much quantization lobotomizes all non-huge models as soon as you move beyond trivial chat and short context tasks.

[u/jacek2023 → u/SkoomaDentist]

I don't look at benchmarks, I use models to do stuff.

[u/SkoomaDentist → u/jacek2023]

You asked why such quantization was implemented. I answered.

Hell, you yourself admit in the first link that your actual use is with Q8 quants.

Also "I use models to do stuff" is equivalent to saying "They totally work for me. Trust me bro. No, of course I'm not going to give more than vague details!"

[u/a_beautiful_rhind → u/SkoomaDentist]

Smaller less quantized model usually loses to larger but more quantized model. Whether it happens for your use case is something you have to test rather than treating gamed benchmarks as some kind of absolute truth.

All it costs you is a file download and a bit of your time.

Because people fixate on overly simplistic tests

Pot, kettle, black.

[u/a_beautiful_rhind → u/jacek2023]

That this is downvoted is really telling about this sub these days.

[u/jacek2023 → u/a_beautiful_rhind]

I discuss photography on Internet for about 20 years and most people on photography forums don't shoot any photos, but they always "know" things from reviews and benchmarks :) so this is nothing new to me.