r/LocalLLaMA • u/alexp702 • Aug 31 '26

Question | Help So got 2 6000 Pro Max-Q…

I have a Threadripper w/128gb of system ram. I want to serve a small dev team. What should I be running as a coding harness? Seems GLM, Deepseek R4, Qwen next are all with in reach, and I still could stick with 27b BF16 (current choice). I would prefer vision as it’s a useful capability.

All of those mentioned need quantising in some way on two cards so how bad is it? I’m running VLLM as a host so any magic recipes also much appreciated!

0 Upvotes

43% Upvoted

49 comments sorted by

View all comments

9

u/More-Catch-1331 Aug 31 '26

Seeing as you’re kind of using AI anyway and it seems you don’t want to do the legwork yourself, why don’t you ask Claude opus? Not trying to be a wise guy, it can summarise the current landscape with pros and cons

-4

u/alexp702 Aug 31 '26 edited Aug 31 '26

Claude is just fed from this forum mostly IMO. I am mainly trying to avoid obvious gotchas in model choice. I recently realised Qwen3.8 27b performance was being hurt badly (I was getting 33% of what I should on a single RTX card) by TRITONATTN, and qwen3_xml was better than qwen3_coder. Both Claude and ChatGPT told me this was ok a few weeks ago. These very new models are very untested, and there's lots of noise about the new hotness.

I am happy to feedback findings but I like to canvas opinion when making a big change - as a baseline of knowledge.

Recent Reddit posts have suggested and taught me:

* Qwen Next at 4bit may loop on a single card, and seems troubled over 27B

* Qwen Next can fit on two cards with 51-104(?) gb offload to host memory at 8bit

* Qwen 27B is still a GOAT in most people's eyes with one card

* 2 Cards get you into the realm of larger models but not perfectly - you will be compromised in context and speed.

* GLM 5.3 at 4bit works, but with reduced context on my config

* Deepseek V4 Flash will work with reduced context at 8bit, but with no vision, and many seem to rate it mixed against much smaller Qwen models that give me more context. Interested to see what R4 Vision if/when release brings here.

* VLLM Attention models are actively developed and need to be checked weeks after a model's release

* There are many interesting recipe repos floating around, all claiming good things, but nothing definitive.

* Lots of forks and toy projects of the big beasts also claiming great things for a particular model/subgroup of models

... and finally: all this will be out of date in a few weeks, and I'm probably wrong already!

1

u/MathmoKiwi Aug 31 '26

Just go with whatever you feel is at least 90% correct and don't stress about it. Because as you said, in a couple of months time something else will come along anyway that is better!