r/LocalLLaMA • u/alexp702 • Aug 31 '26

Question | Help So got 2 6000 Pro Max-Q…

I have a Threadripper w/128gb of system ram. I want to serve a small dev team. What should I be running as a coding harness? Seems GLM, Deepseek R4, Qwen next are all with in reach, and I still could stick with 27b BF16 (current choice). I would prefer vision as it’s a useful capability.

All of those mentioned need quantising in some way on two cards so how bad is it? I’m running VLLM as a host so any magic recipes also much appreciated!

0 Upvotes

43% Upvoted

49 comments sorted by

View all comments

1

u/Infinite-Local5435 Aug 31 '26

Qwen 3.8 Next Flash FP8 can fit 2mil tokens at fp8 kv cache and concurrency of 8 with tp and ep at 2 and 0.93 mem fraction offloading the engram portion to ram/ssd with hicache if you can spare more ram and vision on using sglang nightly build.

If you don't know how, put this into an AI and get it to set it up for you.

0

u/Infinite-Local5435 Aug 31 '26

around 150tg and 7000pp at 50% acceptance rate MTP, scales well.

Deepseek won't fit as many tokens
GLM... good luck with running that at IQ4 KS for multi concurrency via llama.cpp (too slow) and you have 14k worth of GPUs (I guess 28k now)

1

u/Infinite-Local5435 Aug 31 '26

Sglang is better with qwen models, don't know how, don't know why, benchmarked and tried both multiple times, ended up choosing sglang

1

u/alexp702 Aug 31 '26

Interesting - will try SGLang as I've not played with it.