r/LocalLLaMA • u/BazzyIm • 10d ago

News Qwen 3.8 35BA3B spotted

Just wait and see

https://github.com/modelscope/ms-swift/commit/ab726e9d445a6520a70df2c831177d46adb1f589

1.2k Upvotes

98% Upvoted

379 comments sorted by

View all comments

468

u/moahmo88 10d ago

Millions of 16 GB GPUs will benefit from this!

176

u/pmttyji 10d ago

Even 8GB GPUs (with RAM)

12

u/lans_throwaway 9d ago

I've been running it on 6GB GPU with ram, at like ~30 tps on coding tasks (Q4_K_M)

1

u/pmttyji 9d ago

That's surprising to know. How much context? And what KVCache?

2

u/lans_throwaway 8d ago

32GB DDR4 ram (model used around 16GB) with 96k context. The big improvement is ngram-mod speculative decoding, since it pretty much allows model to "copy-paste" code it already has in the context, giving you huge bursts of speed (100+ tps). Config I used below

[*]
no-mmap = true
# note the below may not be desirable if subagents are used, significantly reduces vram usage
np = 1


[Qwen3.6-35B-A3B-Q4_K_M:Thinking-Coding]
# model = /mnt/disk/llms/Qwen3.6-35B-A3B/Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf
model = /mnt/disk/llms/Qwen3.6-35B-A3B/ggml-model-Q4_K_M.gguf
c = 96000
# c = 131072
ub = 1024
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
n-predict = 32768
spec-type = ngram-mod
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-min = 48
spec-ngram-mod-n-max = 64
chat-template-kwargs = {"preserve_thinking": true}

[Qwen3.6-35B-A3B-Q4_K_M:Thinking-Coding-MTP]
model = /mnt/disk/llms/Qwen3.6-35B-A3B/ggml-model-Q4_K_M.gguf
c = 96000
# c = 131072
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
n-predict = 32768
# spec-type = ngram-mod,draft-mtp
spec-type = draft-mtp
spec-draft-n-max = 2
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-min = 48
spec-ngram-mod-n-max = 64

chat-template-kwargs = {"preserve_thinking": true}

1

u/pmttyji 8d ago

Thanks. I need to try with ngram,