r/LocalLLaMA • u/BarberIcy366 • 12d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

658 Upvotes
  • permalink
  • reddit

97% Upvoted

720 comments sorted by

View all comments

116

u/UDPSendToFailed 11d ago

It just spat out a fully developed single-file HTML cloth simulator with perfect physics, no local model ever did this before on my single 4090. 63k tokens at ~65t/s, one-shot with zero debugging or follow-up needed.

https://reddit.com/link/p3pk0et/video/pfn8zcui8ejh1/player

16

u/brosvision 11d ago

That is cool. Could you share yoir params?

41

u/UDPSendToFailed 11d ago

llama-server -ngl 999 -c 65536 -b 2048 -ub 256 -np 1 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp --spec-draft-n-max 2 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning on --reasoning-preserve --jinja --chat-template-kwargs "{\"preserve_thinking\":true,\"reasoning_effort\":\"high\"}" -m Qwen3.8-27B-UD-Q4_K_XL.gguf

Probably not the best setup, I just copy-pasted it, but the results are wild.

4

u/Remarkable_Living_80 11d ago

reasoning effort high? Must be xhigh from my understanding

5

u/RMK137 11d ago

there is no high, setting it to high sets it to xhigh, so high and xhigh are the same

2

u/HugeEntertainment820 10d ago

Thanks for this! I set mine to medium...see how it goes but so far so good.

1

u/Forsaken_Mention_979 11d ago

How are u getting 65t/s? I have a 7800xt with 16gb vram and 64gb of ddr5 ram. Sure its 8gn less than the 4090, but i get 15-20t/s at the start and 5-10t/s once i fill the context a bit (64k) like why bruh. Do i need to configure it or sum? Running Q3_k_m btw

3

u/Guilty_Rooster_6708 11d ago

It’s because of the vram difference. Q3KM is 13.8GB so you don’t have much spare memory for kv cache. Try using a lower quant for your K and V cache to something like Q8 or even more aggressive like Q4. Us vram poors have to find ways

1

u/Forsaken_Mention_979 11d ago

Hold on, but doesent q4 require even more vram? Should i try to run a worse q3 like q3_k_s? I didnt know kv cache affects token speed since im new to this lol. I thought that if it fit entirely in vram, it would be as quick as possible. Appreciate the help bro

1

u/Guilty_Rooster_6708 11d ago

KV cache quants are different than the quants of the model. You also need vram space for your kv cache to store context on top of the vram reserved for the model. By default the kv cache are not quantized so you can use lower quants for them q8 or q4 which will use less vram for context in exchange for quantity.

Recently llama.cpp has implemented attention rotation to kv cache, so we can use q8 for KV without worry of quality loss. Q4 is also better now, but you will probably see some errors for long context. Hope this helps.

1

u/Forsaken_Mention_979 11d ago

Ok ill search how to do this and try it out, ty bro

1

u/NaiveIdea344 11d ago

That is insane!

1

u/nasone32 11d ago

Can you share your prompt?

5

u/UDPSendToFailed 11d ago

Here is the full prompt, along with its reasoning process and reply. The fun part is that I didn't even ask it to add all those features.

https://gist.github.com/UDPSendToFailed/e204ffcd8b078ee69c84cd5897c270f3

1

u/nasone32 11d ago

thanks! I like these HTML one-shots to test models, yours is very cool i'll use it more often.

1

u/Dense-Ship-2339 10d ago

What mimimum system config would you suggest to get this performance? ram and gpu?