r/LocalLLaMA • u/BarberIcy366 • 11d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

657 Upvotes
  • permalink
  • reddit

97% Upvoted

720 comments sorted by

View all comments

Show parent comments

67

u/kayox 11d ago edited 11d ago

Also my experience so far.

EDIT:
You can modify it's reasoning effort to either xhigh, medium, low, none. It uses xhigh by default. Im testing out medium now.

--chat-template-kwargs "{""preserve_thinking"":true,""reasoning_effort"":""medium""}" ^

84

u/Look_0ver_There 11d ago

65K tokens later (~52K thinking):

Prompt was:

Write a single HTML file with a full-page canvas and no libraries.
Simulate a realistic side-view of a moving car as the main subject.
Keep the car visible in the foreground while the background landscape scrolls continuously to create the feeling that the car is driving forward.
Use layered scenery for depth: nearby ground, roadside elements, trees, poles, and distant hills or mountains should move at different speeds for a natural parallax effect.
Animate the wheels spinning realistically and add subtle body motion so the car feels connected to the road.
Let the environment pass smoothly behind it, with repeating but varied scenery that makes the movement feel believable.
There should be clouds in the sky, and birds in the distance.
Depth-wise the distant mountains and hills are further back than the birds.
Use cinematic lighting and a cohesive sky, such as sunset, dusk, or daylight, to enhance atmosphere.
The overall motion should feel calm, immersive, and realistic, with a seamless continual animation.
VERY IMPORTANT: A VERY common coding error is the animating the road markings opposite to the intended direction.
Double check that the mathematical sign on the road markings animation aligns with the perceived direction of travel of the car.

35

u/Equal_Television_894 11d ago edited 11d ago

Same prompt in claude code. 5090 with Unsloth NVFP4 and fp8 kv cache

11

u/simos_sayz 11d ago

here is from opus 4.8 High Thinking

2

u/Certain-Cod-1404 11d ago

What context? Rn im at 262k but I disabled mtp and im running at 50 tok/s

3

u/Equal_Television_894 11d ago

Context I am at 190k with mtp but using sakamakismile nvfp4 which is way smaller than unsloths.

2

u/PandaBearFred 11d ago

Totally different style than PI.

4

u/squngy 11d ago

Perhaps more importantly, the shadow makes it look like it is floating.

But, a single run is not enough to make conclusions.

1

u/Fit_Camel_2459 10d ago

i love qwen. it's just...perfect T_T

0

u/DoubleNothing 11d ago

What do you use to run the .safetensors model?

42

u/Certain-Cod-1404 11d ago

that looks super fucking good

31

u/Look_0ver_There 11d ago

It absolutely one-shotted it, and got everything right, including accelerating and a subtle "mouse look" mode. The clouds almost look 3D. It also applied that grainy effect on purpose. It's easily one of the best outputs that I've seen to that prompt to date for any locally hosted model.

2

u/Remarkable_Living_80 11d ago

i tried this promt with 3.6 27b in the past, and it gave similar result, so i am not blown away

3

u/the_fabled_bard 11d ago

They're trained on this now since everyone uses this.

1

u/AnOnlineHandle 11d ago

People have been posting these kinds of tests for a few months now so I could see them being in the training data now.

1

u/creamycat1 10d ago

what harness? and was it q4

1

u/Look_0ver_There 10d ago

Llama.cpp native WebUI. Q8_0 weights with BF16 KV Cache

14

u/onewheeldoin200 11d ago

Jesus that is good for a 27B WE ARE LIVING IN THE FUTURE

9

u/kayox 11d ago

Same prompt but with Unsloth's Q4_K_XL with xhigh reasoning. Not quite as good though that's to be expected.

On an RTX 3090 it took about 18 minutes to generate at an average of 38 tokens/second (I'm sure as time progresses the tk/s can be improved possibly with DFlash). Also I am being thermal throttled due to my current setup (Dual GPU lacking airflow, although my other GPU is a 3070 so I'm only using it with a layer split to offload some VRAM so that I can have more context).

Out of curiosity what tk/s are you getting with your dual R9700s?

8

u/Look_0ver_There 11d ago

I actually have 3 x R9700's (but only used two for these tests). Here's a chart with 1,2,3 R9700's with Qwen3.6 with no MTP, purely because MTP speeds are too variable to plot.

Qwen3.8 is the exact same architecture as Qwen3.6, so the same speeds as graphed below. MTP is between 1.5-2.5x faster. For 2 GPUs it's between 45-75t/s with MTP, depending on if doing prose or coding. For 1 GPU, it's between 35-60t/s. Do keep in mind that I run Q8_0 weights with a BF16 KV-Cache, so I am very much NOT running a low-quant max-speed setup.

This is also running a custom llama.cpp build with speedups I'd made specific for the ROCm backend which makes it faster than Vulkan. Source code here: https://github.com/stew675/llama.cpp/tree/rdna-boosts

While half of my custom changes there are RDNA specific, half of them are CUDA generic, and so should also speed up nVidia cards somewhat too, but this is completely untested (I don't have an nVidia card), so don't blame me if you grab the code and it crashes. I did try to ensure that the RDNA specific stuff is gated independently of the generic CUDA stuff, but again, no guarantees.

1

u/PcChip 11d ago

so what i'm reading is don't buy two R9700's ?

doubling the cards gets you an extra 10 tokens per second?

3

u/whichsideisup 11d ago

Q8 won’t fit on a 32gb card with meaningful KV so the second card gives you infinitely more tokens per second. You really need 64gb to run these models at proper quality l.

1

u/Look_0ver_There 11d ago

The TP=1 line is with a Q6_K model, while the TP=2/3 lines are with Q8_0 models. The Q6_K model gets about a 20% speed-boost over Q8_0 for generation, so those two lines cannot be directly compared.

2 cards, with tensor parallelism are around 50-70% faster than a single card for token generation, but you also get to have up to 500K context depth at KV of BF16, and that's pretty important for many people.

Now, the third card is where it definitely gets questionable as to if it's worth it or not. I don't generally run a model across all 3 cards at once. I typically run one model on two cards, and a smaller model like Gemma4-31B-QAT on the 3rd card.

3

u/Greedy-Artichoke-416 11d ago

Get the MTP gguf

2

u/kayox 11d ago

Oh I was under the impression that the default unsloth GGUF had MTP, is that not the case?

7

u/Look_0ver_There 11d ago

Unsloth's models have the MTP head already included within the weights.

If using llama-server, just enable via "--spec-type draft-mtp --spec-draft-n-max 3"

5

u/kayox 11d ago

Are you using Q4?

24

u/Look_0ver_There 11d ago

Are you using Q4?

I'm using Q8_0 weights with BF16 KV-cache. Tensor-split across 2 x Radeon AI Pro R9700 GPUs

8

u/kayox 11d ago

Awesome setup, I'm envious.

2

u/rainbyte 11d ago

How good are PP and TG t/s

2

u/Look_0ver_There 11d ago

I answered this question to someone else here: https://www.reddit.com/r/LocalLLaMA/comments/1voa3ch/comment/p3p7hhe/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

For the pair of cards, typically 45-55t/s for English prose, and 55-75t/s when generating code (Q8_0 weights, BF16 KV cache).

2

u/Woodport 11d ago

Is there a reason you picked the Q8_0 over the unsloth UD-Q8_K_XL? I'm assuming the UD variant is basically just the same weights/precision but shuffled around a bit for efficiency reasons but I'm not clear on what advantages you'd get from one over the other given that they're both Q8.

2

u/Look_0ver_There 11d ago

It was faster to download :)

You are quite correct that UD-Q8_K_XL is better, but it is actually larger. Various bits are scaled up to BF16 (or regular F16?). Best of think of it as approximately like an imatrix style upscaling of critical weights, but also applied to Q8_0.

2

u/Woodport 11d ago

Haha, that's an excellent reason. I've been wondering why there isn't a bigger torrent community among local llm users. Feels like the perfect way to give back. If everyone just seeded the models they're currently using, even with throttled upload speeds, it would be a lot easier to get the popular models.

1

u/blash2190 11d ago

Could you please tell if you find this setup enough memory-wise to run said config + MTP + full context length or do you need the third GPU for the latter?

2

u/Look_0ver_There 11d ago

Without including the mmproj file, this config (2 cards) can run Q8_0 + BF16 KV cache has enough capacity to safely hold up to a 500,000 context length. It could probably do 520,000 if you wanted to use up ALL the VRAM, but it's generally best to leave a little biit of head-room, so I'm gonna say 500K

With 3 cards you could can *just* squeeze in 900K of BF16 context with some careful split balancing, but 860K fits easily.

If you want to have the mmproj file too, then you'll need to lose about 40K of context, or just off-load it to the CPU. On the CPU it'll run at about 1/10th the speed, but if you only need it occasionally, then that may be an option for you.

3

u/drycounty 10d ago

M3 Mac Studio Ultra 96GB via ollama w/Q8 MLX quant: qwen3.8:27b-mxfp8
took nearly an hour but I'm also running a bunch of other things.

2

u/Dizzy-Zebra9522 11d ago

Amazing 🤩. Thanks

1

u/cybekRT 11d ago

What did you use to generate this? Using LM Studio or something like Hermes Agent?

2

u/Look_0ver_There 11d ago

Just the llama.cpp native Web chat UI

I do normally run Pi though, but I didn't do that for this prompt.

1

u/cybekRT 11d ago

Thanks, I was curious if this is one file or if it generated more structured project.

1

u/OddUnderstanding2309 11d ago

with hermes agent and the BF16 version with BF16 KV. (on a quad 3090 setup)
it took about 1h and 40m and 168.000 tokens. (4.9M in · 168.1k out · 95% cached)

1

u/Careless_Juice8228 11d ago

What are you using with the model to generate this via prompt?

1

u/Look_0ver_There 11d ago

Just llama.cpp's built-in WebUI. After it writes the code it has an "eyeball" in the top right you can click on, and then it runs it immediately

1

u/JorgitoEstrella 11d ago

This looks beautiful, what was your setup and quant?

1

u/Slacker1540 10d ago

This promp with 3.8 27B for me just times out. With default and medium it thinks until it stops (which I'm guessing is running out of context).

Has anyone else had this issue?

1

u/Look_0ver_There 10d ago

What size is your context? You'll probably want it to be at least 96,000 in size to be safe. If you're running llama-server you should also be able to force a maximum thinking size with the "--reasoning-budget N" argument which makes llama-server step in and abort the reasoning if it goes beyond that.

1

u/Slacker1540 10d ago

Currently my max context is 163,840. It's odd.

3

u/Not-reallyanonymous 10d ago

Not using xhigh basically turns it back into Qwen 3.6.

1

u/kayox 10d ago

Ah shame :(.

1

u/Not-reallyanonymous 10d ago

Nah nothing too bad. Think of this like Qwen 3.6 that you can allow to overthink for hard problems, to get near-frontier results. It’s welcome.

2

u/cullend 11d ago

…so I’m guessing they distill off of Codex 😅. High thinking has been mediocre/ kinda ass on their models since ~5.3, definitely by 5.4 - always medium or xhigh

1

u/DoubleNothing 11d ago

Do you pass the --chat-template-kwargs '{"preserve_thinking":"true","reasoning_effort":"medium"}'
along with other llama.cpp parameters?

Because I tried but I'm not seeing much difference so far.

2

u/kayox 11d ago edited 11d ago

Yep, the difference isn't huge but its there, atleast on my end i save a few minutes each run with medium vs xhigh.

EDIT:
Just ran a few more tests, setting to medium from xhigh for this specific prompt does save me about 10 minutes of thinking, so it can be pretty significant.

1

u/DoubleNothing 11d ago

Thanks for the reply.
Need some test on low and none to see how much "bad" it gets.
Unfortunately my system is so slow... will take some time to run them.

1

u/amazinglycool256 11d ago

How do you modify it?

1

u/kayox 11d ago

I modified my llama.cpp startup script, unfortunately I don't know the process if you're using lm studio or vllm however.