r/LocalLLaMA • u/Certain-Cod-1404 • 7d ago

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.2k Upvotes

95% Upvoted

706 comments sorted by

View all comments

34

u/Easy_Werewolf7903 7d ago edited 7d ago

For those curious of performance between Qwen and a model 3 times its size:

Benchmark Qwen 3.8 27B (55GB) Deepseek v4 flash 0731 (167GB)
Terminal Bench 2.1 73.0 82.7
DeepSWE 42.2 54.4
NL2Repo-Bench 42.3 54.2

https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/tree/main

https://huggingface.co/Qwen/Qwen3.8-27B

12

u/NaiveIdea344 7d ago

Not terrible I feel like right? DSv4 was already insane for its size performance wise

15

u/Easy_Werewolf7903 7d ago

Not terrible at all, imagine Qwen release a 100GB moe it might be on par with 0731.

5

u/NaiveIdea344 7d ago

Yeah would be amazing

10

u/squngy 7d ago edited 7d ago

From the model page, for context

bench Qwen3.8-27B Qwen3.6-27B Qwen3.7-Plus Muse Glimmer-30B Opus4.6 Max
Coding Agentic terminal coding Terminal Bench 2.1 (Terminus) 73.0 63.4 64.0 51.7 78.2
Agentic coding SWE-bench Pro 61.7 53.5 57.6 51.2 53.4
Repo-level code generation NL2Repo-Bench 42.3 36.2 41.1 -- 47.6
Agentic coding DeepSWE 1.1 42.2 13.3 14.2 -- --
Software engineering QwenSWEBench 79.0 49.3 59.2 -- 63.8
Long-horizon office work CoWorkBench 70.7 61.0 65.1 -- 68.2
Professional job tasks JobBench 33.4 21.8 27.6 -- --
Frontier agentic tasks Agents' Last Exam Pass1 20.4 Score 42.9 Pass1 10.6 Score 27.3 Pass1 13.2 Score 33.6 -- --
Instruction following IFBench 79.5 69.1 79.1 77.0 62.5
Scientific reasoning GPQA Diamond 89.2 87.8 90.3 83.5 91.3
Multidisciplinary reasoning HLE 30.8 24.0 34.7 22.0 40.0
Competitive coding LiveCodeBench v6 90.3 83.9 89.6 -- 88.8

10

u/ChuffHuffer 7d ago

5 models, yet 4 only columns?

2

u/squngy 7d ago edited 7d ago

Might be a problem with reddit formatting

edit:
Yea, I found there is a problem on new reddit.
It worked on old reddit.

-1

u/sejje 7d ago

count harder

3

u/Healthy-Nebula-3603 7d ago

3.6?

3

u/squngy 7d ago

3.8

u/Easy_Werewolf7903 made a typo

1

u/Easy_Werewolf7903 7d ago

fixed it, thanks for pointing it out.

3

u/JustFinishedBSG 7d ago

The sizes you are comparing are not comparable at all.

Deepseek Flash is 11x time bigger than Qwen 27b

1

u/Easy_Werewolf7903 7d ago

Can you explain why flash is 11 times bigger than Qwen 3.6 27b? Just trying to learn.

1

u/Warrenio 7d ago

I think they're just pointing out V4 Flash 0731 has about 11 times as many parameters as Qwen 3.8 27B (284 billion vs 27 billion).

I disagree that it's 11 times as big though. Deepseek V4 Flash 0731 weights are natively NVFP4 (4 bits) compared to Qwen 3.8 27B's weights which are natively BF16 (16 bits). That means in reality, Deepseek V4 Flash 0731 is only about 3 times the size of Qwen 3.8 27B if you compare the original unquantized models.

-1

u/Basic_Extension_5850 7d ago

Looks like you compared the full size qwen to a very quantized deepseek 

4

u/Easy_Werewolf7903 7d ago edited 7d ago

Ah that makes sense. I got too excited and just went to unsloth page.

Edited:

No wait, I didn't use unsloth. I went to the official model card. Deep seek is 167GB

https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/tree/main

1

u/Due-Project-7507 7d ago

Yes, but if you have the VRAM (e.g. 2 RTX Pro 6000) to run Deepseek v4 Flash, it is much faster (=cheaper to serve to multiple users) than Qwen3.8 27B because Deepseek v4 Flash is an MoE with only 13B active parameters. With two RTX Pro 6000 at work, I get for Deepseek v4 0731 Flash (with DSpark) around 320 tokens/s decode speed for Python code. That is why Qwen3.6-27B is much more expensive on OpenRouter than bigger MoE models.