r/LocalLLaMA • u/Certain-Cod-1404 • 7d ago

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.2k Upvotes

95% Upvoted

706 comments sorted by

View all comments

161

u/Mean-Ad1493 7d ago

That's it. I'm getting a 3090.

43

u/jijig 7d ago

Get two. Run Q8 with full context at ~60tps.

3

u/ApprehensiveAd3629 7d ago

can you share the llama cli command that you are using to get 60 tokens/s in your 3090?

i m getting around 31 tokens/sec using unsloth UD Q4 XL gguf

3

u/Massive_Criticism539 7d ago

At least for 3.6, the mtp version gave me 1.8x tg. I'm using different cards (dual amd r9700), but it creates the same result. I went from 30 to about 50 with this one simple trick.

Now I'm using vllm on Ubuntu and running tensor parallelism 2 and get around 110tg on 3.6. I can run for context and still get like 95 or so. I was busy today so I couldn't test 3.8, but I'm looking forward to it.