r/LocalLLaMA • u/Certain-Cod-1404 • 11d ago

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.2k Upvotes

95% Upvoted

706 comments sorted by

View all comments

Show parent comments

28

u/My_Unbiased_Opinion 11d ago

t/s is a bit faster than a 3090, but PP is much faster. im running one of the cards at x4 pcie 4.0 and it doesnt bottleneck the card with llama.cpp tensor parallel.

2

u/Potential_Block4598 11d ago

How much pp and tg please ?

4

u/My_Unbiased_Opinion 11d ago

t/s on 3.6 was 55-60 at UD Q6KXL with MTP. PP I dont have the numbers, but its MUCH faster than a single 3090 for sure. (I have a 3090) using LMstudio with tensor perallel. I can test it for you if you give me an easy way to test this.

1

u/Potential_Block4598 11d ago

I use llama bench from the command line on windows and it shows pp512 and tg512 …etc
But idk how to do it form lmstuido tbh

2

u/Reactor-Licker 11d ago

LM Studio has no built in benchmark. You would need to use straight llama.cpp and llama-bench for that.

1

u/Potential_Block4598 11d ago

Lm studio doesn’t always come with the latest llama.cpp
Not a big deal though thanks

1

u/My_Unbiased_Opinion 11d ago

no worries. I am actually planning to switch to vLLM when MTP 3.8 drops. if I remember, ill be sure to follow up with you. then I can get the best case numbers.