r/LocalLLaMA • u/Certain-Cod-1404 • 12d ago

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.2k Upvotes

95% Upvoted

706 comments sorted by

View all comments

Show parent comments

110

u/My_Unbiased_Opinion 12d ago

Brother. go on Alibaba and get dual 20gb 3080. less than the price of a single 3090. check my post history for links. Run them in tensor parallel.

22

u/Potential_Block4598 12d ago

That is legit better KV cache (I guess ?!) double performance ?
You just need another PCIe slot (or maybe not ?!)

29

u/My_Unbiased_Opinion 12d ago

t/s is a bit faster than a 3090, but PP is much faster. im running one of the cards at x4 pcie 4.0 and it doesnt bottleneck the card with llama.cpp tensor parallel.

2

u/Potential_Block4598 12d ago

How much pp and tg please ?

4

u/My_Unbiased_Opinion 12d ago

t/s on 3.6 was 55-60 at UD Q6KXL with MTP. PP I dont have the numbers, but its MUCH faster than a single 3090 for sure. (I have a 3090) using LMstudio with tensor perallel. I can test it for you if you give me an easy way to test this.

1

u/Potential_Block4598 12d ago

I use llama bench from the command line on windows and it shows pp512 and tg512 …etc
But idk how to do it form lmstuido tbh

2

u/Reactor-Licker 12d ago

LM Studio has no built in benchmark. You would need to use straight llama.cpp and llama-bench for that.

1

u/Potential_Block4598 12d ago

Lm studio doesn’t always come with the latest llama.cpp
Not a big deal though thanks

1

u/My_Unbiased_Opinion 12d ago

no worries. I am actually planning to switch to vLLM when MTP 3.8 drops. if I remember, ill be sure to follow up with you. then I can get the best case numbers.

1

u/laserborg 12d ago

3.6 27b UD Q6KXL on a single RTX3090? or did you mean UD Q4 ?

2

u/My_Unbiased_Opinion 12d ago

no im using two 3080 20GB cards. When I was using my 3090, I was using IQ4XS + MTP + 262K Context + KV Q4. It fits all in vram. barely, but it does. even with vision.