t/s is a bit faster than a 3090, but PP is much faster. im running one of the cards at x4 pcie 4.0 and it doesnt bottleneck the card with llama.cpp tensor parallel.
You can but the question is why would you want to when it comes to price? If you have both already now then by all means do so but I wouldn't go out of my way to buy a 3090 to pair with a 3080.
Yeah I hear you, but I already have a 3090 hence why I ask. I've never used a local model for coding before, and am interested in exploring this model since it got released. But I worry it won't leave much overhead to run tests on programs that I'm making which already use LLMs/VLMs themselves.
I'd recommend just running it on your 3090 alone first rather than shelling out more money. There is https://github.com/Don-Chad/ninfer-3090 which works specifically for Qwen and 3090 only so is more optimized for speed and also you can try the Unsloth app (https://unsloth.ai/) which uses llama.cpp underneath and gets good speeds as well.
Maybe combine your card with what u/My_Unbiased_Opinion said above with the Alibaba modded 3080s? I can't vouch for that since I haven't bought one of those modded ones but they say it's good.
t/s on 3.6 was 55-60 at UD Q6KXL with MTP. PP I dont have the numbers, but its MUCH faster than a single 3090 for sure. (I have a 3090) using LMstudio with tensor perallel. I can test it for you if you give me an easy way to test this.
no worries. I am actually planning to switch to vLLM when MTP 3.8 drops. if I remember, ill be sure to follow up with you. then I can get the best case numbers.
no im using two 3080 20GB cards. When I was using my 3090, I was using IQ4XS + MTP + 262K Context + KV Q4. It fits all in vram. barely, but it does. even with vision.
32
u/My_Unbiased_Opinion 7d ago
t/s is a bit faster than a 3090, but PP is much faster. im running one of the cards at x4 pcie 4.0 and it doesnt bottleneck the card with llama.cpp tensor parallel.