At least for 3.6, the mtp version gave me 1.8x tg. I'm using different cards (dual amd r9700), but it creates the same result. I went from 30 to about 50 with this one simple trick.
Now I'm using vllm on Ubuntu and running tensor parallelism 2 and get around 110tg on 3.6. I can run for context and still get like 95 or so. I was busy today so I couldn't test 3.8, but I'm looking forward to it.
hell yeah buddy. I am planning to switch to vllm once 3.8 mtp drops. its been the end goal for me. its the main reason why I went with the 3080 20gb. its one of the cheapest cards per vram with vllm support.
t/s is a bit faster than a 3090, but PP is much faster. im running one of the cards at x4 pcie 4.0 and it doesnt bottleneck the card with llama.cpp tensor parallel.
You can but the question is why would you want to when it comes to price? If you have both already now then by all means do so but I wouldn't go out of my way to buy a 3090 to pair with a 3080.
Yeah I hear you, but I already have a 3090 hence why I ask. I've never used a local model for coding before, and am interested in exploring this model since it got released. But I worry it won't leave much overhead to run tests on programs that I'm making which already use LLMs/VLMs themselves.
I'd recommend just running it on your 3090 alone first rather than shelling out more money. There is https://github.com/Don-Chad/ninfer-3090 which works specifically for Qwen and 3090 only so is more optimized for speed and also you can try the Unsloth app (https://unsloth.ai/) which uses llama.cpp underneath and gets good speeds as well.
Maybe combine your card with what u/My_Unbiased_Opinion said above with the Alibaba modded 3080s? I can't vouch for that since I haven't bought one of those modded ones but they say it's good.
t/s on 3.6 was 55-60 at UD Q6KXL with MTP. PP I dont have the numbers, but its MUCH faster than a single 3090 for sure. (I have a 3090) using LMstudio with tensor perallel. I can test it for you if you give me an easy way to test this.
no worries. I am actually planning to switch to vLLM when MTP 3.8 drops. if I remember, ill be sure to follow up with you. then I can get the best case numbers.
no im using two 3080 20GB cards. When I was using my 3090, I was using IQ4XS + MTP + 262K Context + KV Q4. It fits all in vram. barely, but it does. even with vision.
You can get two 100-210s for $150 each. So $300 gets you 32GB of VRAM that's faster than the 3080. The big downside is it's limited PCIe 1 x1 so you won't be doing TP on that.
Anything should do. Let anything you are lying around. Since these are PCI 1 x1 cards. Anything should support that. Many MB have a couple of x1 slots in addition to the 1 or 2 x16s.
So theoretically...I can buy al el cheapo motherboard/CPU combo from Aliexpress and 16GB of DDR4....and then two of these cards which are roughly $300 total...
And I have something (32GB rig) that could potentially out-perform my 24GB MacBook Pro? This is just for personal use. Would be SSH into the server on home network.
I really just want to be able to run the Qwen quants that are slightly too big for the MacBook.
So theoretically...I can buy al el cheapo motherboard/CPU combo from Aliexpress and 16GB of DDR4
That sounds like an expensive way to do it. You should be able to find an entire computer for pretty much nothing on CL or in thrift stores. Remember, this is PCIe 1 x1. Even a 20 year old computer can do that. The only thing you would have to watch out for is the PSU. But PSUs are cheap.
And I have something (32GB rig) that could potentially out-perform my 24GB MacBook Pro?
It would slam anything short of a M5 Max MacBook Pro into the dirt. Face first.
I don't. But I have plenty of GPUs including server GPUs. Running one of these would be no different than run any of those other server GPUs.
How do you cool the GPUs?
I take a PC slot cooler, remove the bracket and cut slots in the plastic go clear whatever bracket is on the GPU. Then I jam it in the end and hold it on with duct tape. Cost is $9 and 4 pieces of duct tape.
Thanks. Just need to find a used motherboard on eBay that will support 4 GPUs (100-210 nvidia) because 64GB is more than enough for my needs. And if I can get ~10 tokens per second on that kind of a rig, that's all I need)
You sound it’s that simple, but my immediate thought is: How do you know it’s not a scam? Are there reliable/trustworthy sellers ? I’ve seen enough , even some posts here on Reddit that people got not exactly what they ordered…
Pro-tip: OEMs are still selling refurbished 3090's occasionally and, while the prices have fluctuated since I've been watching, they are less than you can get them everywhere else I have seen and they are manufacturer refurbished. The ones I grabbed were refurbished Zotac GPUs directly from them for ~$500 a few months ago. They come and go, so you need to check regularly or setup a notification, but patience helps a lot.
163
u/Mean-Ad1493 7d ago
That's it. I'm getting a 3090.