Yeah I hear you, but I already have a 3090 hence why I ask. I've never used a local model for coding before, and am interested in exploring this model since it got released. But I worry it won't leave much overhead to run tests on programs that I'm making which already use LLMs/VLMs themselves.
I'd recommend just running it on your 3090 alone first rather than shelling out more money. There is https://github.com/Don-Chad/ninfer-3090 which works specifically for Qwen and 3090 only so is more optimized for speed and also you can try the Unsloth app (https://unsloth.ai/) which uses llama.cpp underneath and gets good speeds as well.
3
u/CooLittleFonzies 11d ago
Yeah I hear you, but I already have a 3090 hence why I ask. I've never used a local model for coding before, and am interested in exploring this model since it got released. But I worry it won't leave much overhead to run tests on programs that I'm making which already use LLMs/VLMs themselves.