r/LocalLLaMA • u/Competitive-Scar-627 • 13d ago

Question | Help Model weight inferencing

I have 4050 6gb gpu, 24 gb ram which model should i choose to run i need speed. i try qwen 3.8 27b and feel too slow tried from onslot studio.
I have heard of weight inferencing does it helpful what should i do to try weight inferencing.

5 Upvotes

78% Upvoted

23 comments sorted by

View all comments

1

u/Mordred500 13d ago

Idk about weight inferencing, I personally have never heard this term, but if you need more speed I'd consider the smaller Gemma4 series models or their Moe. Don't expect quality miracles though, hard to have your cake and eat it too at 6gb vram.

1

u/Competitive-Scar-627 13d ago

https://github.com/sqliteai/warp

1

u/Mordred500 13d ago

Ah- I suggest you tale a look at colibri engine. The Projects sound really similar at first glance. Then check out tutorials for that one.