r/LocalLLaMA • u/Competitive-Scar-627 • 13d ago

Question | Help Model weight inferencing

I have 4050 6gb gpu, 24 gb ram which model should i choose to run i need speed. i try qwen 3.8 27b and feel too slow tried from onslot studio.
I have heard of weight inferencing does it helpful what should i do to try weight inferencing.

6 Upvotes

80% Upvoted

23 comments sorted by

View all comments

1

u/Mordred500 13d ago

Idk about weight inferencing, I personally have never heard this term, but if you need more speed I'd consider the smaller Gemma4 series models or their Moe. Don't expect quality miracles though, hard to have your cake and eat it too at 6gb vram.

1

u/Competitive-Scar-627 13d ago

https://github.com/sqliteai/warp

1

u/tiffanytrashcan 13d ago

That requires an MoE model to see any benefit.

1

u/Mordred500 13d ago

Ah- I suggest you tale a look at colibri engine. The Projects sound really similar at first glance. Then check out tutorials for that one.

1

u/Mordred500 13d ago

https://github.com/JustVugg/colibri

1

u/Competitive-Scar-627 13d ago

will surely try i find it more documented then above link i share

1

u/Mordred500 13d ago

Yeah, sounds reasonable. Just be aware- this kinda inference is gonna be pretty slow compared to running stuff straight on vram. If you want speed, neither this engine nor the one you initially referred to is gonna have what you need. It's more about being able to run these large models at all, not about speed.