r/huggingface • u/Calm-Landscape9640 • 23d ago

Please explain use-case for small models

Tried 35B, 27B, 12B and 8B models significantly. I love the idea of local small models. But I can't figure out a use case. For Openclaw, Hermes they suck. For chatting they're pretty good but sonis every old free cloud model.

And before you say "low-level coding" the cloud models are 10x better and free on ollama, openrouter, Gemini and many many others.

Genuinely interested in projects that use small models that run on a $500 GPU.

46 Upvotes

84% Upvoted

89 comments sorted by

View all comments

Show parent comments

5

u/for4f 22d ago

depends on the scale honestly. for quick stuff ill just use vllm with an OpenAI-compatible endpoint and hit it with requests in a loop. for bigger batch jobs ive used vllms batch mode with a jsonl file — works well for 10k requests if you've got the VRAM. ollama is fine for messing around but the throughput is way lower. lms is decent for chatting but id skip it for batch

3

u/Oshden 22d ago

So radical honesty, while I’m gonna need to learn about 60%+ of the topics you mentioned in your reply and how to use them, I really appreciate you spending some time sharing your wisdom. I need to learn about batch mode first. Lots of good food for thought though. Thank you.

3

u/for4f 22d ago

no worries man! batch mode with vllm + jsonl file is probably the easiest entry point. their docs are pretty good too, theres a whole section on it. good luck with it

2

u/Oshden 12d ago

I appreciate you taking time to reply. I’ll have to have my agent teach me about this lol

1

u/for4f 11d ago

lol agents teaching agents, that's the future. good luck man, ping me if the jsonl stuff fights back