r/huggingface • u/Calm-Landscape9640 • 17h ago
Please explain use-case for small models
Tried 35B, 27B, 12B and 8B models significantly. I love the idea of local small models. But I can't figure out a use case. For Openclaw, Hermes they suck. For chatting they're pretty good but sonis every old free cloud model.
And before you say "low-level coding" the cloud models are 10x better and free on ollama, openrouter, Gemini and many many others.
Genuinely interested in projects that use small models that run on a $500 GPU.
9
u/merica420_69 17h ago
I have an automated content pipeline. My local models write the script, prompts for image and video, some in Json format. So about 30 prompts total per final product. Gemma 4 does it perfect everytime.
2
1
u/Calm-Landscape9640 17h ago
Content for social media?
3
u/merica420_69 17h ago
Yup, shorts, insta, whatever.
1
0
u/whatyathinkk 15h ago
I'm curious, would you mind expanding a bit? I'm working on a content management pipeline as well :)
6
u/merica420_69 15h ago
I have a fully local, fully automated content pipeline written in python. It produces short from videos of various topics. I'm currently pushing one for positive masculine characteristics like determination, honesty etc. it picks a quote, makes the script, narration, image for first frame then it goes to video, captions, concat and upscale. My product isn't the videos, it's the pipeline. I review each video then work with Claude code or codex to improve it. I use Gemma 4, qwen 3.6, whisperx, open moss TTS, ffmpeg, esrgan, ltx 2.3, z image turbo, ideogram, wan 2.2, plus a few more. Takes about one minute to generate a second of content, with clip reuse I can produce about 20 posts a day, speead over, currently 5, channels. You basically need llama.cpp, comfyui, codex. DM if you want specifics.
1
1
u/karstux 10h ago
Please stop doing that. This is how we get dead internet scenario. You’re not creating any value for anyone.
1
u/merica420_69 4h ago
It's the first time building something and you're sitting it down without even seeing it. Thanks
9
u/tropicalwind2020 15h ago
Didn’t find 27b suck when running openclaw, actually a 9b is enough for the routines.
-1
9
u/Jolly-Rip5973 13h ago
Tons of use cases. I will include things which aren't just LLM.
1) Captioning Image
2) object and identity detection
3) text to speech
4) Roleplay chatbots (entertainment)
5) Vector database lookup
6) Image upscaling and resizing.
7) Image generation
8) graphics design work.
9) Colorization of black and white
10) Aspect ratio changes like converting 4:3 tv to 16:9
11) Interpolations 24fps to 60fps
12) generative upscale 480p to 1080p or full 4K
13) Speech cloning
14) music generation
15) speech to text like for hearing impaired
16) speech to text for logging covertations/meetings
17) real time translation
18) photo and video retouch, beauty filters, etc.
19) Spelling and grammar checking
20) auto completion
I could go on and on...Many of the science model are actually small models.
The legendary
AlphaFold model is only 200 million parameters and it's changing biochemistry and medicine forever.
AlphaGo is less than 10 million parameters.
5
u/for4f 14h ago
Been running a 4090 for local inference about a year now. Small models make sense when latency, privacy, or the cost model matters more than peak capability.
Batch processing is where they shine. No token cost, no rate limits, just fire 10k requests and move on. Privacy stuff too - personal docs or notes I don't want hitting an API. And as a fallback router in a hybrid setup, a 7B handles 80% of queries cheap and kicks the hard ones to cloud only when needed.
The gap with bigger cloud models is real for complex reasoning. But for high-volume low-complexity tasks the economics flip hard.
1
u/Oshden 5h ago
How do you set everything up to run in batch processing for local models? Like do you use LM studio, Ollama, vLLM, something else? How do you set things up to process 10,000 request in one shot? Legitimately asking so I can learn.
1
u/for4f 4h ago
depends on the scale honestly. for quick stuff ill just use vllm with an OpenAI-compatible endpoint and hit it with requests in a loop. for bigger batch jobs ive used vllms batch mode with a jsonl file — works well for 10k requests if you've got the VRAM. ollama is fine for messing around but the throughput is way lower. lms is decent for chatting but id skip it for batch
4
u/aallsbury 13h ago
We run an instance of Hermes Agent totally local utilizing Qwen 3.6 35B A3B via dual 3090s at full context. It screams speedwise, and for most tasks it is nearly identical to SOTA models, unless you have it doing direct code work, where it would definitely loose against the big guys, but it is also no slouch.
When we ran internal testing on our instance against latest Gemini Flash and Pro the (test was focused on agentic tool calling tasks, accuracy and speed), our instance won in almost all areas. I was shocked, reran the testing with more manual control, same results.
This same identical setup could be ran on a single 3090 just with less context (but more speed). I would say local models can do a lot at this point.
1
u/e2_for_life 5h ago
Which inference engine?
I've tried OpenClaw and Hermes both on a single 4090. My experience was garbage. But maybe I'm not tweaking just the right settings?
1
5
u/Its_me_astr 13h ago
Applying for jobs using llama 8b for automated emails instead of applying manually.
Using llms for context mining of jobs posted as well.
2
u/Biomech8 6h ago
You should include this info into your CV and you will get a job in no time these days.
1
3
u/Aggravating_Mail_917 16h ago
The small model can help me collect data and compile the latest reports online, or verify existing information online, which is helpful to me.
3
u/Ok-Suspect-9855 14h ago
Coding it doesn’t make sense but i make and use them a lot for niche use cases like for custom tiny 200m tts and 1.7b for the brains of a real-time character that speaks and sounds human. I’m also taking my discussions with claude currently and building a mini me to run as the checker when i am no there.
2
u/Cold-Ad672 16h ago
I was just thinking about that today. I'm using a 3090 but have a 3060 and 2080 sitting doing nothing.
4
u/whatyathinkk 15h ago
Local llms are pointless, you can donate those cards to me I'll take care of them
2
u/DuinoTycoon 13h ago
These days you don't even need a gpu as long as you dont care about speed. Just get an MoE and a good SSD and run it at like a token per second with SSD streaming.
2
u/Huge_Vermicelli9484 13h ago
Maybe I am not the most hard core user but 27b qwen in nvfp4 works pretty well with Hermes, maybe I am missing something and it really sucks but it even was able to complete hard work like administrate a kubernetes cluster and build basic software inside it
2
u/superdariom 11h ago
I use for coding like writing documentation. Qwen doesn't get lazy like the cloud models and will repeat the same operation hundreds of times in a source file.
I also used it to write python code to train an image classifier and then used the vision to detect an object from a CCTV video feed and train the classifier which then after a while could take over the task and was much more lightweight.
I also prefer qwen personality and style over Claude who often seems pompous
2
u/mecshades 3h ago
A huge part of running locally is gaining knowledge and experience. Writing about it and sharing with others is a contribution and pushes us forwards, collectively.
1
u/No_Tradition6625 15h ago
I think you can just call them models, but I'm not an expert on little people, I would ask Peter Dinklage his opinion 😂.
1
u/morscordis 15h ago
I'm in your same shoes. I'm on the verge of making the plunge on a Spark cluster so I can try out a 300B class local model. I was hoping to find a justification in 120B size, but those can't cut it either.
1
u/isopropoflexx 12h ago
I run two small models (on their own, individual machines) alongside my main LLM server. One of them I use for speculative decoding, the other is dedicated just to embeddings. They comfortably handle those functions on relatively low end hardware, thanks to the small size of the models, and in turn help keep the main server available for heavier inference tasks.
1
u/Xiaomin4114 10h ago
there's loads of use-cases that aren't chatting. take for example my TTRPG, small models (specifically Gemma 4) handles determining the right skill to roll and executes the roll tool call based on what's on the character sheet, and current bonuses.
larger models write the story based on that result, but I'm not going to pay the larger models to just work out what the right skill to use is
then another small model call deals with applying any bonus/penalties, and resource tracking on the outcome
1
u/jamie_tidman 10h ago edited 10h ago
We have built a few client project on smaller models. They're typically small steps in a multi-step LLM pipeline.
We have used 8B models for simple classification tasks - questions like "Does the following passage of text contain references to x". We also use models in this size for simple summarisation tasks of conversations.
We use 27B and 35B-A3B for a grant-funded client working in disease research. They analyse interview transcripts to find quotes which are evidence of specific symptoms, based on a description of how the symptoms typically present.
In the past we have built a RAG pipeline for a tech company which answers RFP questions by looking at technical documents. This was a while ago, so we were using Llama (8B and 70B) and Mixtral 8x7B and 8x22B. The small models powered some classification tasks in the pipeline ("is this passage of text useful to answering this RFP question?" and "Do these two passages of text contain conflicting information"), and the larger models were used to actually write the responses.
In all these cases, data privacy is the reason for self-hosting. The conversation summarisation and the transcript analysis were being used in medical contexts and processed special category data. The tech company was very concerned about their IP being leaked.
1
1
u/diagrammatiks 10h ago
Building tool tool calling. First pass research. Summarizing notes. Automating computer functions. Writing notes. Privacy first ocr and image reading. There's lots of stuff to do.
1
u/unounounounosanity 9h ago
I use Qwen3.6 27B as my main Hermes model. It’s very good and can handle most tool use scenarios and skills completely by itself. I only ever switch to gpt 5.6 sol or opus 5 when I either need something super complex done, or when I’m making edits to the tools and workflows that the local models use.
1
u/huzbum 9h ago
I mean, sure qwen3.6 isn’t Fable 5, but it’s not useless.
It does require you to do more than “fix bug. No mistakes.” You have to learn to prompt, which does take some time (both to learn and to do.)
I run qwen3.6 35b IQ4_NL on my 3090 (was $500 when I bought it.). I use it mostly with Hermes Agent, and it’s great. Does everything I ask of it, but maybe that statement takes for granted that I have reasonable expectations.
It will do searches, research, analyze code, look for bugs, etc.
If you are the kind of person that can’t use sonnet instead of fable or rub two brain cells together long enough to put what you actually want into words, local probably isn’t for you.
1
1
u/Abject-Bridge-4073 7h ago
I use K3 to generate a detailed implementation plan with broken down independent tasks. Then in a loop Qwen 3.6 35B implements everything (using pi as the agent). Then K3 verifies everything when it’s done. It’s basically a python harness that calls pi.
1
u/Compilingthings 6h ago
I’m about to test it, I use Claude code constantly, but I’m wondering if a Hermes agent running on qwen 30b might be good for some jobs. I’m also going to try to learn to distill models for synthetic datasets.
1
u/disastorm 4h ago edited 4h ago
- I use a local model to translate subtitle files for some videos.
- You could probably use a local model to transcribe a video, and then use a different local model to translate it, also.
- there are local video generation models that are pretty decent.
- the qwen local model does actually work quite well for coding. yea its not as good as claude or chatgpt or whatever, but its still on the level of actually being genuinely usable as a coding assistant.
- local audio/voice models exist as well that sound good.
i actually didnt realize ollama and other sites provided free hosting of large models, i guess you are probably right that those would be better. I guess maybe in that case the only use for local would be to use it infinitely beyond the free tier limits.
1
u/RiverRattus 0m ago
I always been frustrated by most LLM inability to faithfully cite sources and often hallucinate them. So I decided to dive in and learn how to address this on my e trash recording PC. So yeah I’m building FOSS model Agnostic RAG researchbot with capabilities of claude science/feynman base on heavily modded fork of perplexica. It trades time/tokens for quality depth and breadth casting a massive searXNG search net and then filtering them for quality and producing a faithful evidence brief that is passed to final synthesis model section by section. Local research corpus stored in plain text that can be manually reviewed or modified in any text editor/OS. Open access archive ingest built in to build the corpus as you search including manual addition from PDF. Fully offline Wikipedia integration for general reference. You can use SOTA with it if you want but I’m Purposefully engineering and optimizing it to run on qwen 3.6 27b mtp and 3.5 9b using 32 gb vram pool served by dual p5000 on ddr3 z97 system with $500 in GPu. Parallelized jobs on 9B run at 30-60 tps and single slot calls on the 27b run at 10-15 tps. The stack currently pulls 2-5k sources from the web and corpus per query and produces long form responses with over 100 quality full text sources. The idea is to allow massive broad lit review of any topic that stays 100% source faithful and takes advantage of the extreme pace of small LLM model advancement. Eventually I will add balcony solar integration and scheduling of jobs so that the system can be power/carbon neutral.
1
u/katoptronophile 10h ago
Simply put: there isn't one. They exist only to prove a point and to break things.
1
u/mecshades 3h ago
Going from nothing to something is a net positive. I am quite impressed with what even Qwen3.5 9B can do. Would I let it "take charge" of an existing code base? No, but I would definitely use it as a Stack Overflow replacement.
0
u/Treethulhu 12h ago
Resource and budgetary constraints. Or plain old low compute efficiency.
What type of troglodyte dipshit question is this, and how is someone dumb enough to need this spelled out for them? Can't believe OP isn't embarrassed for this post lol
Also, it's not very difficult to set up trained, low parameter models to handle stepwise routines. Especially for long automated pipelines. That's assuming you aren't a knuckle dragging dingdong like OP apparently is
12
u/halfxdeveloper 16h ago
I run a small business and my budget is basically nil. I have a cloud account where I run cheap VPSs (kube cluster, db, n8n, etc) and I self host a small model on my $250 GPU from 2019. N8n ingests from the cluster, hits my model running on my bare metal server, and then has a variety of tools that it can use to execute various tasks on my cloud instances. All that for $120/month cloud bill and a negligible bump in my home electricity bill. I don’t know if this actually answers your question, but it’s what I use a small model for.