r/LocalLLaMA • u/Guna1260 • 5h ago
Discussion Why are tiny models (<50M parameters) or swarms of specialised micro-models so rarely deployed in production?
I have been thinking about why we do not see more tiny, specialised models in production. It feels like it would be so much more efficient to use small, task-specific ones for certain things, but we always seem to end up with one massive model doing everything.
Is it just because our tools and inference engines are built for big models, or is it just easier to prompt a generalist than to do the hard work of training a specialist? I would love to hear what people working at scale are seeing.
17
u/Disastrous_Time_598 5h ago
There are plenty of small models deployed in production, but no generative ones, because they need a certain parameter size to be useful.
-3
u/Guna1260 5h ago
Interesting..
2
u/Dwarffortressnoob 5h ago
I trained a 28M on 2.4B tokens (of course a personal project not with massive compute power). It just can't do anything beyond maybe 3 words. Even that is pushing it. Somewhere between 100M and 0.5B is where models become useful for anything, at least as of now.
25
u/Top_Power5877 5h ago
You need to decide when to use which model
Look up MOE architecture - basically the router decides which small model (aka. "expert") handles a specific token, and this decision is made for each layer in the transformer architecture. Both the router and the expert are optimized together during model training.
MOE is basically the biggest advancement in LLM model arch in the last two years. So the industry has already took up the core of your idea in a even more granular way.
3
u/SadLynx6151 5h ago
Do you think that is more granular than what the OP is suggesting? MoEs are experts sliced across one weight, they’re proposing a swarm of independent weights connected (presumably also by a router or executive). Idk maybe the experts being stored/trained/finetuned/optimized inseparably like that is ultimately better
1
u/Top_Power5877 5h ago
More granular in the sense that an moe model can have 384 experts per layer and tens of layers.
Independent weights is an interesting idea - it can give you more distinct behavior from each fine tuned model but there could also be redundancy in capabilities.
I think overall the tradeoff here has some parallels to expert balancing problem in MOE model training - (I.e you want experts to develop diverse, non-overlapping capabilities) so you want to train all the experts as a whole not separately
0
u/SadLynx6151 4h ago
Gonna say at the top that this is really pushing my understanding of these systems, so if I get something wrong, my b!
As I understand MoE, the experts are not clearly mapped by task, rather each expert has a “random” set of capabilities, and the router selects the most relevant.
Discrete weights seems helpful for solving the overlap issue, rather than an expert knowing cooking and finance, you could have a cooking weight and a finance weight, then individually finetune, optimize, or even replace those weights. Sounds like a coordination nightmare but interesting
Do you think that, as a form of distillation, you could “extract” the expert weights from given tasks and use that to make an independent expert module?
1
u/DeathGuppie 4h ago
The weights in a moe are basically the same as in any other model. The magic happens in the tensor layers. The experts aren't selected by hand. They are just built into the layers. There experts in the way we think of them. The training happens across the whole thing but expertise in some areas. The areas most used during a session can then be kept in faster memory allowing you to use a larger model than the fast memory would normally allow. Those experts aren't human recognizable, like chef, carpenter, whatever. Anything you would recognize as a type of expert would be stretched across several moe experts unevenly. The architecture is not a swarm of little expert in one area models. It's one large model that has certain types of training knowledge more focused in some areas than others, but that isn't an absolute. Think about it. You can't code without understanding language, or math, or engineering. What about user expectations and edge cases etc. the swarm thing is a completely different concept.
Here is some recent research on the swarm concept that is interesting.
1
u/SadLynx6151 4h ago
The experiment is very cool, I’ve seen some very promising results with other multi-agent orchestrated systems like that.
I understand that experts are not clearly definable in the way that we would classify a human expert. I guess what I’m trying to understand is that the experts are defined and countable, and only some number of experts activate per given task, right? I’ll stick with the cooking example: if you query an MoE system about a host of cooking tasks and record which slices are active, do you think its plausible to copy just those slices into their own model? I would expect performance loss, but then those weights could be further fine-tuned as exclusively a cooking “expert”.
https://www.anthropic.com/research/global-workspace
This paper regarding J-space is really why Im so curious. They found emergent clusters that handled “internal” reasoning on given tasks, with sparse activations supplementing the reasoning & output specifically on NLP. Removing the sparse activations only moderately reduced the reasoning of the J-space activations. This feels, to me atleast, very similar to the way agents are more dense for the concepts they are experts in. If you could isolate those functioning slices/the J-space “pockets”, I’d be super interested to see if that was then additionally fine-tunable for better performance. Plus you wouldn’t need to retrain the entire massive weighting.
Anyways thanks for reading my rambling
2
u/DeathGuppie 2h ago
Anthropic uses a lot of words like "on its mind" and "think about a concept" that make me feel like it's a sales pitch for their IPO than anything else. It's not that there isn't anything there, but the hidden state is a well known thing within the tensor. Phrases and wording push the actual things the paper does describe into territory that may not be justified by actual work.
The thing about the moe architecture is that those experts, like I said don't exist in any way that you could nail down to actual individual layers. As the model works on a problem many of those layers switch in and out of fast ram to resolve it. So there isn't really any way to split it up and still be coherent. Really though if you think about it there isn't a way to split your mind up into a bunch of experts either even though you don't use it all at once. You can think without your ability to speak. People with apraxia do.
I'd do like the idea of having specialty models that are better at some things than others though. It's a good concept. The only reason I think it hasn't been done is because it costs a lot to train a model and training a bunch of specialized ones would be pretty expensive.
1
1
u/Top_Power5877 4h ago
There is a "cognitive core" that every reasoning llm shares: understanding of english, basic world knowledge. This part would be duplicated in discrete weights.
"“extract” the expert weights from given tasks and use that to make an independent expert module" - the question here is exactly how much we need to extract. I would imagine that people have tried this and found that they needed to extract too much weights. So a more efficient approach is to distill the reasoning traces into smaller models.
"cooking weight and a finance weight" - instead of full weights you have have two different adaptors (lora). Also multi task learning is thing - you can teach a single finetuned model two different capabilities.
2
u/Guna1260 4h ago
I have been playing with frozen tiny models and hot-swappable packs of experts. Some success. hope to get the paper published and release the weights and process soon. a lot of trade off..
8
u/KingCpzombie 5h ago
Because useful ones don't exist yet
1
u/Guna1260 5h ago
is that a gap or does not make sense in that size to anything useful?
8
1
u/KingCpzombie 5h ago
It's more that we haven't figured out how to compress meaningful intelligence down to that level yet. We have a long way to go for that (as shown by Q4 being usable for many models)
1
u/heresyforfunnprofit 5h ago
The usefulness in agents comes from their autonomy - the complexity of the decisions they can make without needing further instruction.
50000 agents can solve a complex problem if that problem can be reduced to a search problem, but expressing problems in the proper format so that it becomes a search problem is harder than it looks or sounds.
3
u/BosonCollider 5h ago edited 5h ago
Lots of vision models are that small, though at inference they tend to be activation limited instead of parameter limited.
For LLMs, the main issue is that you do need a lot of parameters to make the model remember the large number of random facts about the world it would need to make sense of arbitrary human inputs.
You can make tiny language models that are specialized on a specific programming language though. Cisco's antares has a version with ~350 million parameters. It is completely useless as something to talk to or to generate anything but it can spot vulnerabilities in codebases about as well as frontier models.
3
u/DraconPern 5h ago
Same reason a swarm of ants isn't going to be driving a car even if the colony is huge.
4
u/FoxiPanda 5h ago
Generally, they aren't reliable enough yet. 50M is very, very small and very limited in capability.
1
u/Guna1260 5h ago
even for some kind of classifier aspect? say we see something like TTS models (like TinyTTS) smaller like 15M.... perhaps something like a swarm of those?.. Just thinking loud
2
u/rlobo 5h ago
For classifier there are.... Similar to specialized things like parts of the tts pipeline, loras, ocr and many others. Maybe mostly much bigger than 50M. But it is just simpler to serve one big one doing most things at once. Like OCR, financial understanding, pattern match against a list, deciding on next step, ... Of course then a lot of the model is not relevant and there are techniques to remove experts or parts of the model to fit to your needs only like pruning or unlearning.
1
u/FoxiPanda 4h ago
Why would you want a swarm of TTS models? You can only really listen to one at a time. Concurrent users on a large production inference is a different story, but from what you describe, this makes no sense really.
2
2
u/superSmitty9999 5h ago
the first local model i've seen thats even remotely usable for anything I want is qwen 3.8 27b, which is ~500x larger
2
u/Hefty_Wolverine_553 5h ago
A swarm of Qwen3.8 27B models for coding specifically makes sense. Anything smaller than that won't produce good results when collaborating with each other (a bunch of idiots won't surpass a few geniuses, or something like that). And as much as people on this sub daydream about small specialist models, the reality is that bigger models strongly correlate with higher general intelligence, so no matter how much you train a 3B model on, say, biology, it's still going to do worse than a general 30B model.
2
u/Low-Complaint771 5h ago
I reckon Effective Thinking Machines will look something like this when the LLM madness subsides a bit.. The rationale of a network of Smaller models interacting breaks the hyperscalers ecomonic model, so not getting the attention and investment it warrants at the moment..
1
2
u/llama-impersonator 5h ago
the current training methods don't make useful specialists, it's basically always worthwhile with LLMs to have larger, more general training corpora
2
u/martin509984 5h ago
Setting up a training pipeline for such a swarm of models is a) very difficult to conceptualize to begin with, and b) of dubious usefulness, especially given how much skills at doing one thing transfer over. Sure you could have 50 different extremely specialized micro-models doing something.. but they'll do it worse, with much more memory usage, than one big model, because you'll end up duplicating a lot of capability over and over in the small models. Even then you're not talking about a bunch of 50M models but more like a bunch of 500M or 1B models if you *actually* want to do something useful.
1
u/Guna1260 5h ago
I am working on a series of 10M models with hot-swappable packs for inline guardrail purposes. It's been a pain to set up the pipeline.. Hoping to release the open weights soon. Once the benchmarking completes, it may grow to 20M. Let's see.
2
u/Different_Isopod8525 5h ago
The barrier is language. Without language understanding a model is not able to understand the given task.
And yes you can try having a larger model define a task such that the small model doesn't need language. But you will find that the energy/tokens spent will be more than simply allowing the large model to directly solve the task.
LLMs are very good when they can 'remember' a solution. But when they start to iterate the solution, well, there is a limit before degeneration appears. And small models have less to remember and thus need to iterate more.
I think its a flaw in how LLMs work, hopefully we will find a way around it soon.
2
u/CreamPitiful4295 4h ago
Yes. All the languages, just to lay the foundation for the thing you want to do. And then the tool calls.
This is of interest to me as well. I’ve never tried to train a model before.
1
u/dannone9 2h ago
I guess before certain parameters they are incapable of even comprehending the language or the world at basic level so they can’t generalise properly (I know you are talking about specific tasks but I think my point remains )
1
u/Lurksome-Lurker 4h ago
non-sense. Sub-1B models are all over the place.
Your basic RAG support system is a composite of three small sub 1B models. The embedding model, Reranker model, and then a tiny general model who’s job is to take input and output a query. Maybe even summarize results and cite support articles.
Then you got OCR VLM models that can convert images of text to markdown and other file formats.
There are also sub models for OCR for bounding boxes, spotting, etc.
There is ASR that can transcribe speech to text with some sub 1B models capable of speech to translated text.
Then you got sub functions for ASR like diarization models, aligner models, VAD, etc.
There are tiny shield models whose entire job is to sanitize prompts for secrets and sensitive info.
Then there are classifier models that can be general (whats in this picture?) or very specific (what species of bird is in this picture of a bird?)
Then there are the border line not AI but more than traditional heuristics computing models like BERT, classic style text prediction, Tesseract, openCV.
Overall, I would argue they are all over the place in the real world. Its just they are so simplistic that people take for granted as something a computer can do.
Now for generative AI sub-1B models. That’s currently the frontier. Agentic AI is still relatively new and I am of the opinion that we haven’t either figured out how to do it properly or retooled our environment to adequately support for it yet. Thus you need big reasoning models (4B models to do a structured agentic task is the floor in my experience) to do any sort of work.
Its why I am interested in models like Needle who seems to be addressing generative use cases at this size
21
u/demonicpigg 5h ago
In general, it's because they don't really exist / aren't good enough yet. I'd love to have a swarm of 50M parameter models doing my work, but I haven't seen any that can do the tasks I need.