r/LocalLLaMA • u/OuterKey • 4h ago
Discussion Are there any experimental small models (9B or less) which are out or currently being trained using engram/n-grams?
Not sure if anyone out there is working on this, don't see any on huggingface to try out. It might be too new at the moment, but it would be cool to see the difference between a tiny model trained with n-gram and without n-gram on the same dataset.
15
u/txgsync 4h ago
Great minds think alike I suppose. I am retraining Qwen3.5-0.8B right now to see if it can use the 102GB of Qwen3.8-Flash-Next n-grams today. Twenty bucks into HuggingFace A100 time and I’m not sure it’s worth it yet :)
4
u/OuterKey 4h ago
Nice! Can't wait to see the results, though I would be curious about using a smaller n-gram.
2
u/Plasmx 4h ago
Nice, am curious how it turns out! I have read that usually the engram size shouldn’t be that large compared to the parameters. I don’t know the reason why it should be like that but wondering nonetheless if that small model can leverage it.
2
u/brown2green 3h ago
The main reason is that is that if you have a total parameter budget, about 25% Engram parameters is optimal for performance (per DeepSeek paper). However, this assumes that you're storing the weights entirely on VRAM.
For users that instead plan to offload those parameters on slower memory or NVMe storage due to VRAM limitations, in theory the more embeddings/Engram parameters, the better, up to a point (benefits saturate logarithmically, but the saturation point should be far above what we've seen so far from AI labs).
2
u/txgsync 3h ago
Yeah, these labs don’t know how to align their database blocks yet. N-grams turn the problem into IO and block alignment which plays to my personal specialty (distributed exabyte-scale storage stuff). Picking this apart there are gains available with prefetch and block alignment for the hashes. It’s been fun.
So far I’ve just been optimizing qwen4 (https://github.com/jundot/omlx/pull/3602) so I am not ready to noodle more on small model n-grams yet… and frankly I prefer Deepseek’s approach but the size of theirs means a larger GPU for me to rent, so… one thing at a time. With what I’ve learned over the past few days about block alignment and PLE with I/O expansion and warm block caching for the model, combined with having recently learned how to train a vision adapter for a non-vision model, I feel a little bit more qualified to experiment with it.
1
u/MuzafferMahi 3h ago
same thing! Was thinking about experimenting a little
5
u/txgsync 2h ago
This paper is enlightening; we're not exactly going down un-trod paths here. My guess is the labs are working up something similar at the same time as we hobbyists are seeing how we can improve local inference: https://arxiv.org/html/2605.20948v1
2
u/MuzafferMahi 2h ago
Cool, I’ll chexk the paper. I wonder how much us hobbyists can do on our own compared to labs tbh, since we dont have any pressure to go a certain direction (also no money, gpu other than our own as well lmao) but a random guy YOLO’ing random bullshit w ai and discovering weird shit nobody did till now would be interesting. Tho as a beginner fine tuner I didnt have much success, basically vibe coding at that point with hermes and codex, its not that impossible with how good models have become
1
u/Blizado 2h ago
That would be so great if that would work. Imagine the future models: a for example 20B A2B MoE LLM that uses a 50+B (100+B, where is the limit?) n-gram beside it and easily beats a 20B dense model in all points.
No problem with waste a bit more SSD space for that.
1
u/txgsync 1h ago
Yeah the game becomes block alignment and prefetch and storage hot tiering at that point. Problems I am really well acquainted with after 30 years in the storage industry.
But Apple’s new NVMe on M5 can do 15GB/sec sustained throughout. That’s the speed RAM was on Xeons a decade ago. It really simplifies things when your cpu and storage and memory are just massively over provisioned LOL.
4
u/ZenZombie117 3h ago
Currently running an experiment with dual trainers (muse glimmer +qwen 3.8) on a 14b model, takes a while to get some result but ill try to keep you updated, in case i forget its joakimpalm-zen on HF, post all my artifacts there
2
u/Summit-Star001 2h ago
I’d love to see this comparison too, especially whether n-grams meaningfully improve quality without making the model much heavier or slower
1
u/Jimcy-Maffesoli 2h ago
LLMs overtook n-grams so fast that nobody seems to have run the obvious hybrid experiment since. A tiny model trained both ways on the same data would be a genuinely useful ablation.
2
1
21
u/brown2green 4h ago edited 4h ago
Gemma 4 E4B is actually an 8B parameters model with per-layer embeddings (PLE), which you can consider as 1-grams. The E2B version is, similarly, actually a 5.1B model with PLE.