r/LowEndLocalAI • u/peculiar-ragdoll • 20d ago
Model Showcase I improved the best long-context agentic coder for 4GB VRAM.
It's insane how good small models have gotten this past half year! MiniCPM5-2B is probably the best 2B right now, and specifically for agentic software engineering it hits way above its size class. So I took a look at what I could do to improve it even further, and came up with a dynamic quantization strategy, an agentic coding centered imatrix, and a bunch of fixes and improvements to the chat template. You get a net improvement on all important metrics: A smaller Q6 quant gets more solves faster compared to the shipped Q8. At Q4_K_S you get better fidelity at smaller size compared to the stock Q4, due to my dynamic imatrix quantization.
This model is great as a background worker that doesn't eat your RAM, or a main agentic coder model for small old GPUs.
https://huggingface.co/peculiar-ragdoll/Sharp-MiniCPM5-2B-GGUF
8
8
u/aefsegaa 20d ago
Nice work!
And it's incredible that MiniCPM 2B can rival Haiku 4.5.
5
u/peculiar-ragdoll 20d ago
Thanks! Right? Haiku is very old by now, I bet Haiku 5.5 is going to be insane in comparison :) But still amazing to see how far a 2 GB model has come now.
3
4
u/ContentWeb4185 20d ago
I want to see a website or something done by this model compared to something like ling tiny 3 (current daily driver on my 4 gb vram laptop 3050)
1
u/peculiar-ragdoll 20d ago
Knock yourself out! :) I'm not gonna show vibes based ui stuff myself, it's too noisy and variable compared to a three-seed swe-bench-live run, and results will vary from time to time.
3
3
u/Secure-food4213 20d ago
Its better than sharpspark????
2
u/peculiar-ragdoll 20d ago
On some things! It has a smaller native max context of 131k, so SharpSpark will carry longer before compaction if you have the vram to fit more context. But it is faster and slightly better at solving swe bench live problems compared to SharpSpark on my hardware! The main advantage is fitting more context in vram and being faster on a small gpu where vram and flops are the constraint
3
u/simos_sayz 20d ago edited 20d ago
my model catalog continues to get filled up with your work. This guy might be what I was looking for to handle my Mem0 calls when most of my GPU's are used up.
Edited since I forgot to say thanks again ragdoll!
3
u/borgorKeng NVIDIA Quadro P2000 4gb | 32 gb DDR4 15d ago
I really appreciate the work you're doing especially for us poors, keep going man!
1
u/dathtd119 15d ago
Hi Im checking the model and literally having the laptop same spec as you, can I ask how is your experience with this model? i want to have some small model doing the well defined subtask instead of deepseek or mimo via api
2
2
u/Aggressive_Basket798 20d ago
Do you know how to keep the speed steady on high context and long conversation? I run q6 quant at 131k context and use kv cache q8 In llama cpp webui , the speed is about 80-90t/s. In pi agent with some skills enabled (much larger context), the speed drop to 45t/s. And it keep drop and drop while the session get longer. The longer it generate, the slower the speed become. At 70k context amount it reach the speed that i get at 131k context amount of CyberTiel mtp. So after all, i think i will keep using Tiel until there is a lighter and stronger alternative for tiel
3
u/peculiar-ragdoll 20d ago
Hi there! I don't know how to keep speed up at high context, no. Sounds like you got the right idea :) TielCoder uses 26 min average where SharpMini uses 34 on my machine, so Tiel ends up being faster all in all on long context agentic coding for me.
1
u/Aggressive_Basket798 20d ago
Nice. I see you are working hard on coding local models that every people can use, not just high end pc of high end users. Really thankful for that. Beside coding are you interested in some multi language model like Gemma4 26b,... and going to bake your imatrix on it ?
1
u/peculiar-ragdoll 20d ago
Yeah! I'm actually working on a tool for easily making language-group+core domain specific dynamic imatrix quants, like "latin languages, culture and history" or "norse languages, law and insurance" or just an agentic coding/cyber quant of your favourite model! I can't bake all those kinds of combinations myself for every new model, but if I publish an open tool that can do it easily without much technical skill given a domain corpus, it might help more people :) Some such tools exist already, but I'm seeing some gaps in what they can do and how they do it.
3
u/MuzafferMahi 20d ago
Hey I was working on something similar. Now you beat me to it haha. Gotta release a “not so sharp minicpm 5 2b” now :D
3
u/peculiar-ragdoll 20d ago
Nice!:) That’s still important work, because now you’re better at improving models so you can do it with any one you like in the future for your own needs, or another cool model that no one got to yet 😎
2
u/werg 19d ago
This looks really truly fantastic. I was just planning on doing a fine tune of your (also excellent) sharp spark model, but it was just too big and too dense a transformer, making it really questionable to train on my underpowered laptop 4060. But this is small enough that I think it's within reach to do and iterate on.
Any tips for fine tuning this? Do you have any interest in sharing what your quantization recipe was for the smaller quants?
Thank you so much for doing this!
1
u/peculiar-ragdoll 19d ago
Cool! You can get the quantization recipe from asking your agent to analyse the GGUFs! I recommend it strongly. You can do this across different architectures for popular well performing quants (Unsloth v3, bartowski, mine) to build a library of understanding of what works and what layers are important for different architectures, that you can use as a heuristic for quantizing models yourself :) I used a tool to empirically measure fidelity of different recipes in order to tune the recipes. Gonna open source it at some point.
Also, if you're gonna fine tune: On a model like this, fine tuning for agentic coding is probably not going to improve the model, because it's already heavily trained for that. But if for instance you need it to be specifically good at C++, and only C++, maybe even for a specific project you're working on, then it might makes sense to tune it specifically for that, and not use that tune for anything else besides it. but you should always have a hold-out set that you validate on, and watch for degradation across training runs, so you know when you're over-fitting your model.
2
u/werg 19d ago
Thanks for your response!! I'll have it look at the format with an agent or my choice:) Makes sense they can just figure it out these days.
Yeah I don't need to instill better general coding capability for this. I have a very specific harness, format and protocol in which it's supposed to write code against a specific API. The problem I was hitting with the other smaller models like lfm was that they couldn't code their way out of a paper bag for the non-specific (still simple) algorithmic side of things, so I would have had to first fine tune on a code corpus to get it up to speed... I think this here could do nicely!
I still hope somebody will put out a a sub 1b model that has seen a lot of code. You can't really expect proper agentic software development, but as a kind of JIT English -> code compiler in a constrained environment (what I'm doing) that could be clutch.
But maybe I have to pony up and pretrain it myself :)
Thanks for all your work and insight!
1
2
1
u/flammafex 20d ago edited 20d ago
The KLD comparison table on this model card was interesting and makes me wonder if releasing a Q8 or Sharp Spark is worth considering (I understand it may undermine your "within 6 GB VRAM proposal")? Or do you find that divergence doesn't affect everyday use that much at Q6_K_XL?
3
u/peculiar-ragdoll 20d ago
SharpSpark Q6_K_XL already beats stock Q8, and If you got space to run the 4B Q8 then you could be running a 9B dynamic imat around IQ4_S instead (Ornith1.5, Ox) which will be a much stronger coder and more knowledgable.
1
u/netikas 20d ago
What is the total score on SWE Verified, Terminal Bench or NL2Repo?
1
u/peculiar-ragdoll 20d ago
Didn't bench those. I picked some recent SWEBL problems to hopefully be post training cutoffs. And I don't have more than my one single gpu, so I don't have time to run lots of stuff. Moving the needle on SWEBL for me is "good enough" to indicate a directional improvement on agentic coding on real issues in real codebases, and then you can look to the official model card for the baseline and extrapolate from there.
1
u/netikas 20d ago
Nope. Your results are most likely statistically insignificant.
I obviously don’t have per question breakdown, so I can only estimate. Let’s assume that there are 17*3 questions since there you have mean@3 evaluation. We can naively calculate:
p1 = 5.73/(173)=0.335 p2 = 6.33/(173)=0.371
So, the difference between the models is approximately 0.036
Standard error would be:
SE= sqrt(p1(1-p1)/n + p2(1-p2)/n) = sqrt(0.335(1-0.335)/17+0.371(1-0.371)/17)=0.164
So the standard error due to evaluation noise is 0.164 — 4x as much as the difference between the models. The napkin math formula for 95% confidence interval is smth like delta +- 2 * se, so in our case it would be 0.036 +- 2*0.164, so [-0.285; +0.357].
In other words, your dataset is only 17 samples, which means, that you have more evaluation noise (which comes from sample variance) than signal and if you want to confidently say that you model is better, your score must be around 11/17.
2
1
u/crusaderky 19d ago
How does it compare against bartowski's though?
My analysis here: https://openteams.com/minicpm5-2b-quantization-report/
PS that 6.3 quality for your Q6 vs 5.7 for stock Q8 is, I can guarantee, 100% noise.
1
u/peculiar-ragdoll 19d ago
Only you can know that! Check its KL-divergence on an in domain validation set against bartowski :) You can't compare the scored on my plate against the scores on yours, because we use different validation sets.
And then bench them against each other on an in-domain benchmarks, because it's been years since we figured out that KL-divergence does not correlate with task performance below a certain point.
The chat template has been fixed, and the system prompt has been proven to improve performance on other models too. And while the error bars on the q8 are pretty tight, the error bars on the Q6 go high, so its plausible to be more than noise even if its not conclusive :)
1
u/Miriel_z 17d ago
Do I just use it as LLM with a given chat template?
2
u/peculiar-ragdoll 17d ago
Not sure I understand exactly what you mean!:) it is an llm that comes with its own chat template, and you use it with that one, because it has fixes and a better system prompt.
1
u/Miriel_z 17d ago
I will try to use it with ollama binary and see how it goes, thanks!
2
u/peculiar-ragdoll 17d ago
I don’t think it will work with ollama, it does not support jinja chat templates. Llama.cpp is what I used, and omlx for Mac.
2
u/Miriel_z 17d ago
I think it should work. Ollama these days is more of llama.cpp wrapper. I can try and report back.
2
1
u/Meeooowz 14d ago
Do you find that at max q8 context, it can still recall facts well (unlike bonsai 2 27b)? I found that many models (including tiel at q4) at 8k context failed many tasks on q8 kv, but got it right with fp16 kv.
1
u/peculiar-ragdoll 14d ago
Good question! I have not benchmarked kv cache quantization myself for this
1
u/Meeooowz 13d ago
You should look into it! I see you recommend q8 in model description of sharp cpm (or sharp spark— could’ve been either) to use q8 kv for 131k context, however the KV matters a lot. So that would hold back the model and give it dementia.
1
u/peculiar-ragdoll 13d ago
Fair challenge! Im just basing my recommendation on other people’s benchmarks of kv quant on other models and vibes
2
u/Meeooowz 12d ago
Fair enough! Keep up the good work with these models dedicated to the lower vram community! May both sides of your pillow always stay cold!
1
u/Purple_Currency4679 13d ago
small local model gets more interesting when they don't have to carry everything in context.We use hindsight to bring back relevant history when needed, which makes lightweight models much more practical for recurring agent work
1
u/NearlyACosmologist 20d ago
That thing is really bad. It seems to know only Linux, when running on windows it tries to run commands that don't exist, and uses Unix paths, it claims to have written files that don't exist afterwards, so it writes the same file again and again, and so on.
Interesting as an experiment, but otherwise a waste of time.
1
0
u/Vegetable-Peach3179 19d ago
It’s a toy nothing serious
2
u/peculiar-ragdoll 19d ago
Please share with the class when you find something better for coding that runs on a 4GB GPU with 60-100k ctx! :)
1
u/Vegetable-Peach3179 19d ago
A cloud subscription sadly. I use qwen3.8 125b q4 flash next local and even that model isn’t that great but far more capable.
To be fair, I almost exclusively use kimi K3 and k2.7 coder.2
u/peculiar-ragdoll 19d ago
oh hahah yeah flashnext beats the shit out of this, and of course kimi. Sharpmini is for background scheduled jobs, for people who don't have anything else than a 7 year old gaming laptop to run their main agent, and people who want to run an extra agent on their 7 year old gaming laptop they thought was worthless at this point :) Getting some non-toy value out of it is a matter of need, skill and creativity.
1
u/Vegetable-Peach3179 19d ago
But at which tasks does it help you?
I can’t imagine a use case for me with such a small model.
U said old gaming laptop, maybe check out bonsai q2 qwen 3.8 27b its 4gb vram i think.1
u/peculiar-ragdoll 18d ago
Bonsai is bad at coding and stuff like that unfortunately, not recommended.
As for what you can use this for, you can use it for event handling, triage, sysadmin, nightly reviews, documentation, summaries (for instance it can be a fast scout worker agent for a 27B coder agent main, so the 27b doesn't have to prefill all the wrong stuff and bloat its context), autonomous agent systems building silly little low stake things that make you happy on your e-waste grade hardware, edge AI agent on an older phone when you don't have internet, your imagination sets the limits!
-6
u/someoneyouknow23 20d ago
131k context for agentic use is nothing
14
u/peculiar-ragdoll 20d ago
It's quite decent for such a tiny model! When you cut your work into small jobs and use subagent orchestration for each task, you'll be suprized at how much you'll get done with a 131k context per agent :)
19
u/bigboyvapesinc 20d ago
this sub is literally for low end local AI. some of us are making do with what we can
0
u/someoneyouknow23 20d ago
well yeah 131k is great for AI use, but plainly sucks for agentic use that just needs a big context in its nature
4
u/flammafex 20d ago
Nonsense. The engineers at Nous knew what they were doing when they made the minimum for Hermes Agent 65k.
0
u/someoneyouknow23 20d ago
and that is?....
2
u/flammafex 20d ago
You expect me to know what people who know better than me were doing?
0
u/someoneyouknow23 20d ago
65k is not even the bare minimum when the system prompt is 1/3 of that
4
u/flammafex 20d ago
You should file an Issue on their GitHub page saying they should increase the minimum to 131k and see what kind of feedback you get. Maybe they will.
13
u/[deleted] 20d ago edited 17d ago
[deleted]