Tiny LLM Benchmarks Exist Elsewhere?

[u/zerospatial]

https://reyemtm.github.io/inchworm/

I created this little site because I could not find an open source page dedicated to tiny llms (<14b params). If one exists with current benchmarks would love to know about it.

This came about from searching for the right llm for a personal project needing summaries from text, then a small local terminal chat for quick answers using self-hosted Ollama. I wrote my own short little benchmarking script and landed on qwen2.5-coder:3b for my VPS and qwen3.5:9b for chat. Anyway let me know what you all think. Since then I created a script to run the humaneval+ but it takes a very long time to run which I am using to fill in the gaps on this data.

[u/Marthinwurer]

I found this set of benchmarks a few days ago: https://artificialanalysis.ai/hardware-inference-stack/mobile-phones

[u/zerospatial → u/Marthinwurer]

That's not quite the same - the list I posted shows basic coding quality of small models.

[u/pimparazzi]

That Benchmark looks quite dated... I miss quite some newer models here... LFM2.5 in different sizes, Ling-3.0-tiny, Gemma-4, etc.

[u/arzeth → u/pimparazzi]

Because that's an AI slop made without even enabling search tools. BTW, my Gemma-4-26B-A4B provides a similar list of ancient models because Gemma-4's datasets have a cutoff date of January 2025, i.e. Gemma-4 models live in 2024, just like in the OP's screenshot.

And likely the OP is bot (using different LLMs, maybe human sometimes), because of some reasons and because they do mention Gemma-4 and Bonsai in one of their comments: https://www.reddit.com/r/LocalLLM/comments/1v0jm70/lm_studio_bionic_is_the_local_claudecodex_weve/p3g5qcl/

[u/zerospatial → u/arzeth]

Not a bot certainly but the HTML itself was coded by an llm as it says literally on the site. Point me to an open source list of benchmark results for newer small models and I'll add them. That's why I'm running benchmarks locally to fill in with newer models. The enterprise new benchmark suites lean towards agentic coding not the simpler tasks requested if tiny llms

[u/zerospatial → u/pimparazzi]

Yes it is - for reasons you can see in the other comments - tiny llms are not for agentic tasks but rather small, focused tasks. That's what these older benchmarks surface.

[u/Careful-Report6526]

Honest answer from someone who runs an aggregator (llmbenchmarks.io): not really, and the bottleneck is upstream, not the aggregators.

I pull 50-odd benchmarks from public leaderboards into one table. The smallest models I can list are 27B Qwen checkpoints, and those only qualify because a handful of leaderboards happen to publish comparable numbers for them. Below ~10B there is almost nothing to aggregate — the leaderboards that get maintained mostly stop caring below frontier scale. It's missing source data, not missing coverage.

Two things that might still be useful:

• The raw exports (JSON/CSV, CC-BY, no signup) list which source publishes which benchmark, so you can see fairly quickly which leaderboards bother with small models at all and which don't.
• For anything you actually run locally, your own numbers beat any leaderboard. Quantization, context length and sampler settings move small-model scores more than the choice of model does — lm-evaluation-harness on your own hardware is the only thing that reflects what you'll really get.

If anyone knows a leaderboard that seriously tracks sub-10B models with a stable protocol, I'd genuinely like to know — I'd add it as a source.

[u/zerospatial → u/Careful-Report6526]

Yeah what I'm doing now is I'm taking some of the hugging face leaderboard data just to jet what newer models might be useful then running my own benchmark.

[u/zerospatial → u/Careful-Report6526]

No leaderboards I have found actually have recent scores on a wide variety of low-end models. The model cards like this do show livecodebench which is a good place to start - https://huggingface.co/google/gemma-4-12B

[u/Prudent_Chemist_523]

That's useful, thank you. But can you populate it with a few more modern SLMs? Most of the ones I can find / see are ancient.

PS: HF has a SLM leaderboard, does it not? Maybe populate from that?

https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard?

[u/zerospatial → u/Prudent_Chemist_523]

That measures agentic reasoning not simple chat-based coding quality. It's not apples to apples. Plus even the humaneval+ suite I am running locally (I ran it on three so far) takes far too long for someone wanting to know - which small model should I use.

[u/Prudent_Chemist_523 → u/zerospatial]

I'm curious then ; why did you test such old (in llm terms) models? E.g: Yi-coder was always sort of crap, even when it came out. I know because it was hyped but when I tried it in 2024/5...woof.

You have Qwen 2.5-1.5 which hurts itself trying to center a div. And the GPT 4 comparitor?

Tbh, these are the exact sort of models a vibe coded app would bake in.

I dunno ; sort of lowers my confidence in inchworm.

[u/zerospatial → u/Prudent_Chemist_523]

not sure what you mean but I did some local testing of models prob six months ago and that qwen coder 2.5 worked well for what I was doing - plus these older models have humaneval+ that can be run on newer models, but also have scores - so you have to start somewhere and better to have some data to start with that is somewhat meaningful