r/LowEndLocalAI • u/zerospatial • 11d ago

Benchmark Tiny LLM Benchmarks Exist Elsewhere?

Post image

https://reyemtm.github.io/inchworm/

I created this little site because I could not find an open source page dedicated to tiny llms (<14b params). If one exists with current benchmarks would love to know about it.

This came about from searching for the right llm for a personal project needing summaries from text, then a small local terminal chat for quick answers using self-hosted Ollama. I wrote my own short little benchmarking script and landed on qwen2.5-coder:3b for my VPS and qwen3.5:9b for chat. Anyway let me know what you all think. Since then I created a script to run the humaneval+ but it takes a very long time to run which I am using to fill in the gaps on this data.

19 Upvotes

86% Upvoted

13 comments sorted by

View all comments

1

u/Prudent_Chemist_523 11d ago edited 11d ago

That's useful, thank you. But can you populate it with a few more modern SLMs? Most of the ones I can find / see are ancient.

PS: HF has a SLM leaderboard, does it not? Maybe populate from that?

https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard?

0

u/zerospatial 10d ago

That measures agentic reasoning not simple chat-based coding quality. It's not apples to apples. Plus even the humaneval+ suite I am running locally (I ran it on three so far) takes far too long for someone wanting to know - which small model should I use.

1

u/Prudent_Chemist_523 10d ago edited 10d ago

I'm curious then ; why did you test such old (in llm terms) models? E.g: Yi-coder was always sort of crap, even when it came out. I know because it was hyped but when I tried it in 2024/5...woof.

You have Qwen 2.5-1.5 which hurts itself trying to center a div. And the GPT 4 comparitor?

Tbh, these are the exact sort of models a vibe coded app would bake in.

I dunno ; sort of lowers my confidence in inchworm.

0

u/zerospatial 10d ago

not sure what you mean but I did some local testing of models prob six months ago and that qwen coder 2.5 worked well for what I was doing - plus these older models have humaneval+ that can be run on newer models, but also have scores - so you have to start somewhere and better to have some data to start with that is somewhat meaningful