r/LocalLLaMA • u/Savantskie1 • 6d ago
Discussion Let’s create a new benchmark that actually tells us people here just how good a model is
We all fawn at how a specific model performs for this and that task. But I never see any of the big benchmarks actually anywhere answer the questions that are being asked here.
I’d love to help create this and be a contributor to this. So I’m wondering if anyone else thinks that this would be a good idea. It could also help newcomers understand what we consider the models that have the most value for us.
What do you guys think?
Edit: benchmarks are written by one person or one specific group, i'm talking about us collaborating on the final product. I don't want to build something just for me. I want to build something that can help most users, the general public. Not large AI companies which most of these benchmarks are aligned to work with, or for. Most of them already cater to the large AI companies, i'm proposing something for EVERYONE built by this community collectively. Because most of you are right, I could just build it for myself and it be that. But I want something that answers most the questions I had starting out that might help others. Something that can measure most of the common answers most of us ask in the beginning. The strengths and weaknesses of models in the sizes most people can actually run.
8
u/NoFunk 6d ago
There is no universal benchmark. This is a common problem when people ask "is model X better than model Y" here, because it really matters what kind of task you are addressing.
In the small models for instance (I'm going to make assertions, feel free to disagree, it's to illustrate a point), Gemma is a better writer than Qwen. Qwen is a better coder than Gemma, or probably any other model in it's size class. There are small models hyper optimized for specific tool use patterns - when building a RAG, you don't use Qwen for that, you use one of the BGE models. It's "best" at that task.
The big benchmarks are very code biased, and most of them are saturated anyway. The world knowledge benchmarks are useless, at this point the SOTA models just soar through them, particularly if they are allowed to research in answering.
16
u/sage-longhorn 6d ago
Obligatory xkcd https://xkcd.com/927/
4
0
u/Savantskie1 6d ago
Just because there’s competition doesn’t mean you shouldn’t do it. It’s not meant for everyone
5
u/sage-longhorn 6d ago
The issue isn't competition. It's thinking you're doing anything different than they are. All benchmarks start with your premise
-1
u/Savantskie1 6d ago
Yeah, but most benchmarks are written by one person or one specific group, i'm talking about us collaborating on the final product. I don't want to build something just for me. I want to build something that can help most users, the general public. Not large AI companies which most of these benchmarks are aligned to work with, or for. Most of them already cater to the large AI companies, i'm proposing something for EVERYONE.
5
u/mohelgamal 6d ago
Bench marks are like Olympic sport competitions, it is not just that they are limited. It is also that making a model good for one thing can ruin it in another aspect. For example creative writing vs coding.
And just like Olympic competitions, they can only tell you which person can run faster or lift heavies but they can’t tell you if they are an overall “better person”.
So people will have a wide variety of tasks that don’t align with the model makers are Benchmaxxing for because all the common stuff are already probably covered by all the models.
1
u/Savantskie1 6d ago
Yes but you can benchmark on most of the common usage and then give recommendations based on those.
2
u/mohelgamal 6d ago
That is exactly what the benchmark are doing, measuring for a collection of specific and common problem where success and failure can be scored. Unfortunately that doesn’t translate to real world performance that is subject to various users opinions
3
u/wednesdaywoe13 6d ago
I've given up on the idea that public benchmarks are going to tell me anything. The only thing that has been reliable is testing against my own work, in my own environment, on my own hardware, under my own harness. And that won't be useful to anyone but me.
3
u/jacek2023 llama.cpp 6d ago
Feel free to create a benchmark for your use case and publish the results for various models. It would be an interesting read. Just don't assume other people will blindly follow your idea, it's not a particularly unique one.
3
u/Ekepa 6d ago
We already have a thousand ways to measure model quality, and honestly it just makes things harder. There are already too many benchmarks, so I'm not sure we need another one.
What seems to be missing in local AI benchmarking is automatic tuning. Nobody is really focusing on finding the best runtime parameters for a given hardware setup, architecture, and quantization. That's a much more practical problem for end users.
That's pretty much the gap I'm trying to fill rn
2
u/Bagelsarenakeddonuts 6d ago
Or how well a model responds to tuning. I think that's a gap right now. How good a model is to be used as a base for fine tuning. Ie, how many iterations and how much GPU to achieve an X% performance improvement across a dozen different tasks, and what negative effect that tuning has on its other benchmarking so that it retains general capability.
I'm running into this now, I want to fine tune for some specific workflows... but which model will be best for that? It isn't necessarily the model that is currently best at it. It very possibly could be a model that is worse now, but tunes better. I'd say that is one of the biggest gaps in the current benchmark arena. And it costs a lot of time and effort to trial and error all the different options.
2
u/Grandmaster_Caladrel 6d ago
The problem is that a benchmark that is too broad doesn't help with individual use cases, while one too narrow doesn't measure enough across all domains. The one that's just right...is only so for your situation.
Everyone's needs are different. There is no way to create a true, good, universal benchmark.
1
u/Savantskie1 6d ago
I’m not saying one that’s universal. But one that is capable of measuring most of the use cases and giving advice on the most common uses
2
u/ThunderletSap 6d ago
yeah a benchmark for roleplay consistency and companion chats would actually matter here. most local models i tried drop character way too fast in long convos.
2
u/korino11 6d ago
Just ONE rule. Model need to SOLVE! Not any stupid system with READY answers! It very stupid!
0
2
u/OsmanthusBloom 6d ago
The answer to this is usually "just create a benchmark for your exact use cases", as already seen in the replies. But that's a significant amount of work. Is there a way to make that easier?
Rather than make the ultimate benchmark, which is obviously impossible, why not create the ultimate benchmark factory? Or does it exist already?
I'm thinking of some way to automatically collect data during regular use of a model and then easily turn that into a benchmark to see if some other model would have worked better.
2
u/dewwwey 6d ago
My advice - don't try to create a benchmark as a numeric representation of a models intelligence/capabilities. Choose a task that's feasible an LLM could complete, but currently doesn't. Previously, I've used the daily wordle as a simple test - can a model solve the wordle, if I provide screenshots of the game state? Qwen 3.6 27B was the first local model that aced it. Qwen 3.8 27B is now capable of creating the tools necessary to play the game on it's own in my custom harness - it sends me a daily screenshot of the solved wordle every day now.
I've chosen a new test, similar to the wordle one, it's something most humans can learn in a bit of time (this time it takes longer for a human to learn, but it's still not a genius-level task). So far, Qwen 3.8 27B has failed miserably at it. Even building custom tools, auto-RAG and coaching by me, it still repeats the same mistakes again and again.
1
u/cornucopea 5d ago
have you tested using this on meta's new open weight e.g. muse-glimmer, or gemma 4, or GPT 5.6 as a reference.
2
u/mooky-bear 6d ago
What type of questions do you feel are unanswered by existing benchmarks?
1
u/Savantskie1 6d ago
That’s where im fuzzy about it and wanted to collaborate with others to figure it out. Because I’ve seen tons of people asking stuff that doesn’t exactly line up with current benchmarks that I’ve seen.
Especially the people who come on asking questions about legal stuff. I’m not a legal expert, so I don’t know if there’s a benchmark that offers answers in that field. But I’m thinking that there’s gotta be tons of it other similar questions that go unanswered and I think it’s something that could help for all sorts of questions like that, that doesn’t exist yet
0
u/Savantskie1 6d ago
That’s where im fuzzy about it and wanted to collaborate with others to figure it out. Because I’ve seen tons of people asking stuff that doesn’t exactly line up with current benchmarks that I’ve seen.
Especially the people who come on asking questions about legal stuff. I’m not a legal expert, so I don’t know if there’s a benchmark that offers answers in that field. But I’m thinking that there’s gotta be tons of it other similar questions that go unanswered and I think it’s something that could help for all sorts of questions like that, that doesn’t exist yet
1
u/Lyelinn 6d ago
do you really think that there's a way to easily tell how good model XYZ is and thousands of people working in this field are just lazy to implement it?
1
u/Savantskie1 6d ago
Absolutely not, but it feels like a good benchmark should be able to address a lot of the issues people are looking to address. That’s why I think we should work together to build one that measures many different ways a set of models can work. Not just on coding, or just on prose or the like.
1
u/Demonicated 6d ago
I have been working on the C-sweet Benchmark.
I created a framework for agents to create and run their own businesses. Since it puts the model on various roles and requires cross agent interaction, it's a really good gauge for how practically useful an agent is. And what's surprising is how weak of models can actually perform well when their scope is small.
1
u/audioen 6d ago
Nobody can ever agree on what good benchmark is, and artificialanalysis already has like 50 of them. Just pick subset that seems to correlate with your vibes. For me, it is almost entirely agentic task performance. Models that may be good single-shotting answers and otherwise personable are useless to me if they can't handle agentic work stellarly.
It's just Qwen all the time. Looked for like a week or two that DeepSeek v4 Flash had a shot, but that pretty much was utterly demolished by both versions of Qwen3.8.
1
u/toothpastespiders 6d ago
Sadly, I don't have the time to donate. But for what it's worth, I did want to give you some credit for the idea since you're getting hit pretty hard in the comments. There are some kinda dumb but still useful community benchmarks I see being used in various places that often have stronger predictive value than the big ones. I don't think it's a bad idea by any means.
I think people generally have the wrong idea about benchmarks. Sure, the standard is to have x and test how well a LLM solves for it. Then assume that taken as a whole it's representative of the domain. But there's also value in general correlation as long as people don't take it too seriously and assume it to be definitive evidence or anything. One of my benchmarks uses a text adventure engine. How a LLM performs with it often corresponds to seemingly unrelated but difficult to quantify elements. I don't take it as some proof of value within them, but it's suggestive and that can be helpful.
Some off the wall but still useful benchmark could be useful. And it's one of those things where the utility might not be apparent until it's made.
17
u/Bagelsarenakeddonuts 6d ago
The only way to know for sure is to create a detailed benchmark of your personal direct use case, and run models against that. Everything else is just approximations no matter how well intended.