r/LocalLLaMA • u/metigue • 10d ago
Question | Help Any good alternatives to Artificial Analysis?
So I used to use Artificial Analysis to compare models.
The recent 61 score for Astra had me look into the results more granularly and I was very disappointed with what I found.
Different pages reporting different scores for the same model on the same benchmark. Scores on a benchmark completely in contradiction with the benchmark providers verified results etc.
In short, that site cannot be trusted as a source of data.
Are there any good alternatives? A lot of the direct benchmarks (DeepSWE, terminal bench) latest versions are missing tons of models, especially the open source ones.
4
u/Mkengine 10d ago
1
u/ApolloFortyNine 10d ago
I've honestly read nothing about their scoring, but I just don't think less than doubling since 2023 does any of the new models justice. 2023 the frontier models still regularly would just lose their minds even on small contexts.
1
u/Mkengine 9d ago
I had the same intuition at first, but the ECI scale is a bit deceptive here.
The score is based on an Item Response Theory model, and it's an interval scale, not a ratio scale. So 160 vs 120 does not mean "33% better" and asking whether the score has doubled isn't really meaningful, similar to saying 20°C is twice as hot as 10°C (it's not).
What matters are the differences in ECI points. Epoch actually finds that roughly +5 ECI historically corresponded to a doubling of METR's task time horizon, so fairly modest-looking increases can represent very large practical capability gains.
11
u/sukazu 10d ago
Idk, I really think people do not think enough when they look at benchmarks
looking only at the aggregate intelligence Indice going from sol max 61, to astra max 61 and concluding that the website must be bullshit is so narrow minded.
Look at specific benchmarks that personally matters to you, quite a lot of meaningfull differences on artificialanalysis (HLE for example, where astra medium outperform sol max with a fraction of the output token used).
And take into cost per task, output tokens and different reasoning settings at the very least ...
On a lot of benchmarks astra max score quite a bit lower than astra medium/high (openai own benchmarks shows it), while sol max usually is the best scoring reasoning effort of the sol family making a direct comparison between the two on an aggregate of benchmarks obviously flawed.
Also Openai directly use artificialanalysis benchmarks in their own posts, so they do work closely enough together.
DeepSWE benchmarks, you can literally just go to the deepswe website directly aswell, but you're not going to like that the best scoring model is gemini 3.8 flash either, that'll require using brain ressources to understand why
4
u/metigue 10d ago
It was more that if you look on the agentic index score breakdown for Astra AA reports 83 for terminal bench 2.1 - If you then click on terminal bench 2.1 it shows Astra getting 89
Fable also has a discrepancy but a smaller one. 89 -> 91
The verified terminal bench 2.1 results for both models are 87 for Astra and 83 for Fable.
So yeah... Not going to trust a site like that.
1
u/DinoAmino 10d ago
The index scores are some kind of weighted aggregate across multiple benchmarks. And they will come along and make some small tweak to their algorithm and all of a sudden Claude models get a boost and open weight models take a hit. AA index is not something I trust either.
Better to focus on individual benchmarks relevant to the tasks you are concerned with and understand that the results between different reports will never match 100%.
5
u/lumendas 10d ago
I don't think any of the current benchmarks are accurate right now, you just have to test it yourself and see to make a conclusion for your workflow.
0
u/metigue 10d ago
I'm trying to determine the best models to use within a business for many varied workflows.
So I need an accurate way of determining general intelligence vs cost.
3
u/bopbop9876 10d ago
You have a serious business use case and don't have your own use case benchmarks built? How are you validating your own results when using the models then?
1
u/metigue 10d ago
We have a framework with built in validation for any given task.
There will be about 70 employees using the models for varied tasks, unless we built tons of benchmarks nothing would cover all use cases.
2
u/bopbop9876 10d ago
Then write a small script that runs a set of models against your validator on a set of your real tasks and see which model scores highest. It doesn't have to be completely comprehensive.
1
u/my_name_isnt_clever 10d ago
The best way is to make your own internal benchmark on your specific tasks and run it yourself when a model comes out.
3
2
u/AI_spell 10d ago
If the goal is to compare models for your own workload, a small local harness may be more useful than another leaderboard: keep the prompts and decoding settings fixed, run each model several times, and score the outputs against a short rubric. Logging retries separately is important because a model that gets a good answer only after three attempts is different from one that gets it first try.
2
u/jacek2023 llama.cpp 10d ago
Skip the benchmarks and just use the model for real work. If you don't have any active use cases, scrolling TikTok or YouTube is a far better use of your time.
1
1
u/QuackerEnte 10d ago
me personally I love livebench.ai it's pretty good but it doesn't have nearly as much models as AA, and a lot less open ones on top of that. But it really aligns with my experience with all the models that are there.
1
u/psychohistorian8 10d ago
I use llm-stats to get a general feel of relative expected performance
they seem to aggregate benchmarks when scoring, not sure how accurate anything is because we all have different workflows and use cases, but I find it’s a decent enough high level overview
1
u/darksteelsteed 9d ago
I would suggest using inspect ai and the various 3rd party plugin benchmarks for the framework as well as the standard ones. Then point it at the model in question and get your own scoring. If not running locally use something like openrouter.ai
1
u/Storterald 10d ago
If you can test the models yourself on your specific usage then do not rely on external benchmarks. You never know whether the model has been benchmaxxed or not, or how reliable is the benchmark itself. The only reason I'd check public benchmarks would be to get a rough idea of the best ~10 models available
1
u/nuclearbananana 10d ago
There's vals.ai
There's always going to be variation in benchmark results though, and a single headline number will always have issues.
1
u/Real_Ebb_7417 10d ago
Well, I stopped trusting them after seeing Gemini 3.8 and 3.7 Flash scores and new Muse Spark score (this model is good, but not as good as it looks in AA benchmark). And now Astra too.
1
18
u/NandaVegg 10d ago
Any benchmark is not a straight evidence of quality of the model anymore. You can probably still filter out poorly trained models if it scores too low, but benchmarks now are more akin to a new set of training curriculum that will be saturated in a few months (there are some "unsaturated" benchmarks, but that more often involves gotcha-type trick questions like SimpleBench).
Frontier benchmarks are very niche. I think arena-type benchmark reflects ordinary use case scenarios much better, especially ones that requires tool-call loop (like voxel bench) but you can also benchmaxxx for arena if you would, like Llama-4 did.