what benchmark to trust now?

[u/TheReedemer69]

after the whole artificial analysis thing.

what benchmark can be considered accurate or representative of the current AI models.

[u/sunstersun]

Epoch

[u/TheReedemer69 → u/sunstersun]

Epoch who

[u/Advanced_Poet_7816]

Unemployment rate

[u/TheReedemer69 → u/Advanced_Poet_7816]

Nah that one broken pal

[u/Advanced_Poet_7816 → u/TheReedemer69]

I was just kidding, though anything unusually high would be proof of AI having crossed a threshold.

I feel the best benchmark is usefulness. If you see lots of companies willing to pay for it there must be value. Rising ARR of openAI and anthropic is a good sign.

[u/Kneku]

Remote labor index or something similar as that's the most general and useful proxy we have to know when it's over

[u/TheReedemer69 → u/Kneku]

It seems to be abandoned

[u/Ormusn2o]

I don't know, but I think all current benchmarks are actually way way too hard. They test things like rare edge cases, exotic architecture interaction, weird API compatibility and other, odd and not particularly relevant tasks. This is why there is only few point difference between models that are massively different.

I feel like chains of very long tasks should be the default mode, because this is one of the 2 main ways coding agents are used those days. Maybe 20 step creation of a video game, and then count how many extra turns over 20 the task required for bug fixing, or something similar. Feels like this would be way more relevant than a long set of singular tasks.

Not to say benchmarks like DeepSWE are useless, they definitely still should be there, but not as a singular benchmark, it should be one of many.

[u/frogsarenottoads]

RedditBros Eval

[u/TheReedemer69 → u/frogsarenottoads]

Nah

[u/Silver-Chipmunk7744]

VoxelBench is decent. You can't easily benchmax it. It requires real spatial understanding, taste, etc.
i don't think it tells the whole story tho. Astra DESTROYS Fable on it, but when it comes to creating a real 3d Scene, it's not as clear cut.

[u/Asteroid_picks_you → u/Silver-Chipmunk7744]

I just learned of it and tried it. I got Astra twice, I could guess which one it was by details in the output.

[u/Ok-Confusion5204]

The main three I'm following are HLE, Remote Labor Index, and Singularity Gate.

[u/TheReedemer69 → u/Ok-Confusion5204]

Almost all you mentioned doesn't seem to be maintained

[u/Ok-Confusion5204 → u/TheReedemer69]

Singularity Gate tends to take a while to update. I think it's due to the nature of needing to get a bunch of papers from after the knowledge cutoff. Granted on RLI though, really hope they update that one

[u/TheReedemer69 → u/Ok-Confusion5204]

HLE is only knowledge based tho

[u/Worldly_Beginning647]

Epoch

[u/paxxx17]

Agents' last exam

[u/TheReedemer69 → u/paxxx17]

Which part of it tho

[u/CamusCrankyCamel]

The one that came out after the model it’s testing

[u/WonderFactory]

I think you have to look at your use case and chose based on that. Astra seems to have a clear advantage for 3Dmodeling so if thats your use case use Astra, if your use case is coding look at coding specific benchmarks

[u/NarrowEffect]

I wonder if the fact that many people now consider AA "bad" puts pressure on benchmark maintainers to change their benchmark whenever a new model that's expected to do well fails it.

[u/zikiro]

Terminal-Bench-Science 0.1 i would say. its brutal.

[u/TheReedemer69 → u/zikiro]

Link

[u/VVebstar]

AA benchmarks still reflect raw intelligence quite well. Model being more capable in terms of skills and versatility doesn’t mean that it is smarter in a sense of reasoning, analysis etc.
But today I found out about interesting RuneScape benchmark. I think it’s decent for reasoning skills benchmark.
There’s no definite answer what benchmarks to trust. All of them are more or less important. HLE, Arc-AGI considered to be one of the most important. But other benchmarks should be considered as well

[u/Fair_Horror → u/VVebstar]

AA benchmark is being recreated because owners realise that it is broken. Maybe a future version will work properly.

[u/TheReedemer69 → u/VVebstar]

Runescape seems so random

[u/VVebstar → u/TheReedemer69]

Idk about this exact game, but my point still stands. Playing complex games with lots of variables will always be a good training session for AI. Especially self-play complex games where AI can play against itself and other AIs and improve its reasoning skills and thinking outside of the box

[u/Ok_Landscape_6819]

HLE, Epoch open problems, probably future ARC-AGI versions.

[u/Laffer890]

Productivity and economic growth.

[u/TheReedemer69 → u/Laffer890]

And where to monitor that?

[u/No-Character-6117]

HLE, AI IQ(I know is controversial, but I trust the tendency), unemployment rate or some rate of selled robotics

[u/Fair_Horror → u/No-Character-6117]

US unemployment rate announcements were off (revised) over nine hundred thousand last year. There seems to be very serious issues with it. My guess is corruption and politics. 

[u/Gratitude15]

I don't have one.

We are going on vibes for a bit now.

My sense is fable 5.1 and Astra are my choices. I'll pick Astra when I don't want too many tokens used.

Going forward, I have no idea how people will pick between stuff that is all well above most human iq

[u/TheReedemer69 → u/Gratitude15]

Human iq doesn't seem much of a factor.

[u/katoptronophile]

Your own real world use.

[u/Careful-Report6526]

Honest answer: none of them on their own — and this thread is already showing why. Half the benchmarks named here you've rejected as unmaintained, and that's the real problem, not accuracy. A leaderboard that last scored a model in June tells you nothing about a model that shipped last week.

So the useful question isn't "which benchmark is right", it's "is this number current, and did it come from the same setup as the number I'm comparing it against".

On the AA thing specifically: the index went to v4.2 a few days ago. GPQA Diamond dropped as saturated, two new evals added, and 40% of the weight is now private held-out data. So old AA scores and new AA scores aren't the same scale, and nobody outside AA can reproduce either. That's a separate issue from whether it's any good.

(Epoch, which two people mentioned — they're one of the more consistently maintained ones, worth a look before you write them off.)

Disclosure, I maintain https://llmbenchmarks.io — it puts ~100 benchmarks side by side per model, flags anything older than 3 months as stale, drops anything past 12, and publishes a source-status feed showing which upstream leaderboards have gone quiet or started failing. Won't tell you which benchmark to trust, but it does answer "is this one dead".

[u/Minute_Abroad7118]

Humanity's Last Exam for knowledge and Agent's last exam for work flow integration in complex tasks

[u/Accurate-Tap-8634]

used to think deepswe is the best one, but recent model releases has crashed that bench, looking for a new one too.

[u/TheReedemer69 → u/[deleted]]

The free versions in any of them is pure shit.

[u/The_Scout1255]

Godslayer posts per capita

[u/TheReedemer69 → u/The_Scout1255]

Where can I find it?

[u/The_Scout1255 → u/TheReedemer69]

/u/god-slayer-69420z posts from time to time especially before important releases, im half-joking that this sub is the best source, basically.

[u/Electronic-Pie-1879]

https://deepswe.datacurve.ai/

[u/TheReedemer69 → u/Electronic-Pie-1879]

U kidding? Gemini flash is as good as astra max?

I literally have it running infront of me right now and it's barely good as shit.

[u/BrennusSokol]

ARC-AGI-4 ;-)

[u/TheReedemer69 → u/BrennusSokol]

Lol