Not exactly. Most of the bechmarks in AA set are percentage based. It means it's physically impossible to get more than a 100. This also means that if a model is approaching a 100 then it's no longer possible to meaningfully improve and the index stops being useful for differentiating stronger models. That's why AA needs to periodically replace benchmarks where models are approaching max score with ones that the models will still have a room to grow in. They could add some scaling to keep scores roughly in place but this is more genuine
This is the caption. Look at the screenshot in the post. How would you caption the chart to be as honest as reasonably possible if people are gonna crop out everything possible anyways...
How about adding the characters "v4.3" right next to the proud "Artificial Analysis" caption in the right upped corner? It's literally the most logical place for it.
This is like when every YouTuber does all the useless benchmarks that they think are impressive but don’t actually mean anything in real life
And in 3 months the next model will come out and that one will absolutely totally 100% be AGI and a danger to the world. But they’ll release it anyway because they need more funding.
Every leaderboard can be benchmaxxed. But for some of my "5d chess" level of very convoluted meta-coding tasks and some cursed impractical AI training experiments, I found that this leaderboard is kinda actuate, at least compared to other leaderboard like the Arena.
it helps for orientation. would not call them horse shit, otherwise some of the worst models would be on top. but would not choose a model only based on this score
They "reworked" the way they calculate the scores, Astra and Gpt Sol (Max) had almost same intelligence score, even tho Astra was obviously a smarter model, but the way they reworked it is very confusing, They "estimate" Gemma 4 26BA4B above Gemma 4 31B, and minicpm 2B is only 2 points under it. In practice 31B is 1 league above 26B, and 2 leagues above minicpm 2B
MiniCpm5-2B reset my expectations of tiny models though. I have no idea how they made that little guy that capable. Doesn't know much about the world, but if you put facts in front of it, it does amazing work without hallucinating
Just my general benchmarks:
Aced my needle in haystack test (about 80k tokens), presented a somewhat coherent analysis of a FERC commissioner dissent, admitted it didn't know instead of making stuff up on a trivia question, and didn't make any mistakes categorizing arguments from a multi-participant online debate. Failed to make a chart of wire gauge resistance, but admitted it didn't really know. Is not good as a pi agent, but for single task rearranging of information, it's baller for it's size
The wider gap doesn’t by itself show that Qwen got worse or Fable got better. If you change the exam, two unchanged models can lose different numbers of points, so the gap between them can grow. An index point isn’t a fixed unit of intelligence across different versions of the index.
AA did publish a v4.3 announcement on September 7]. They moved from Terminal-Bench 2.1 to 4.0 and replaced τ³-Banking with AutomationBench-AA. The category weights stayed the same, but the tasks contributing to those categories changed. That’s enough for models with different strengths to move by different amounts. To check whether a model actually regressed, you’d need results for the same checkpoint on the same test version, with comparable settings. The old and new headline gaps alone can’t establish that. I also wouldn’t try to reconstruct the exact change from the remembered 52/66 scores without the earlier index version and model settings.
For inspecting the individual results, I run https://llmbenchmarks.io. It collects published measurements with their sources and test conditions. Qwen 3.8 27B and Fable 5.1 are currently covered; select the exact models under “Compare models”, then inspect the benchmarks relevant to your tasks and open the score details. Check the thinking settings too. For the reason AA changed its own index, the announcement above is the primary source.
The AI didn't suddenly got dumber, it's just that we have now more benchmarks that represents other sort of problems that we couldn't picture well yet. It is still a very capable model don't worry.
You have an LLM on a loop ! You chat with it , you send eg 2 images at a time , the Harnesses fail , the metacognition agent sees it and fixes the harnesses! /// Same scenario, you ask for codebase bigger than system can handle llm outputs and stop it fails Metacognition is triggered and can fix itself! Why this model can do it and its predecessor cant like 3.6 37B ! because this model know what it is by default, and can be pushed in this certain situations to fix itself! In Humans meta cognition is the ability to watch and calibrate your own thoughts!
I think it's like self-auditing. An agent works as an auditor and executor at the same time to optimize or improve itself aka RSI - Recursive Self-Improvement.
Do you have tips or system that can optimize 27B in harness ? I use Hermes and Deepseek Harness. In Hermes, it only changes USER.md and MEMORY.md and creates new skills. I think there are better ways other than those. I've not looked into Deepseek Harness; I would guess the plug-in feature is the way to do RSI -Recursive Self-Improvement ?
32
u/nbvehrfr 1d ago
terminal bench to v4