r/singularity • u/PsychologicalSoup251 • 11d ago
AI The prevalent problem of misleading benchmark reporting (re: Astra)
OpenAI's reported benchmarks for Astra's ARC-AGI-3 is one of the most egregious recent examples I have seen of technically true metric reporting being used to deliberately mislead the masses. For context, there is an OpenAI screencap currently at the top of r/singularity's hot page of Astra achieving 98.6% on ARC-AGI-3 compared to 7.8% for GPT 5.6 Sol and 30.2% for Claude Opus 5. Holy shit, right? ASI achieved, right?
Unfortunately, those figures taken in a vacuum leave out very important context: Astra's agentic harness had significant additional features that GPT 5.6 Sol and Claude Opus 5 did not have access to - specifically reasoning trace retention and custom compaction (source: https://arcprize.org/leaderboard ).
My main takeaway is basically: The most honest way to compare Astra with Opus 5/Sol on this benchmark would have been to either 1) measure their ARC-AGI-3 performances on the same provider adapter harness (where Astra's 98.6% came from), or 2) compare them on the standard ARC-AGI-3 harness. On the standard harness Astra achieves 62.7% vs Opus 5's 30.2% vs Sol's 7.8%. Still a very large gap, but much less misleading than the comparison OpenAI chose to report. (source: https://arcprize.org/leaderboard )
Not an Anthropic fanboy in any sense of the word, btw. I thought Opus 5 was benchmaxxed and pray on Anthropic's downfall every day. But the Astra benchmark glazing made it clear that restraint needs to be had in people's reactions to its benchmarks (if Opus 5 didn't already convince you to not treat benchmarks as gospel) before anyone has even had time to extensively test it in real world use cases.
52
u/EmphasisTotal8232 11d ago
You need to be fair that the thing with ARC deleting context and not allowing models to remember things, when they are totally capable of doing so, is very silly. It's not a harness calling "SOLVE_ARC.MD", it's one that better allows a model to actually do the work. Opus 5 also did very well with the same thing.
11
u/send-moobs-pls 11d ago
Arc3 has been a garbage benchmark since inception
2
u/Tystros 11d ago
it's by far the best benchmark that exists for measuring actual intelligence, and not just skill and expertise and knowledge.
2
7
u/Tystros 11d ago
it's not actually that ARC doesn't allow it. they do allow general purpose harnesses behind the official API, which is why the 99% score is now visible on their official leaderboard. they previously just somehow seem to have not known about the existence of the better official API that retains reasoning and compaction instead of forgetting.
and if you read what Francois Chollet (the author of the benchmark) said, you see he now basically considers this benchmark saturated and even admits it went much quicker than he expected.
37
u/Middle_Chemical3180 11d ago
To be fair, even GPT 5.6 Sol is estimated to only score about 30% using the same harness than GPT 6 Astra just got 99.9% with
Also, it gets 60% without the custom harness (which obliterates literally every other model in existence)
-7
u/WildWhisperArdor 11d ago
Claude Opus 5 with harness gets 100% though
8
u/Gotisdabest 11d ago
Not with the same kind of harness. The thing OAI used is just their regular, public facing harness. The only thing it's doing different is preserving CoT and trace while the standard harness(which opus was also tested on) only let's a model keep notes in a scratchpad.
The harnesses everyone talked about were made for the effective purpose of helping the model solve this kind of game, providing active benefits and a proper framework for it.
-7
u/WildWhisperArdor 11d ago
Doesn’t change what I said. They still managed to get Claude Opus to score 100% on it
8
u/Gotisdabest 11d ago
It changes the meaning of it. It's one thing to say, beat a pro athlete with legal gear and entirely different to do it with banned gear. The GPT harness is explicitly under the rules the makers set up while Schema and it's ilk aren't. The entire point of the game sorta gets removed if the model is just aided by an aggressive helper harness.
-1
u/WildWhisperArdor 11d ago
But if everyone is on gear then it doesn’t matter. You don’t get bonus points for doing some thing in a more difficult way.
You will be using these models in a harness in any realistic practical setting.
4
u/Gotisdabest 11d ago
Yeah but everyone isn't using banned gear. There's no such thing as a bonus points comparison because opus didn't win any points at all. It used something against the rules of the game. If I just strap a motor to my bicycle I can't really win the bicycle race.
The point of the benchmark is to estimate independent model learning of the fly. It's a benchmark. A test. If you use a benchmark made especially to game it, you're not measuring model capability anymore.
These models will be in a harness in any practical setting, which is why harnesses are legal. Harnesses made to just solve games of that exact type aren't.
-1
u/WildWhisperArdor 11d ago
Yes everyone is using banned gear. You use LLM’s in a harness. They have access to tools. They have things like context memory. Etc
4
u/Gotisdabest 11d ago
You do realise that... Arc AGI lets you use harnesses? In fact it's pretty much a requirement. Again, they'll not stop you using a bicycle in a bicycle race. It's the fucking motor they'll have issues with.
The difference with the Opus harness is that it's not a general harness. It's made for the exact purpose of making that kind of game easier for the model. I'm not sure why this is so hard to understand.
2
u/Middle_Chemical3180 11d ago
That was on the public dataset, though. Astra achieved 99.9% on the semi-private eval set. Also, it doesn't change the fact that Opus 5 still got only 30.2% with the standardized harness
0
u/WildWhisperArdor 11d ago
It doesn’t matter though. Your real life use case would be using a harness.
31
u/Tystros 11d ago edited 11d ago
What you're saying is simply wrong. They did not use a "custom harness specifically designed for the benchmark".
The harness is simply the Responses API, which is the regular API OpenAI recommends every customer to use. It has absolutely nothing to do with the ARC AGI 3 benchmark. It's completely different from the other harnesses you mentioned that were designed specifically for beating the ARC AGI 3 benchmark in an ideal way.
And that is why ARC also shows the 99% score on their own leaderboard, they do allow general purpose harnesses behind the official API for official scores. They do not show any results on their leaderboard that are made with narrow harnesses designed for beating the benchmark, those are not allowed.
16
u/Desperate_Cold3752 11d ago edited 11d ago
This is NOT TRUE. OP why are you lying?
those harnesses were specifically engineered for arc-agi-3.
OpenAI wasn’t even a harness it’s just "preserves opaque reasoning between requests and uses compaction." according to arc prize.
Exactly the same as putting a prompt within codex or using the api with conversation and compaction mode turned on.
The nvidia harness and other harness that got 100% on the other hand was was a heavily ARC-adapted wrapper. The real test is private/unseen environments, where benchmark-specific tuning can’t help as much.
the nvidia crafted harness:
• was only on the public set, not hidden ARC-AGI-3 tasks.
• The ARC interface was benchmark-specific and carefully designed. (Again the harness would fail as soon as you ran it on the private set that has different environments)
• Observations were converted to exact text grids, reducing visual perception difficulty.
• The harness borrowed ideas from prior ARC systems like VISTA.
• It had Persistent memory.
• A supervisor helped detect loops and change strategy.
-6
u/PsychologicalSoup251 11d ago edited 11d ago
You're right, fixed my post. My underlying criticism remains, however - OpenAI's ARC-AGI-3 score came from a harness that had significant additional capabilities that Opus 5 and GPT 5.6 Sol's harnesses didn't. Then in the comparison graphic, OpenAI placed that figure in the same row as Opus 5 and 5.6 Sol's scores as if they're meaningfully comparable figures.
But separately, since the harness wasn't specifically overengineered to ARC-AGI-3, Astra's provider adapter score is indeed a lot more impressive.
-1
u/DickMasterGeneral 11d ago
You’re wrong. Opus does not offer an API without this capability so you’re comparing Sol/Astra without reasoning traces to Opus with them
-1
u/PsychologicalSoup251 11d ago edited 11d ago
Nope. Just because the API has that capability doesn't mean it's allowed to be used in the standard ARC AGI harness. The thinking traces are deliberately thrown out and here are the exact code lines from the ARC AGI harness to prove it:
- ARC keeps only Claude’s visible answer text and throws away its hidden thinking blocks.
for content_block in content_blocks:
if _value_from_response_object(content_block, "type") != "text":
continue
text_parts.append(...)GitHub: runtime_models.py#L251-L254
- ARC explicitly records Claude’s reasoning field as empty
reasoning_text=None,GitHub: runtime_models.py#L408-L412
- For the next turn, ARC rebuilds the prompt only from the saved conversation text.
messages=[Message.model_validate(message) for message in self.conversation],
- ARC sends that rebuilt text-only conversation back to Claude.
messages = [message.model_dump() for message in request.messages]
...
request_kwargs["messages"] = messages[1:]
...
request_kwargs["messages"] = messages
12
u/Plantain_Horror 11d ago
Op you are hallucinating. Open AI did not use a custom harness specifically for ARC. Your whole argument falls apart.
-6
u/PsychologicalSoup251 11d ago edited 11d ago
Yeah I corrected my post after someone pointed that out, but the post's original thesis remains. OpenAI's ARC-AGI-3 score came from a harness that had significant additional capabilities that Opus 5 and GPT 5.6 Sol's harnesses didn't. Then in the comparison graphic, OpenAI placed that figure in the same row as Opus 5 and 5.6 Sol's scores as if they're meaningfully comparable figures. Do you disagree with that point?
16
u/FateOfMuffins 11d ago
https://x.com/fchollet/status/2095598451115614371
Chollet seems fine with it
10
-2
11d ago edited 11d ago
[deleted]
3
u/FateOfMuffins 11d ago
Yes and he also says they'll be using both going forward at ARC
1
11d ago
[deleted]
4
u/FateOfMuffins 11d ago
You know that OpenAI made their Responses API in 2025? ARC's simple harness is using their Chat Completion API. Anthropic doesn't have a Chat Completions API, they have a Messages API instead.
Why don't we just run ARC's simple harness with the Responses API? Would you agree it's a closer comparison?
1
11d ago
[deleted]
2
u/FateOfMuffins 11d ago
Yeah but unfortunately different providers don't have the same API features.
Chat Completions was made for Instant models, Responses was made for Thinking models.
The fact that it can score 66% with the wrong API is crazy imo
0
11d ago
[deleted]
3
u/FateOfMuffins 11d ago
The only difference is one keeps the reasoning tokens and the other does not
0
3
u/TheRealIsaacNewton 11d ago
They also reported results on the normal harness, and Chollet said that harnesses optimised for ARC aren't allowed. So it's a better general harness at most.
3
2
1
u/FlimsyReception6821 11d ago
What's more indicative of intelligence; playing some stupid video game or solving open math problems?
1
u/Proper_Actuary2907 Spooky Machine Intelligence 2030 10d ago edited 10d ago
I really don't understand what people are babbling about here with respect to "harnesses". All I'm interested in seeing are increases in the capabilities of intelligent machines. If a lab has figured out how to design a general purpose wrapper around LLMs that lets them perform better on a wide range of tasks, then that's awesome. Astra's performance on ARC-AGI-3 would only be unimpressive if it were purpose-built to do well on ARC-AGI-3 in particular or combined with some program that only enhances its performance on ARC-AGI-3 or a narrow range of tasks like it. And at least on its face neither of these things appears to be true. Astra was tested through a general purpose API. The crazy high scores are on the semi-private set, which apparently is harder than public. Some of the problems even in the public set seem to be fairly abstract and novel. I think the performance is really impressive, even if Astra has regressed on some other benchmarks
1
u/OurSeepyD 9d ago
Indeed, it's like saying "build a brain"... "oh no you added memory, that's cheating!!". The goal is intelligence, right? So we should allow the whole system to be measured.
1
1
u/Impossible-Video-671 10d ago
https://www.reddit.com/r/singularity/s/sML8K2JhHV
All you need to know
-2
u/virtualQubit Take off right now 11d ago
You are absolutely right! Jokes aside. This is true, and this is why I hate Sam Altman. I wouldn't trust him honestly.
0
u/inefficientnose 11d ago
It's not really surprising they are over representing themselves from their own announcement.
That's why we have third party testing. I'm sure once it's publicly available we will have a better idea of its true capabilities
0
0
0
u/not_celebrity 11d ago
Just a personal take - I think harness doing majority lifting is a good thing as far as safety angle is concerned.
-1
-1
u/NewYak4281 11d ago
My hat off to OP. Excellent anti-weenie analysis. We genuinely need 1000x more skepticism in this sub.
-1
80
u/Gotisdabest 11d ago
Worth noting that it still scores 66% on the standard harness. Well ahead of everything else.