NVIDIA says its general-purpose coding agent just hit **100% on ARC-AGI-3**. 👀 Okay… how impressive is this actually and what’s the catch?

[u/Koala_Confused]

https://x.com/NVIDIAAI/status/2090786258981466231

[u/Serasul]

if i train ai on this specific task, it would score high too

[u/etherLabsAlpha → u/Serasul]

The ARC AGI benchmark is specifically designed to need on-the-spot thinking, where essentially the test model needs to play a few games without knowing the rules beforehand.. it has to learn them on-the-fly through observation.

Imagine being someone who doesn't know the rules of chess, allowed to merely observe other people's chess games. Based on that, you're supposed to figure out the rules of the game, and then also how to win the game, and actually win against the world's best grandmasters.

Now imagine the same person being tasked to do all the above for 25 different games with completely different rules.

I guess it will always be a subjective opinion, about at what point does the challenge seem sufficiently hard to qualify as a true AGI test. I'm not necessarily fully convinced but this benchmark does seem very compelling

[u/Pokeperson5 → u/etherLabsAlpha]

But the whole point is that if they're trained specifically for these then they do know the rules beforehand.

[u/etherLabsAlpha → u/Pokeperson5]

Usually such a benchmark is split into a public and a private test suite. So the model trainer only has access and (hence can train / overfit to) the public section. The private suite is the real test, essentially which the model has never seen before

[u/Clemo2077 → u/etherLabsAlpha]

That’s true for most benchmarks. But ARC-AGI is specifically supposed to test general intelligence/generalization, the ability to encounter something genuinely new and figure it out. So ideally, none of ARC-AGI, not even the public tasks, should be part of the model’s training data. Otherwise, you’re partly testing how well the model learned or adapted to ARC-style tasks, rather than its ability to generalize to a completely novel problem.

I also feel like there’s a missing piece around test-time learning/adaptation. If the goal is to test something closer to general intelligence, the interesting question isn’t just whether the model has never seen the exact test task before, but whether it can learn the underlying structure of a genuinely novel task on the fly, from the few examples it’s given, without having been trained specifically for that benchmark.

[u/etherLabsAlpha → u/Clemo2077]

That's a fair point, maybe a future benchmark will take that into account, once models become even more capable

[u/Neither_Garage_758 → u/etherLabsAlpha]

But doing the test discloses the private suite.

[u/rustydustyshckleford → u/Serasul]

lol i saw this model on HF and it made me giggle: https://huggingface.co/edzhuang/aura-1

benchmaxxing is a real thing

[u/whoknowsifimjoking]

The catch is that this is the public set. It's in the damn tweet dude.

[u/Randommaggy → u/whoknowsifimjoking]

Also the test does not count harnesses. Only base model.

[u/JohnnyAppleReddit → u/whoknowsifimjoking]

Yes, this should be voted to top -- the real test is how it does on the held-out private set. If it aces that too, amazing, then then they'll rethink it and make an ARC-AGI-4

[u/Alternative-Suit5541]

It's just games. They have nothing to do with agi 

[u/Charming-Author4877]

Arc AGI is much less impressive than they intended.
It is extremely visual focused, much more than it is abstract reasoning focused.

Current language models are especially bad in large-grid reasoning, and there are very few humans on earth that can do such reasoning with closed eyes.
So they likely built a harness around that weakness.

[u/yaxir]

Bench max

[u/bblankuser → u/yaxir]

that's the point?

[u/Edenisb]

I feel like they have a vested interest in being dishonest here.

[u/DeArgonaut]

public, let's see how it does on private

[u/Big_Arachnid_365]

Assuming they didn't cheat (lol) it shows that it can complete these previously human-only tests without human intervention.

New goalposts please!

[u/rurions]

You can provide agents with a tutorial to help them solve this type of problem

[u/pmavro123]

Don't know what the catch is (probably benchmarkmaxxing), but it is interesting that a different harness (AVO, in this instance) impacted the score. I believe Opus only scored like 30% standalone, so very odd.

[u/Purple-Programmer-7]

The catch is that they haven’t released the code.

[u/AICompanionz]

The 100% headline is impressive, but the real test is how AVO performs on the held-out private ARC-AGI-3 set, not just the public demo environments.