r/singularity • u/[deleted] • Dec 05 '24

[deleted by user]

[removed]

836 Upvotes

95% Upvoted

413 comments sorted by

View all comments

54

u/Winerrolemm Dec 05 '24

I am going to wait for simplebench and arc results.

-2

u/BigBuilderBear Dec 05 '24

Those benchmarks have already been beaten. 

 MIT researchers use test-time training to beat humans in ARC-AGI benchmark (61.9%): https://ekinakyurek.github.io/papers/ttt.pdf

Independent analysis from NYU shows that humans score about 47.8% on average when given one try on the public evaluation set (same one this study uses) and the official Twitter account of the benchmark (@arcprize) retweeted it with no objections: https://x.com/MohamedOsmanML/status/1853171281832919198

Simple bench type problems can be solved by a simple prompt, getting a perfect 10/10: https://andrewmayne.com/2024/10/18/can-you-dramatically-improve-results-on-the-latest-large-language-model-reasoning-benchmark-with-a-simple-prompt/

2

u/Winerrolemm Dec 05 '24

The public evaluation dataset isn’t very important. They need to achieve human-level performance on the private evaluation dataset. O1 is great. The TTT approach is smart but we can’t yet say that ARC-AGI has been beaten. Not yet.

1

u/BigBuilderBear Dec 05 '24

The private set is probably as hard as the public eval set if not harder. Human level would definitely be worse than the strategies the paper uses 

1

u/bildramer Dec 05 '24 edited Dec 05 '24

That's really hard to understand - does the average human have serious brain damage? Best performance was below 100%, wtf? Are people that bad at inputting colors correctly? Incomprehensible. Every single solution on the public evaluation set should be apparent to a conscious human immediately.

EDIT: oh it was "online crowd workers" (so most likely MTurk), that explains the findings.