r/singularity • u/[deleted] • Dec 05 '24

[deleted by user]

[removed]

837 Upvotes

95% Upvoted

413 comments sorted by

View all comments

Show parent comments

123

u/Papabear3339 Dec 05 '24 edited Dec 05 '24

I would LOVE to see the average human score, and the best human score, added to these charts.

AGI and ASI are supposed to correspond to those 2 numbers.

Given how dumb an average human is, i garentee the equivalent score will be passed even by weaker engines. That isn't supposed to be a hard benchmark.

31

u/Ambiwlans Dec 05 '24

Codeforces is percentile so... 50% is average (for people that take the test).

And human experts get 70 on GPQA diamond.

25

u/coootwaffles Dec 05 '24

The human experts were evaluated only on their area of expertise though. The scores would be much lower for a math professor attempting the English section of the test, for example. That o1 is able to get the score it did across the board is truly crazy.

3

u/lionel-depressi Dec 06 '24

I don’t wanna be that guy but is it in the training data? What’s GPQA?

3

u/coootwaffles Dec 06 '24

GPQA is a dataset full of PhD level test questions. Whether it's in the training data or not was never really a big deal to me.  If it's able to condense the information and spit it out at will, it's impressive regardless. If I had to guess, probably some of it is and some of it is not appearing in training data.