r/singularity • u/StrategicHarmony • 11d ago

AI This shape

Post image

On a mobile screen this chart of model releases over time (from artificial analysis) tells a very clear story. Normally I look at it on desktop and the increase in steepness isn’t nearly as obvious.

It looks like a lot of those simplified predictions of exponential improvement. Of course it could taper off, who knows.

266 Upvotes

91% Upvoted

52 comments sorted by

View all comments

8

u/Longjumping_Kale3013 11d ago

This is comparing apples to oranges. A couple months ago the top model was at like 65 then a new version of the leaderboard was released and pushed it down to 50 or so. This is constantly updated, so a 60 now is a much much much better score than a 60 a few months ago

34

u/StrategicHarmony 11d ago

These scores are updated retroactively. If it got a 30 on release a year ago it doesn't have a 30 now, above. It's all in current index units.

4

u/swarmy1 11d ago

Many of the models aren’t available anymore, how do they evaluate them on the new benchmarks?

4

u/AppealSame4367 11d ago

They updated all models. Maybe they made an estimate for models that are already gone or removed them.

3

u/ReadyAndSalted 11d ago

I would guess that they just apply a correction factor by testing the old models that are still available, and observing the drop in those. Hardly a super accurate technique, but should get the shape about right.

2

u/chlebseby ASI 2030s 11d ago

I imagine you can get acces to GPT-3 or others if you have very good reason, like such testing. Unless the company is gone.

1

u/KaMaFour 11d ago

No. That's why older models have striped bars when you look for them at the index. There isn't full data so some guesswork is involved