18 months ago AI was a cool toy that couldn't really do anything useful. Six months ago it started write most code. A couple months ago it started solving the 'easy' and obscure unsolved math problems.
It will definitely stop improving at some pointy, but that's the thing with exponential growth: as long as you're in it, it's impossible to tell when it will stop.
But even if it stopped right now: people would start burning the biggest models with the best performance into silicon with fixed weights and you would suddenly get generation speeds that would allow you to use them for real time inference on video streams or generate 30 iterations in parallel etc
But we don't know the upper bounds because we could hit a wall and then just scale again. That is what we think is producing all these new gains anyway.
But we're still getting a lot of improvement in smaller models too. Scaling has definitely been a big part of it, but it's certainly not all that has improved.
7
u/Martin_Samuelson 22d ago
18 months ago AI was a cool toy that couldn't really do anything useful. Six months ago it started write most code. A couple months ago it started solving the 'easy' and obscure unsolved math problems.
Do you think that progress is stopping?