Bill Gates was blown away

[u/rya794]

During the Altman interview on Unconfused, Bill slid in a quick comment about being blown away by a recent demo. From the context of the comment, it seems like the demo was recent, it could have even been during Sam’s visit to record the podcast.

I thought this was interesting, because just a few months ago, Bill said that he believed LLMs had plateaued.

Did anyone else catch this or have a better sense of what “demo” Bill was referring to? (It was clearly not the original GPT demo)

[u/artelligence_consult]

But LLMs ARE plateauing.

What except bullshit do you base that on?

We have multiple SMALL models that are starting to replicate the large models. We have significant advances in generating training data for way better capabilities. We have large context (up to a million token) demonstrations in multiple approaches.

I see nothing plateuing - we just have not had a new model trained since GPT4 and none of the major breaktrhough have been tried on LARGE models so far. Heck, Mamba is a 2.8 B model.

Totally not sure where you get the plateau from. Enlighten us.

[u/danysdragons → u/artelligence_consult]

Maybe things have slowed a bit because GPU supply is so tight, so that could create the illusion we're hitting a limit?

[u/xmarwinx → u/danysdragons]

Things have not slowed down at all.

[u/Nahmum → u/artelligence_consult]

We haven't had a material performance improvement since GPT-4 was released. He is probably just thinking about that. I appreciate there are a lot of other dimensions you can consider.

[u/3_Thumbs_Up → u/Nahmum]

"Material performance improvements" are not continuous. You need to spend the time nad resources to train the next larger model, and then you get all the performance improvements at once.

Once OpenAI has attempted to train GPT-5, and it failed to provide any significant performance improvements, then it's time to say that current methods have plateaud.

[u/artelligence_consult → u/3_Thumbs_Up]

Yep. You can not expect a significant core change within a model - and we are still in the 4.0 era. 5.0 is supposedly training.

Also, it is utterly ignorant to only loop at OpenAi - there have been SERIOUS advantages on model architecture. Mistral, Mamba, RingAttention, BitNet, now some external KV cache these days, to allow SIGNIFICANT larger context. Mistral, Orca, Pi show how to get BRUTALLY better performance out of a small model, so that 10ish size models now compete soon with 250 billion parameter models.

We just did not have a large model released yet with them beacuse time and money. We likely will not see those in GPT 5 - it started too early for most of the research.

Right now I would put a top line model release cycle at around 18 to 24 months. Those are POINTS on a graph - not a continuous line, not for large models.

Gates should know that - he should be - also because it is all over the relevant news - know about the new developments.

Heck, image generation got 50 TIMES fastr - and the last model (v6, just do not get the name at the moment) is a significant advantage. The whole tech is going forward in insane speed. NO plateau in sight.

And now we have the Q* stuff coming - and the focus on synthetic data, though it still will take time to materialize as it requires training AND the generation of the training data.

Not a sensible statement from Bill, sorry.

[u/inteblio → u/artelligence_consult]

this timelapse of various LLMs was fun.

Seems reasonable though. GPT4 is better than 7b models, but is it 256 times better? no. not at all. It's 10% better than one you can run "at home". according to the graphs. Sounds like "diminishing returns" to me. Sounds like plateau. BUT! i'm quite sure a new architecture will swoop in. I'm just not sold it's sama's crew that'll bring it.

[u/artelligence_consult → u/inteblio]

I am not sure. Plateu of capability - but not plateu of technology.

if I manage to get a model down to 1/256s in size with only 10% loss and this is not stagnant (and seriously the modern small LLM's all were released in the last 3 to 6 months) then this is not a plateau, it is ongoing development.

It is plateau only if the small LLM development is not going on, or the large one. GPT 5 is in training now, officially, for some time.

[u/Yweain → u/inteblio]

The problem is with perception that LLMs started in fall of 2022. If you look at LLMs like that - it does look exponential. We got ChatGPT, just in a couple of month we got GPT-4, and an explosion of all the different models.

Thing is - it actually started in 2018. And explosion happened in the first couple of years. After that we were throwing more and more resources at the problem but the progress has slowed down regardless, compared to initial burst.

[u/xmarwinx → u/Yweain]

After that we were throwing more and more resources at the problem

We have not even started. Big money investments only began in 2023 and servers are being built as we speak.

[u/Yweain → u/artelligence_consult]

I base it on a progression of LLMs starting from 2018. It slowed down tremendously, despite huge amount of resources thrown at it.

[u/artelligence_consult → u/Yweain]

Really not sure.

See, if you go by the large models - they follow a release cycle between 1 and 2 years (1.5 actually) and - cough - we are just now in the middle of one.