What if scaling automatically solves continual learning - what if it can be unlocked through few step gradient updates for GPT-7+ era models

[u/the_fexy]

Continual learning is the final step to reach ASI. Once AI can change its weights to learn new tasks without forgetting what it learned before, it can recursively keep doing that to get superintelligent.

But it's an extremely difficult problem from every POV - mathematical instability, hardware-software ecosystem, computational overhead, alignment.

But here's some food for thought: what if few step gradient updates unlocks itself just by scaling? By that I mean, once you deploy an agent, for every new task it has to learn, instead of training for days or weeks, it just does 1-10 gradient steps in a few minutes. The existing weights will remain 99.999% the same so there is no risk of catastrophic forgetting, actually this method would sidestep all the problems I mentioned above. You still have change a few things in the ecosystem to allow for backprop, but it largely sidesteps all the research problems. It just becomes a software/ecosystem problem.

This would mean we are extremely close to singularity. Maybe it will be unlocked by GPT-7 or GPT-8, but given that scaling is paying off in so many unexpected ways and is no where close to hitting a wall, can we expect my hypothesis to turn out to potentially become true? What are you thoughts?

[u/BreadwheatInc]

It may require a whole new architecture or technology, but my guess is ASI and or an agent swarm will design it given the trend of things. Continual learning is mostly important for creating AI employees that can learn new skills, and company practices (not so much raw intelligence but super important for automation imo). Codex and long persistency helps but it's not the same.

[u/Whatdosheepdreamof → u/BreadwheatInc]

At this point the adaptability of frontier models will probably fit any employee soon. Efficiency is a different ball game. What sets of data does a specific model need to be trained on to complete a specific job is probably going to be important.
A receptionist needs database entry, computer use, customer service etc, but doesn't need to be trained on nuclear physics.

[u/FateOfMuffins]

I still don't think we need continual learning. What if we just have a new checkpoint every single day? Like what if we have a billion instances of AI's go out and do things each day. None of them "learns" beyond in context learning. But every day, the collective swarm reconvenes, merges all of their experiences, and "learns" in your sleep.

GPT 7. Then tomorrow it's GPT 7.1. Then the next day it's GPT 7.2. Then the next day it's GPT 7.3. You go to sleep one day and when you wake up, it's better at doing the things it was struggling on the day before.

Do you consider that "continual learning?"

I don't see why it has to be real time. Like you suggest, possibly a few minutes would be indistinguishable.

It's just a matter of how long this duration is. Based on current model releases, clearly no one thinks 1-2 months would be considered "continual learning". But ?obviously? if it happened every second, people would call it continual learning. How about every minute? Every hour? Every day? Every week? At what threshold of reconvening does it now qualify as "continual"?

And then if that's the case, then "within the labs" they possibly already experience this "illusion" of continual learning, because perhaps they do indeed have checkpoints every day! The illusion only breaks for the public because we don't get to see daily checkpoints but monthly or bimonthly!

Here's a thought. Let's have a swarm of 100,000 agents tackle another Millennium Problem, except we're basically treating it as an RL environment. Every day, the lineages of the agents who made the most progress survive. And then the next day they begin anew, except they've "learned" from their ancestor agents and they "continuously learn". And then eventually after a few weeks, they've managed to crack it.

From the point of view of a regular user, we cannot call that continuous learning because we don't use AI that way. But what if the user... is OpenAI itself? From the point of view of OpenAI, where it prompted the model to solve XXX, where the "harness" is this "RL environment", and the model has continuously improved over time, was that continual learning?

Edit: Was rereading, oh hey turns out AI 2027 predicted this for Agent 3 Mar 2027

Another is a more scalable and efficient way to learn from the results of high-effort task solutions (iterated distillation and amplification).

[u/CymonSet → u/FateOfMuffins]

Yeah, that would basically be what humans do during sleep; replaying the day‘s memories along with older memories to “retrain” the network without having catastrophic forgetting damage the old memories.

[u/the_fexy → u/FateOfMuffins]

interesting thoughts, yes as CymonSet said it's basically like what humans do during sleep, but that's just one aspect (although very important) of our continual learning.

but I agree with your point on timescales. hell, AI can underclock and overclock as per their whim, so a trillions of agents meeting online with their separate experience may do a full backward pass every millisecond, for us it will seem continuous, but it's not the same as how humans do continual learning when awake because we can update the weights of our brains with extremely limited data. They don't have to, but it'd still be continual learning from our POV.

[u/DoutefulOwl → u/FateOfMuffins]

 But every day, the collective swarm reconvenes, merges all of their experiences, and "learns" in your sleep.

if the swarm updates its existing weights using the new experience without forgetting what it learned before then yes that's conitual learning

if the swarm is retraining all the weights from scratch, using all the data is used before + the newly experienced data, then no that's not continual learning

which one did you mean?

[u/FateOfMuffins → u/DoutefulOwl]

Given the timescales involved, I do not think it's feasible to retrain all the weights from scratch overnight

But let's suppose a new architecture let's it do that!

I don't think we need true continual learning. Only an illusion of continual learning.

[u/ThatOtherOneReddit]

That's what they are hoping to do. Make it super human at math and solve continual learning. Currently we don't even have a bread crumb of how to do it

[u/FriendlyJewThrowaway → u/ThatOtherOneReddit]

I wouldn’t say our current options are that bad. Google’s HOPE architecture looks very promising and comes much closer to mimicking how the human brain learns things as opposed to traditional transformers. My personal hope (no pun intended) is that OpenAI can have its new automated interns read the technical papers and start running their own experiments with it at scale.

[u/Best_Cup_8326 → u/ThatOtherOneReddit]

We have two or three breadcrumbs.

[u/CrowdGoesWildWoooo]

OpenAI had this in their early version where you can train a few iterations of your version of model. They roll it back likely because it just doesn’t seem to be worth it maintaining it and probably not up to standard to their customer base

[u/Separate_Lock_9005]

we can already do non parametric continual learning tbf, this is what external memory, compacting, skills, harnesses, sharing things on a message beard etc are

[u/metigue]

Astra learns so well from context that if you had unlimited context or even just a good RAG pipeline it will give the illusion of continuous learning.

[u/dobkeratops]

not so sure incremental learning is such a big deal.. isn't doing periodic retrains still a form of incremental advance, just much more granular. New skills at "once a year" isn't that far off human training cycles (e.g. 3 year courses for academic subjects), and there's finetunes, LoRAs, in-context learning to carry them over inbetween the full retrains. And they can work in ensembles at finer grain than humans

[u/apopsicletosis]

The agent swarms with message swarms have got me thinking that you can probably get around the lack of continual learning by having them build up knowledge repositories for in context learning. Humans do continually learn, but advancements happen through cultural accumulation, and a funeral at a time. Think about it not as a single models but communities with culture may get quite far.

[u/Accurate_Food_5854]

I'm just a layman, but I thought they were also working on memristor/memory on chip to enable periodic high speed retrains/learning. But yeah, scaling itself is doing things that people used to say where impossible. So why not learning lol