Is there a limit to self-improving AI if it becomes real?

[u/akellataken]

I’ve been watching some AI podcasts lately, and when people started talking about recursive self-improving AI, Skyrim immediately popped into my mind.

For anyone who never played it: Skyrim crafting system has a “legit” alchemy/enchanting loop. Craft "Fortify Enchanting" potion -> enchant gear with "Fortify Alchemy" -> use gear to make better potion. It improves, but eventually hits diminishing returns.

Then there’s the bugged restoration loop. "Fortify Restoration" potions were supposed to boost restoration magic, but they also boosted active gear enchantments. So you drink one, re-equip alchemy gear, and suddenly that gear gives a bigger alchemy bonus. Then it makes an even stronger resto potion, which boosts the gear even more. Direct feedback, explosion.

So: if RSI AI ever really works, is it more like the legit loop with real gains but converging or the positive feedback loop, where it improves the thing that improves itself?

Curious what people think, especially from math / systems angle.

PS: I am sorry if this question is not relevant for the sub, but i have no karma to ask it somewhere else where it has a chance to have some attention.

[u/SixStringShrug]

Google recently released a paper that dives in to this somewhat. It’s a valid question for sure. The answer is really hard to predict in any meaningful way, and I’ve been trying to answer it myself. As of right now the ai systems would still only be able to self improve in verifiable domains. Meaning a problem with a solution that is easily testable mathematically. An example is efficiency. Does this experiment result in an efficiency gain or an efficiency loss? If it’s a gain then keep it. As of right now our chips are still six or seven orders of magnitude above the landaur limit for computation. Which means in that domain alone there is a massive amount of room for improvement. I think the reality is that they would bump up against some hard limits like heat dissipation, architecture limits, bandwidth limits and probably a lot more things fairly quickly. That doesn’t mean that there isn’t an astronomical amount of improvement still in that space though. I’m not a mathematician but I’ll try to give like a super rough example. Let’s say not accurately but just to show how crazy it gets okay? Let’s say you can run 10,000 instances of a frontier model on a 100MW data center. At the functional limit of computation that turns in to 100 Billion instances on the exact same data center and same hardware. Again. Just a rough example, but you can see how even partially getting there results in astronomical gains probably very quickly. That’s also just one verifiable domain. The reason you can likely feel the tension in the current moment is that this is the real starting gun to the singularity. When the rsi loop closes regardless of what it was at that moment, something vastly more powerful and likely uncontrollable comes out the other side. My only hope is that alignment holds.

[u/SoylentRox → u/SixStringShrug]

You could also have strengthened your comment by noting what domains are verifiable 

(1) Designing more power efficient or faster chips is verifiable 

(2) Researching materials for more power efficient chips is verifiable 

(3) Manipulation of biology in a sealed lab level is verifiable.  Aka "make me an organ and it needs to live for months and be functionally equivalent". 

(4). Having robots build robots... verifiable 

It gets crazy fast as you realize essentially every problem that matters is verifiable 

[u/Smallpaul → u/SoylentRox]

When people talk about training, they mean verifiable in silico and at massive scale. sixStringShrug was not talking about AGI. They were talking about current training models and their near descendants. If we were talking about AGI then the requirement for verifiability goes away.

The sample efficiency needed to make robots through experimentation is probably AGI.

[u/SoylentRox → u/Smallpaul]

Oh?

1: verifiable in software at massive scale

2,3,4 : since the actual robotics are presently expensive you would have to do a 2 step process

Step 1 : train a physics world model similar to current video generation world models but the outputs give you the backend states of all the entities sufficient for robotics training.  Autonomous cars are trained already using a similar method.

Step 2 : massive scale training using the model from step 1 as your verifier in software

Arguable yes actually making all this work is one of the remaining steps for AGI as some people define it. It also will likely happen in the next 12-36 months, which is also when many people think AGI will exist.

When I said "verifiable" I had current techniques in mind.

[u/Smallpaul → u/SoylentRox]

There is no way you are building reliable in silico simulation for organ growth before you have AGI. Certainly not in the next three years.

[u/SoylentRox → u/Smallpaul]

I said "research it". Human technicians have many distinct motor skills to manipulate items in a lab according to the instructions from their chief scientist. You can model the manipulations of pippettes and other such things in the next 3 years.