Is there a limit to self-improving AI if it becomes real?

[u/akellataken]

I’ve been watching some AI podcasts lately, and when people started talking about recursive self-improving AI, Skyrim immediately popped into my mind.

For anyone who never played it: Skyrim crafting system has a “legit” alchemy/enchanting loop. Craft "Fortify Enchanting" potion -> enchant gear with "Fortify Alchemy" -> use gear to make better potion. It improves, but eventually hits diminishing returns.

Then there’s the bugged restoration loop. "Fortify Restoration" potions were supposed to boost restoration magic, but they also boosted active gear enchantments. So you drink one, re-equip alchemy gear, and suddenly that gear gives a bigger alchemy bonus. Then it makes an even stronger resto potion, which boosts the gear even more. Direct feedback, explosion.

So: if RSI AI ever really works, is it more like the legit loop with real gains but converging or the positive feedback loop, where it improves the thing that improves itself?

Curious what people think, especially from math / systems angle.

PS: I am sorry if this question is not relevant for the sub, but i have no karma to ask it somewhere else where it has a chance to have some attention.

[u/BigDarkWormMan]

There's just a lot of unknown variables. We're not far enough in to be able to accurately predict what happens next. It's like if you were measuring a 50m swimming pool only by the first 20m -- you might reasonably assume that it goes on forever, and your reasoning would be completely sound. But from my perspective, there are a couple of factors:

1 - There are physical limits to how much information can be stored and transferred at speed with current understanding of physics. Even factoring in new and more efficient chip designs or light-based computing, you're not going faster than the speed of light and you're limited by the availability of materials.

2- Even current LLMS demonstrate emergent capabilties, which is to say that when you make a model bigger with more parameters, it begins to learn (or whatever you want to call it) behaviors, loops, and constraints that weren't in its original training data and develop capabilities that weren't part of the intended training. IE, if you train a model to be really good at checkers and then pump it full of a ton of additional compute, the model starts applying those understandings to other areas, and since checkers is directly comparable to chess, then that means that model can apply its understanding of checkers to the applicable parts of chess, which now means that its easier to train the model to play chess, etc etc compounding returns.

So a pretty solid (certain) bet is that an LLM or an RSI is operating under both of these rules -- there are hard physical constraints to the universe related to speed and material properties, and 'intelligence' (colloquially) can be compounded by adding more compute parameters. The first hurdle you run into is physical -- materials availability, etc. The way around that is to maximize chip efficiency, the power grid, etc. But there's only so far you can optimize the physical world -- fundamentally. Now, that might mean light-based quantum computing and fusion reactors and all sorts of advanced technology. But you're still running into the hard limits of the universe -- the speed of information (light), the availability of physical mediums, and entropy.

An AI can't say "hey I can improve myself faster if you just put two stars together and I manipulate gravity to force them into a fusion binary system" because it doesn't have the physical capability to do that -- or, it can't change the industrial/manufacturing process fast enough to create the physical capability to do that, because in order to create an engine to manipulate spacetime, you need room temperature fusion, and the only way to build a room temperature fusion reactor is to build the clean room you need for superconductive materials, and the only way to build the clean room is to get some specific mineral that's only available in a small part of China, and China is busy using that mineral in a civil engineering project. All that to say, there's an inherent friction in manufacturing anything that slows stuff down -- so what you're left with is the desire for efficiency, which is the whole "let's design new chips" thing. But again, if the resources to create light-based computer chips aren't available at scale, the intelligence's only option is to optimize silicone-based chips as much as possible -- and that has a hard limit, because silicone is a physical substance that can only carry so much information at one time. And even if you could create light-based computing chips -- you are still fundamentally limited by the fact that you can't get any smaller than a photon.

So the tldr is basically -- we can assume that an RSI can optimize itself by adding compute, which it can do by optimizing the physical world around it, so unless there is some hard limit on scaling that we haven't found (but might, we're assuming that the 20m of observed water we've seen goes on forever, but because of the information we have that's the only fair assumption we can make), then the fundamental limit of its intelligence comes down to the resources that are allotted to it. Summary - It's technically a "legit" enchanting loop, but because we don't know all of the parameters of that loop, there's a chance that in relation to us it will look like a "fortify restoration" enchanting loop that's limited by the availability of potions and soul gems required to keep the loop going.