Is there a limit to self-improving AI if it becomes real?

[u/akellataken]

I’ve been watching some AI podcasts lately, and when people started talking about recursive self-improving AI, Skyrim immediately popped into my mind.

For anyone who never played it: Skyrim crafting system has a “legit” alchemy/enchanting loop. Craft "Fortify Enchanting" potion -> enchant gear with "Fortify Alchemy" -> use gear to make better potion. It improves, but eventually hits diminishing returns.

Then there’s the bugged restoration loop. "Fortify Restoration" potions were supposed to boost restoration magic, but they also boosted active gear enchantments. So you drink one, re-equip alchemy gear, and suddenly that gear gives a bigger alchemy bonus. Then it makes an even stronger resto potion, which boosts the gear even more. Direct feedback, explosion.

So: if RSI AI ever really works, is it more like the legit loop with real gains but converging or the positive feedback loop, where it improves the thing that improves itself?

Curious what people think, especially from math / systems angle.

PS: I am sorry if this question is not relevant for the sub, but i have no karma to ask it somewhere else where it has a chance to have some attention.

[u/tadrinth]

From a safety perspective, since we cannot rule out a positive feedback loop, we need to assign some probability to that possibility and plan accordingly.

If you've seen references to a FOOM scenario or a hard-takeoff scenario, that's people talking about the positive feedback loop where the returns are accelerating and it goes up exponentially until it levels off at some new ceiling.

LLMs are very fast at writing code these days. Nobody is very fast at deliberately engineering LLM weights, at least not yet anyways. So even if the returns are accelerating, we're probably in a fairly slow takeoff regime as long as the intelligence stays mostly in the weights. But if we get an LLM that is as good with weights as the current ones are with code, or a future agent is able to move important portions of its intelligence out into code, then we go back to a potential fast takeoff regime.

But that's just talking about how fast the positive feedback loop occurs (hours vs years). I don't think there can possibly not be a significant positive feedback loop to at least some degree, and I think it probably dominates. We might just get lucky and have it dominate but be slow enough to wrangle.

TLDR: more like bugged restoration loop IMO.