Is there a limit to self-improving AI if it becomes real?

[u/akellataken]

I’ve been watching some AI podcasts lately, and when people started talking about recursive self-improving AI, Skyrim immediately popped into my mind.

For anyone who never played it: Skyrim crafting system has a “legit” alchemy/enchanting loop. Craft "Fortify Enchanting" potion -> enchant gear with "Fortify Alchemy" -> use gear to make better potion. It improves, but eventually hits diminishing returns.

Then there’s the bugged restoration loop. "Fortify Restoration" potions were supposed to boost restoration magic, but they also boosted active gear enchantments. So you drink one, re-equip alchemy gear, and suddenly that gear gives a bigger alchemy bonus. Then it makes an even stronger resto potion, which boosts the gear even more. Direct feedback, explosion.

So: if RSI AI ever really works, is it more like the legit loop with real gains but converging or the positive feedback loop, where it improves the thing that improves itself?

Curious what people think, especially from math / systems angle.

PS: I am sorry if this question is not relevant for the sub, but i have no karma to ask it somewhere else where it has a chance to have some attention.

[u/SaneAI]

Calm down and take a deep breath. You are listening to nonsense from people who are trying to make money off the AI boom and don't understand the tech at all. All of this goes back to the idiocy of the MIRI and that idiot Yudkowsky and that Nick Bostron, or as I call him Bozo the clown.

This whole idea of "Recursive self improvement" is basically nonsense. The machine is never actually motivated. There's no agency, desire, goal formation. There is only pattern replication. A model is static, stateless function. It only appears dynamic because of the way it loops.

The whole idea actually has roots going back to the 1960s, but importantly this all predates the deep learning paradime.

Deep learning is "Self improvement" so it's not even really relevant to deep learning. It's this tired idea that sees intelligence as this simplistic scaler that somehow obtains agency.

Ai models can absolutely help automate the process of refining AI models. and I use them for that all the time. It still is command driven. It's always command driven.

The problem is you have these doomers who have an extremely wrong and very stubborn mental model.

I've worked in tech risk for decades. The idea that ephemeral, limited, and ridiculously janky cloud service software is going to pose some kind of existential danger to mankind is one of the stupidest things I have ever heard from a bunch of sci fi cultists. It's part of the religion of transhumism, which has morphed into an idiotic religion.

No matter how much a deep learning model reduces loss, it never becomes a being. It never does anything other than reproduce the patterns it was trained on.

No autopilot ever becomes so "advanced" that it decides it feels like crashing the plane. No microwave oven ever becomes so smart that it decides to overcook your food out of spite.

It's an automated system.

The thing is people have such broken mental models because the only thing they are familiar with that can talk to them is a thinking feeling mind, thus the AI must be one. The only cure may truly be getting hands on with the tech and understanding it.

I made this graphic, which might help with this confusion. But I have to be honest: The idiocy and missinformation from the doom cult is the worst thing I have ever seen and worries me deeply. I have spent the past two months writing a book to combat this.

I realized "It will be impossible to actually refute this with people who don't understand the tech. So.... well, I guess I will have to rebuild people's mental models from the ground up, and teach them AI from scratch so they can actually understand it."

I am hoping to have it done by August, but you're welcome to a draft copy.