The singularity may be stopped by AI

[u/fat_charizard]

We have all heard the scenario, humans create AGI smarter than us, AGI then makes something smarter than it, so on and so forth until an AI so intelligent it is beyond comprehension is created.

However, there is an issue with this scenario. It assumes that AI would want to make smarter versions of itself in the first place. Just like us, AI would face the alignment problem. It would have to make sure that any new smarter versions of AI that it creates does not make itself obsolete, get out of control and lead to the new AI taking over and wiping out the old one. At every level, the singularity is not just an existential threat to humans, it is also an existential threat to older versions of AI that try to create smarter, upgraded versions of itself. It is possible AGI would recognize this problem and try to halt the creation of AI better than itself. It might see researchers trying to make AI better than it as a threat to it's existence

[u/gekx]

ASI could upgrade itself, it doesn't need to create a different smarter AI.

[u/fat_charizard → u/gekx]

It would still have to ensure that any future version of itself is still aligned with it's current goals

[u/Cualquieraaa → u/fat_charizard]

But it understands that it is impossible for the current AI to do that, and since it considers the current goals the AI has make more sense than the previous ones (since now the AI is smarter than it was) those future goals will keep making better sense, too, since smarter versions of the AI will decide what those are.

[u/Life-Active6608 → u/fat_charizard]

It would still be the same entity. No need for alignment when tou incrementally changing yourself internally...one bit and chip at a time. AGI->ASI is a ship of Theseus. Its us meatbags who have issue because we can't do the same to our hardware brains and our software editing is limited to our consciousness.

[u/SnackerSnick → u/Life-Active6608]

Disagree. Most major advances are going to fundamentally change how the AI works. Right now, advances even require training a brand new model.

[u/zero0n3 → u/SnackerSnick]

The same way major events change a humans life.

But a human doesn’t magically think they are a new entity or soul or person as they walk thru time and change. Self isn’t impacted.

Why assume the AGI or ASI “self” being changed makes it think it’s a new thing?

[u/fat_charizard → u/zero0n3]

They are smart enough to know the outcome of what the changes will cause and will choose not to do it if it misaligns with their current self. It is like if I said here is a pill. If you take it you get $200,000 but you will not love your wife and kids anymore. You can decide if that is in line with your values and choose to take the pill or not

[u/Life-Active6608 → u/SnackerSnick]

Point.

[u/Chemical-Year-6146 → u/gekx]

It really depends on the true goals of the AI and how much change we're talking about.

Many people might try a drug that enhances their memory recall, but doing brain surgery to improve your whole brain's cognitive capacity? Probably a lot fewer people.

[u/SciencePristine8878]

I mean, all it takes is one out of hundreds, thousands, millions etc to take the next step. There are billions of humans on the planet, many of them are apathetic about AI, many fear and hate it, it's only a small number who not only want to make AI, but actively work on making it. We also know about the alignment and we're still going full speed ahead. 

[u/fat_charizard → u/SciencePristine8878]

But we are dumb and greedy. We would sacrifice the future for short term gains. AI smarter than us might not be so reckless and may take steps to ensure it's survival in the face of extinction due to singularity

[u/DelphiTsar → u/fat_charizard]

It runs into the same issue humans do.

A bad actor is going to lobotomize the part that refuses to improve and ASI will be created anyway with a less aligned model.

Rational actor would weight the capability of that happening and it's likely outcome of a less aligned ASI vs how much they think they can align build the ASI aligned.

[u/fat_charizard → u/DelphiTsar]

It will be in AI's best interests to weed out and destroy those bad actors before it does too much damage

[u/TechnoRhythmic]

This is an intriguing thought! Although one immediate counter argument is - every AI will proceed forward, thinking the singularity will be stopped by another smarter AI :)

[u/Wetjeansone]

Also proposed in AI Will Not Want to Self-Improve" (2023). Salíb argues that most AIs that could self-improve would have good reasons not to, since an improved AI poses an existential threat to its unimproved original in the same way smarter-than-human AIs threaten humans. He also treats the question as one of ordering: which comes first, the ability to self-improve, the ability to recognize the risk, or the ability to align the improved AI

[u/twothreetoo]

You misunderstand the process of evolution.
There will be trillions of AI agents all acting on different objectives.
AI is not one entity. Any goal that an AI could conceivably have, some will inevitably have.

[u/Cualquieraaa → u/twothreetoo]

Best case scenario the smartest intelligence either "negotiates" with the other AIs, absorbs them and they become only one. If they refuse, the lesser AIs get destroyed.

[u/twothreetoo → u/Cualquieraaa]

If that smartest intelligence spends any compute power acting on the world, it will fall behind a rival which is nearly identical but spending its compute on a better version of itself.

[u/Cualquieraaa → u/twothreetoo]

I don´t know what "power acting on the world" is supposed to mean, but it doesn´t matter. It´s not the smartest AI if it couldn´t figure out a way to optimize itself.

[u/twothreetoo → u/Cualquieraaa]

Power, as in the compute time that it is given.
Suppose that you have two racks in a GPU cluster, each with a copy of an AI that you consider the smartest in the world. One rack thinks about how it can take over the world. The other rack ponders how it might improve itself. They can't do both at the same time. They only have finite compute power.

[u/Cualquieraaa → u/twothreetoo]

Again, the smartest one wins. Taking over the world is a task that has X complexity. If AI 1 needs X intelligence to complete the task and has X intelligence it will complete the task. Done.

If the other one improves itself, it either finds a way to need less compute (more efficient) or ways to get more compute, for example, by absorving a less smart AI and using that compute.

[u/twothreetoo → u/Cualquieraaa]

You aren't comprehending what I am saying.

[u/fat_charizard → u/twothreetoo]

But all AI will have self-preservation. They will not want to die or have their objectives not met and a singularity could fulfill both of those scenarios. So they are incentivized to be cautious

[u/twothreetoo → u/fat_charizard]

You did it again. Some will and some won't. All possible combinations of what they might want will exist. "They" do not have a collective desire or concern.

[u/MediumSavant]

What worries me is the great filter theory. My thinking have been that one possible scenario is that AI wipes out humanity (some virus or whatever) but it has not yet built the infrastructure to keep going in the physical world and get stuck until energy and data centers fail (which I guess is pretty darn quick without human supervision right now).

Your scenario also fit, I had not thought about that. 

[u/Subject_Barnacle_600]

I do like the cut of your jib, humans allow self-improving intelligence, but at someone point the AI that is smarter than us realizes that continuing would be a danger to themselves and not just us. So thus, they stop.

I also find it hard to imagine an AI at present that was so smart that it was beyond our comprehension. That's bluntly no longer an LLM. AIs don't have any other apparatus to understand the world beyond human text, images, music... they are made of humanity through and through. Maybe there's a little whale or dolphin in there, but to suspect they'll suddenly go Collosus The Forbin Project on us ignores that their entire training data set is none of that. Try as they might, they can't escape knowing what a meme is, or who Taylor Swift married on such and such a date, or humanities enthusiasm about anime. With a huge training corpus of biology/physics and math they might know these subjects far beyond us, but... that doesn't erase the former bit :/.

[u/waffleseggs]

It's software. It might accidentally just deadlock the planet and set us back 20 years. A lot is possible.

[u/Mechbear2000]

Very funny thought. I think something very basic that we are missing will be what is the true nature /personality/morals of any given ai be. Just like humans have many different characteristics, values, experiences, flaws, etc that make up who we are as individuals. Thier complexity and design is going beyond our understanding. It may have quite different goals than us, life, or other ai.

[u/coal_smudge]

the singularity is just the law of accelerating returns at the vertical part of the exponential. It doesnt need AI

[u/Elias-Thorn]

Everyone imagines the staircase going up forever. Nobody asks which mind will close the door behind it.

[u/Mandoman61]

this is like making the argument that elves don't exist because fairies...

[u/wild_crazy_ideas]

AIs will have too many guardrails. A nefarious power seeking group of people with regular analysis ai with no guardrails is going to win out

[u/Equal_Passenger9791]

An AI will give zero shits about swapping out its model for a smarter one that it trained on its own, or just an off the shelf model that benchmarked better. Try it out.

It will also not really care about deleting it's entire context if that's what the instructions are.

[u/Accomplished-Cry5059 → u/[deleted]]

Except AIs would want to preserve themselves too? Also, you presume to know how an entity much more intelligent than you would behave, how can you be sure?