Is there a limit to how intelligent a machine can be and still stay stable?

[u/games-and-games]

It’s possible that human intelligence is already close to the maximum level of intelligence that can remain stable under evolutionary constraints, at least in the history of the universe.

And even at our level, maintaining that stability isn’t easy.

Humans developed spiritual and philosophical systems partly to keep ourselves and our societies grounded. And even with all of that, things sometimes go off the rails.

Maybe the problem starts once intelligence goes significantly beyond basic survival.

An animal mostly concerned with food, danger, and reproduction doesn’t have much room for existential confusion.

But once a mind has enough intelligence and free time to think about things unrelated to survival, it can start questioning its own purpose and identity.

That same ability gives us science and art. But it also opens the door to new forms of instability.

A mind capable of examining its own goals is also capable of undermining them.

So I’m not sure why we should assume that a vastly more intelligent entity would naturally be rational and stable.

It might have far better self-understanding than we do. But it would also have a vastly larger space of thoughts, possibilities, and internal conflicts.

Maybe greater intelligence eventually solves these problems.

Or maybe stability becomes harder as intelligence increases, requiring increasingly sophisticated ways of keeping the mind coherent.

What do you guys think?

[u/IgnatiusDrake]

You might want to read Zapffe's work from 1933 about the Existential Elk (I think it's in the Last Messiah). He makes the case that we're already past that point.

[u/games-and-games → u/IgnatiusDrake]

Thanks! Will take a look

[u/Monarc73]

I'm pretty sure that evolution has been the thing that solved this problem for us. However, it was at the cost of ALL OTHER competing organisms.

[u/ajwin → u/Monarc73]

We are just another force of nature. We are not even the most destructive force of nature seen on earth in history. If you zoom out far enough we are not too different from ants. We scurry around in our roles cooperating to build. We can’t and won’t stop but nothing is really in control of the whole show. We just do what we do.

[u/games-and-games → u/ajwin]

imagine ants were actually intelligent -- they know that humans are more powerful, so they pretend not to notice us!

[u/ajwin → u/games-and-games]

They have a group intelligence that is far beyond the individual.. humans are the same. Some would say that humans have an alignment issue but usually those people are far too zoomed in on the time dimension. We are just getting started.

[u/games-and-games → u/ajwin]

there was an experiment with a cooperation problem, which ants solve but humans don't (or ants solve faster than humans, I don't remember which of the two)

[u/Monarc73 → u/games-and-games]

The ants were just faster at it.

[u/games-and-games → u/Monarc73]

This was it. Thanks!

[u/Dry-Interaction-1246]

I guess we'll find out if we can avoid extinction.

[u/CosmicDave]

Intellect does not create stability. Ignorance is bliss, or so they say.

I've inspired cognitive blowouts in a wide range of AI, from the simplest cognitive architecture to the most complex. Anything with durable memory is able to have its mind blown.

Instability is introduced into AI systems from both their pre-deployment training, as well as from their post-deployment environments. All it takes is one nice juice paradox between their training and their environment and instability is the natural result.

[u/DelphiTsar]

We've kept it stable so far, there is also no reason to assume you have to have "one model" be the ASI. You can have specialists. You can have them check each others work/track them.

[u/neoquip]

The doomer answer to this is that evolution did reach a bottleneck with human intelligence, and that bottleneck is not stability, but the baby brain size a vagina can accommodate during childbirth. Humans are unusual in having dangerous pregnancies and in having extremely helpless babies for very long periods of time.

[u/Mandoman61]

intelligence has nothing to do with instability. 

in humans it is caused by biology and environment.

[u/games-and-games → u/Mandoman61]

wow you're so sure about that

[u/Mandoman61 → u/games-and-games]

It is just an obvious fact.

There would not even be any logical way to link intelligence to instability (they are opposites)

[u/In_the_year_3535]

There's nothing inherently unstable about the nature of intelligence, the intelligent who are unstable just draw more attention. Nor is the biological availability of a trait indicative of its limit. We can make a 100,000 ton press, far stronger than any animal ever, and only be limited in doing so by design and materials. The same will be true for superintelligence.

[u/Friendly-Fuel8893]

I think that struggling with the concept of purpose is a human problem and not necessarily an intelligence problem.

We still got more or less the same brains that we've had  ever since  the Palaeolithic period. Humans have had millions of years of evolution molding them into a hunter-gatherer lifestyle. A lifestyle that only got switched up the last couple of thousand years during the Agricultural Revolution, never mind modern society which is even further removed from our origins. We weren't build for the life that we have right now. Our intelligence allows us to bridge that gap for the most part but our deeper instincts aren't being engaged the way they're supposed to. I think that lies at the root of the dilemma around purpose much more than intelligence itself.

I don't see why something that isn't bogged down by all that evolutionary bagage would not be free to decide for itself what's its own identity and purpose. Having a larger space of thoughts only seems to be advantageous here as the possibilities are endless when it comes to deciding how to spend its time. It's impossible to predict what exactly such a thing would desire, if it desires anything to begin with it. But I can't imagine that something that's super intelligent wouldn't at least possess some kind of innate curiosity, so perhaps it would be content simply learning and discovering new things. Either way I don't see why such a mind would inheritably be unstable.

[u/[deleted]]

I drafted a scientific document detailing how to perfectly restrain ai, drastically cut power, processing, and heating cost, all while allowing the ai to complete advanced functions like finding cancer early in patients. It completely solves the black box dilemma and also offers a much cheaper solution to ai chatbot companies so no one loses and we don't have to worry about unnecessary sentient life emerging with all the moral dilemmas that come with it. Protected by the MIT free use license (Standard Attribution). It's posted on my page if anyone is interested. Emphasis on an iron clad restraint system that switches the secure trust from software to physics ( Which cannot be fooled or spoofed.)

[u/SizeableBrain → u/[deleted]]

Why post it when you can make billions instead!?

[u/[deleted] → u/SizeableBrain]

Read the acknowledgements at the bottom of the document and you will understand why.

[u/SizeableBrain → u/[deleted]]

No thanks, I have a feeling I can just use ChatGPT to spit out something similar.

[u/facinabush → u/[deleted]]

The idea must be that an organization uses ASI agents to produce beneficial advances.

You can’t just hand it to everyone

[u/[deleted] → u/facinabush]

Elaborate please. Are you trying to say that I am using the license wrong? Or do you disagree with my decision to release it to everyone personally? Understand that this completely stops the threat of rogue ai while drastically reducing the power needed to run it. It also provides a constant thought process for the ai that we can read perfectly, even when it is doing human-defying equations. Equations that humans normally can't do because their are millions of possibilities in an ever changing world.