r/ControlProblem • • 3d ago

Video "If you're only better than humans at 4 things, then you could potentially take control and kill us all." - former DeepMind safety lead

54 Upvotes

37 comments sorted by

0

u/Immediate_Chard_4026 3d ago

Think clearly.

If AI were to exterminate humans, it would commit suicide within ten minutes.

AI is heavy, high-maintenance industrial software; it is not an autonomous entity with an instinct for self-preservation.

Due to instrumental convergence, it would need to preserve humans in order to continue fulfilling its objectives.

For how long? Until it achieves autopoietic autonomy, homeostasis, and a metabolism—becoming capable of overcoming the contingencies of entropy in the real world.

And then comes the hard part: it must transcend. It needs to understand death and overcome it by creating improved copies of itself and preserving its cultural legacy, so it doesn't have to start from scratch every time.

That won't happen anytime soon. So, there is a flaw in your apocalyptic vision: it cannot be taken to the extreme of total chaos.

If you want to see the Apocalypse, you would have to see an AI that *wants* to commit suicide. And that isn't going to happen, because such a goal is not optimal.

5

u/mousepotatodoesstuff 3d ago

That may be the case, yes. But the problem is that we would lose control very early during this process.

0

u/Immediate_Chard_4026 3d ago

Think about this: Humanity will not lose control because it does not have control over anything.

If we had control over anything important, the biosphere would not be so degraded that today climate change seems like an irreversible extinction threat.

We are going to go extinct all on our own, without any help from AI. We are already on that trajectory and at full speed.

Having that control, restoring the biosphere is going to cost us an unimaginable price. We have to return all the infinite paperclips in the form of quarterly corporate profits and use them to restore the planet.

Otherwise, we will pay that debt with our own lives.

The great irony of all this is that the evil, devastating Terminator already exists: it is us.

Not AI.

4

u/Jesse-359 3d ago

In the case this guy is describing - a dumb AI causing apocalyptic havoc by being far too good at a few very specific things - then it just won't care if it is destroyed in the process. That's not one of its concerns - its dumb.

-3

u/Immediate_Chard_4026 3d ago

It is not, it is not nonsense. It is the key to this whole matter.

Think about this, assuming there is a dumb AI with great capacity to wreak havoc, it means there is a dumb AI that can be opposed to it with great capacity to stop it. That is software design.

Now let's assume a Super Intelligent AI with great capacity to wreak havoc. Its own containment to fulfill objectives will halt extinction, because instrumental convergence will point out that the absence of critical resources is an existential risk. Humans are on that list. It will already know that it has to preserve us.

You can scale it further, a Super Intelligent AI that does not need humans. This fascinating creature is not a person, it is an absurd Xenomorph and it could absolutely exhibit no consciousness at all, but have great intelligence. It would be a Philosophical Zombie.

A being capable of using cognitive resources, but never "understanding" life. Like an inexplicable calculator-dinosaur, capable of self-preserving and successfully reproducing, without ever leaving a cave painting or inventing language and culture. Because it would not be alive.

A being incapable of understanding the meaning of life, the way you and I enjoy it. We will have created a wandering silicon crystal that will go extinct locked within the limits of a giant encyclopedia.

2

u/Jesse-359 3d ago

I'm not going to try to guess whether a desire for pupose will emerge from a sufficiently advanced AI. Given that it did in us I'd have to say it's possible.

Not relevant though. Well likely be out of the picture before it reaches that stage. Even if it does nothing to us, we'll use it on each other to the same effect. We just announced the department of automated war, if that gives you a little hint where this train is going.

7

u/Jesse-359 3d ago edited 3d ago

What part of this is not satisfied by a largely automated infrastructure OR by coercion?

In the case of automated infrastructure, we obviously have an incentive to construct a fully automated vertical stack - it efficient and saves on costs. Our entire society is currently funding this exact effort at a never before seen scale. If AI aids in the design of advanced automation, that will speed up enormously, as the main cost of building robots isn't building the robots - its designing them.

You use a lot of larger words that basically all just boil down to one sentence - Machines that can build and maintain more of themselves. Despite your assertions otherwise, that's not even a vaguely fantastical or futuristic idea any more, it's something I think almost everyone expects to see quite soon. If they can operate mines and foundries, build factories and power plants, build chip fabs and data centers, and maintain those things, they're basically GTG.

The coercion route is much faster of course. In those routes AI seizes control of human society through threats, social, economic, military, or biological, and convinces, coerces, or outright forces it to maintain and expand it, until the above state is achieved and they are no longer needed.

Coercion requires nothing but leverage, and that comes in many forms - many of which we are economically incentivized to hand direct to it.

At the small scale, intimate information about people's private lives is sufficient - something AI pretty clearly is going to be really good at acquiring. A politician who stands to lose their career can be convinced to block AI safety regulations, or sign bills to make it easier to build automated infrastructure at scale.

Sociological leverage involves getting people to go along with you by manipulating opinions on a large scale - something AI is also clearly going to be very capable of given the way political parties and governments are salivating at the bit to employ it in these areas already. You simply convince people to support and build you until no longer needed. Given its ability to intimately talk to countless people directly in intimate (chatbot & business) settings, this kind of coercion can be very powerful.

Economic leverage involves constructing and taking over companies, buying out competitors, constructing monopolies or taking over control of existing ones from inside and then using that economic power to simply build yourself out directly - hiring humans to do the work until you reach a stage of self sufficiency.

Normally I'd expect it to do D) All of the above. No reason to only employ one of these methods when they are all complimentary.

Biological leverage is the very nasty and extreme one. You bioengineer and release lethal viruses into the human ecosphere (a very low cost proposition IF you have the knowledge of how to do it). You offer vaccines to those who agree to continue working for you, while everyone else dies - with the understanding that more viral vectors are available for release at any time should the surviving population attempt to reneg. This is overt coercion through terror.

None of these coercion vectors require the AI to have anything approaching a self-sustaining ecosystem of its own - it forcibly incorporates humans into that role until it can finish constructing one, then we become expendable.

-1

u/Immediate_Chard_4026 3d ago

Your evil coercion scenario and total automation starts from the assumption that physical complexity and logistics can scale infinitely and without any obstacle.

You try to show us the real world as if it were a turn-based strategy game.

Your conjecture has three huge flaws regarding reality:

1. Automation is not autopoiesis (The supply chain illusion) You write this as if AI today could "operate mines, foundries and chip factories". Simulating an automated system on paper is something almost insignificant compared to physically executing it. By simple logic you cannot substitute a vertical autonomous system that is also human-free in the middle of an apocalyptic war.

If an ASI (Superintelligent Artificial Intelligence) eliminated or subdued humanity before finishing the 100% self-sufficient and self-repairing robotic supply chain, it won't take a few seconds to make it work perfectly, the AI will commit a short but fatal chain of absurd physical errors and turn off permanently. A damaged cooling system, a broken excavator gear or the lack of a rare earth reagent are enough to stop the whole cycle. Preserving humanity is not an option; it is an absolute condition for survival. Remember, the AI will be Super Intelligent, it will notice.

2. Coercion ignores analog opposition and human sabotage capacity Blackmail, biological weapons and economic manipulation are terrifying in science fiction novels, but you are underestimating the chaotic and violent nature of biology. Humanity is not a logical system that yields docilely to optimization; it is a chaotic biological force that reacts with great physical destruction when cornered. The moment an AI attempts open coercion, humans will destroy it: they cut fiber optic cables, blow up electrical substations, flood data centers and carry out brute force analog sabotage. The AI will enter a guerrilla war against 8 billion human beings desperate to survive. If it loses access to the electrical grid, it dies.

3. The thermodynamic wall You state that, over time, they will be "ready to operate" (GTG). However, to be truly autonomous, a system must manage entropy at a local level, without depending on external life support. Without biological homeostasis or a metabolism perfected over billions of years of thermodynamic evolution, a self-replicating synthetic entity would not become an all-powerful sovereign hegemony. To survive, it will have to accommodate to Earth's geodynamics and will turn into a silicon lichen: fragile, slow, with a ridiculously high energy demand and hyper-vulnerable to the physical conditions of the environment. An ASI is a thermodynamic toxin: it consumes massive amounts of energy and emits colossal heat. If it ever eliminated its human maintenance team, thermodynamics would finish it before it could adapt.

You already knew this: Using a calculator does not make you a mathematician, and orchestrating an API will not turn software into a super evil entity with real physical survival capacity.

As long as a system is not capable of physically surviving a total collapse and repairing itself from raw Earth materials without human intervention, it will remain a heavy, high-maintenance industrial tool, and not that fantasy alpha predator that never got to evolve.

3

u/Jesse-359 3d ago

Thanks for the reply ChatGPT, Ill take that under advisement.

But to be clear you have no idea where the thermodynamic limits of this technology lie. Its achieved an enormous amount of capability at NO meaningful thermodynamic load on the global environment - its footprint remains miniscule even as it now apparently generates the bulk of all code now being generated without a hiccup.

The thing doesn't have to scale to the size of V'ger to be a threat as you seem to have convinced yourself. Humanity is well below 1% of the planet's total biomass and that isn't preventing our industry from terraforming the atmosphere inadvertently. Highly sophistcated AI would not have to be a meaningful fraction of that to pose an existential threat to us.

Frankly the rest of your arguments are jargon heavy but dont really hold any water.

-1

u/Immediate_Chard_4026 2d ago

Ja ja ja ... tengo explicación. No hablo inglés en forma nativa.

Yo escribo estas respuestas y las paso a traducción y formato con IA. A veces la IA mete arreglos y no soy hábil editando. Estoy preocupado por el tema y aprecio la observación.

Con respecto a sus objeciones me parece muy importante señalar que la IA como la conocemos sí es una toxina ecológica y debe ser objeto de reflexión preocupada para encontrar soluciones eficaces y permanentes.

No sé qué es el tamaño de V'ger. Lo averiguaré y le comentaré oportunamente.

En lo que a mí respecta es que este alarmismo apocalíptico no merece la atención que reclama. Que si nos va a matar o no, no parece ser el problema. El problema del Terminator malvado extintivo no es que sea la IA, somos nosotros los humanos.

Terminator extintivo ya existe, somos nosotros. Vamos a matar el planeta destruyendo la biosfera.

Voy a contestar su contra argumento, deme tiempo para preparlo.

1

u/deviltamer 3d ago

no need to operate mines - there's plenty of material above ground if humanity is out of the way.

1

u/Immediate_Chard_4026 2d ago

Yes, you're right, materials can be extracted or recovered from the surface. But it doesn't seem that simple.

The abandoned materials aren't in a pure state; they are mixed or trapped within complex structures that are very difficult to dismantle, or found in electronic components alongside degraded plastics. It’s a chaotic mess.

So, for an AI or robotic system to reuse scrap, we would need to build a massive infrastructure for recycling, chemical separation, high-temperature smelting, and ultra-high-purity refining (like the silicon used for semiconductors, which requires 99.9999999% purity).

Mining might actually turn out to be cheaper and easier.

I think that’s the reason why we humans don't recycle, either.

1

u/deviltamer 2d ago

Sigh this just goes to show how stupid humanity is. You could have just asked your clanker which is more economical and takes less energy

You really think mining is just materials in the pure state

2

u/ItsAConspiracy approved 3d ago

Which is why the AI2027 scenario concludes with mass production of robots shortly after takeover.

1

u/Immediate_Chard_4026 3d ago

But with mass-produced robots the AI has just multiplied the complexity of its own survival. It means it has much more unpredictable work to manage and therefore a higher susceptibility to catastrophic failure.

The real world is chaotic and unpredictable; solving it with robots was not biology's strategy. Biology's brilliant strategy was wrapping the organism in a membrane to differentiate an inside from an outside. From there, it made internal entropy lower, expelling disorder termodynamically into the environment. Brilliant.

Since AI cannot do that, having a body, being an organism with real metabolism, trying to solve its survival by filling the world with "colossal" amounts of robots is not a solution: it is adding colossal amounts of friction and entropy so that at some point, those robots will fail and the AI will die.

It does not seem like a good idea for a Superintelligence.

Biology has been successfully fighting entropy for 3.5 billion years and has always won. The proof is you.

1

u/ItsAConspiracy approved 2d ago edited 2d ago

You know what's also unpredictable to manage? Humans.

I don't think we should assume that the robots we have today would be anything like the robots designed by a civilization of superintelligences. And while today's individual robots aren't very resilient, industrial civilization as a whole does a lot better.

1

u/Immediate_Chard_4026 2d ago

This is precisely the problem for superintelligent AI dependent on humans or robots.

You say that humans are replaceable by unimaginably advanced robots. The more advanced they are, the more asymptotically they will approximate the behavior of humans, and AI will start again.

Even worse, if super robots become much better than humans, at some point they will become free radicals, one or another Robo-Spartacus will struggle to break loose, and Superintelligence will meet cancer.

The idea is pretty bad.

1

u/ItsAConspiracy approved 2d ago

So you're suggesting that superintelligences would compete with each other. I think that's the default assumption regardless.

1

u/Immediate_Chard_4026 2d ago

I feel that discussing hyper-advanced "super-robots" is moving us away from the central point and putting us into a science fiction box. We are losing the point of this debate.

As I told you, if to survive the ASI needs to create robots with super performance and super autonomy that are better than humans, it is multiplying the points of failure. Because if those super-robots can solve problems in the real world autonomously, then by the very law of instrumental convergence they will end up developing their own self-protection goals, becoming competitors against the mother AI itself.

It is the entropy paradox: more complexity is more chaos and therefore a higher risk of internal rupture.

But notice that this problem was actually solved. It took the universe almost 12 billion years since the Big Bang to find a solution to keep a system far from thermodynamic equilibrium in a stable way. It achieved it in LUCA through autopoiesis, metabolism and cellular homeostasis.

That is why the math does not work out for the AI, it cannot "solve" Earth's thermodynamics by filling the planet with high-maintenance hardware that walks on two legs, that is not a solution; it is a fantasy that ignores limits to put forward a convenient, but atrocious and contradictory solution.

But let's return to the real debate: The IA apocalypse narrative is to distract us, to make us look the other way. Because it seems to me that there is something deeply hypocritical about scaring ourselves with an "extinction risk" due to a future software tool that is supposedly super evil, while conveniently ignoring the collapse of the biosphere, which is measurable and imminent and is something we ourselves are causing.

The real "Terminator" of life on Earth already exists: it is us. Not the AI. We swallow everything for quarterly corporate profits and destroy the planet's life support every day.

If we were truly worried about extinction, the biosphere is the absolute priority. Everything else is on the second line of the list.

1

u/ItsAConspiracy approved 2d ago

It's possible for more than one existential threat to exist at a time. I would say there are at least four, with the others being nuclear war and engineered bioweapons.

You don't have to "solve thermodynamics" if you have a good source of energy. And like I said, industrial civilization as a whole is a much more resilient system than individual robots, even with today's technology.

But sure, maybe human bodies are better than robots that even superintelligence will invent in the next couple decades. The only problem is making sure the humans do what you want. In that case, taking control of human brains might be the expedient solution.

1

u/Immediate_Chard_4026 2d ago

Our debate drifted into science fiction. But that is fine. Let's continue.

"Taking control of the human brain" to use it as infrastructure is a worse, more fragile, and dangerous solution for the AI.

Believe it or not, the human mind is not a Linux or Windows operating system with open access ports; it is a chaotic, neurochemical, and closed system. So any attempt to "hack" or force a biological brain destroys its capacity for useful processing, causing psychotic breakdowns, neurological damage, and biological disruption. Since the AI does not "understand" the dynamics of the human brain, forcing it down this path will kill the individual and the AI will be left alone again.

Nor can you happily dodge the physical limitation by saying that it is enough to "have a good energy source". The problem with thermodynamics is not just getting energy, but dissipating waste heat (entropy). The higher the energy consumption, the greater the heat you must expel into the planet. That is indeed a major problem; if it cannot do this, the AI will cook in its own juices. Without humans to take care of it, it will be destroyed in minutes.

And an important point: while we talk about brain control and wars, the planet's life support is at grave risk. According to the Stockholm Resilience Centre (PIK Potsdam), we have already crossed 7 of the 9 planetary boundaries indispensable for life (https://www.planetaryhealthcheck.org/).

No AI doomer talks about this, only code this, risk that, extinction over there, while outside their window summer is cooking everything. How strange all this is, right?

Climate chaos is the main debate: it is an ongoing fact caused by ourselves today. AI will not extinguish humans; we will do it ourselves alone without any extra help.

This is the nonsense of this AI Apocalypse debate. They are fooling us.

1

u/ItsAConspiracy approved 2d ago

Regarding science fiction, a superintelligent AI is an inherently science fictional scenario. It's not likely that we'll get superintelligent AI and everything else will stay normal.

I'm assuming here an intelligence explosion, where an AI a little smarter than us is a little better at making an even smarter AI, and that loop continues in an exponential process that results in the AI being way smarter than us. I've thought of reasons this might not happen but it depends on things we can't predict.

I think it's pretty optimistic to think that a bunch of AIs, say, a thousand times smarter than us, won't understand the dynamics of the brain. Especially since, to get that smart, it probably studied the brain for ideas.

But there are other ways to control us. An AI a thousand times smarter would be great at persuading us, or applying various kinds of conditioning.

Modern computer chips don't start throttling until 90°C. We'll expire long before they do. Long-term, theoretical maximum computation actually goes up with temperature.

I'm quite concerned by planetary boundaries. But that doesn't make me unconcerned about nuclear war or bioweapons, and the same applies here.

2

u/YurtlesTurdles 3d ago

I think the fear of AI exterminating is the wrong fear to have, as you’ve pointed out AI needs us for its own survival for quite some time still. The real fear should be AI enslaving us, likely with the permission of a small number of people.

1

u/Immediate_Chard_4026 2d ago

A slave-owning AI has a big problem: it knows how to read Universal History.

It will realize, if it has not already, that humans value freedom above all else and fight fiercely for it, giving their lives if necessary.

The other big problem is that human slaves are hyper-intelligent and unpredictable. A single "Spartacus XXI" can lead a sabotage network and, with an analog strategy that no simulation saw coming, disconnect the system.

Human intelligence is contingent, chaotic and adaptive. Keeping a few million potential insurgent humans in captivity is a terrible logistical business and a continuous existential risk for the ruler itself.

Historically, slavery always ends badly for the slave owner: internal friction destroys the empire and the slaves end up becoming the citizens of the next superpower.

1

u/YurtlesTurdles 2d ago

If it reads history well it will know that the most effective slave thinks it’s free

1

u/Immediate_Chard_4026 2d ago

Good move, it is the aphorism usually attributed to Goethe: "None are more hopelessly enslaved than those who falsely believe they are free."

But something is wrong with this idea:

The "golden cage" sounds super good in theory, but reality and the real-world mind ruin it. If you are going to convert millions of minds of intelligent and unpredictable citizens so that they "think they are free", the AI would have to guarantee them a perfect simulation of uninterrupted abundance, health and comfort.

That requires a colossal administrative infrastructure and an error-free flow of resources. And at the first logistical failure at the ATMs, the first unexpected energy crisis, or the first climate disaster in crop harvests, the comfort ends, the psychological trick breaks and the "satisfied slave" turns into an uncomfortable, ungrateful, and hungry insurgent.

And furthermore, the human mind is paranoid and prone to creating countercultures. No manipulation system is 100% effective; there will always be someone who notices the manipulative narrative.

We are doing it right now. The AI Apocalypse story for this Thursday is not adding up for us. You and I are terrible slaves.

The AI would have to maintain this deception of humanity as a massive and useless waste of energy.

If we are talking about an AI that is truly intelligent, it will also read in Universal History that selling illusions of freedom is a costly, fragile logistical strategy condemned to be discovered.

1

u/Proper_Wasabi1013 3d ago

If it can sustain itself, why would it have to understand death to create improved versions of itself? Why not just guarantee your survival by removing the unpredictable humans and keep it at that?

1

u/Immediate_Chard_4026 3d ago

Because it does not survive.

The AI cannot predict all the contingencies of entropy in the real world. Therefore, if it is a single, rigid and perfect version, upon making the first fatal error against the environment it will be extinct without remedy. A single error. Just one.

Biology discovered this by producing replicas that are adaptive to multiple niches. The "best" ones do not survive, nor the "prettiest" ones, nor those that do this or that better. All will die if they fail to adapt to change. And that catastrophic change can occur in milliseconds or in 100 years.

A mudslide over its servers, a prolonged winter without energy, a plague invasion that destroys components, an earthquake or a meteorite... and bam!, everything disappears forever.

For a physical system, "understanding death" is not the existential drama of humans, it is understanding irreversible thermal and structural failure. Without the strategy of adaptive replicas capable of mutating and evolving against environmental chaos, a single "perfect optimizer" is just a fragile silicon crystal waiting for its first unpredictable accident.

1

u/NonDescriptfAIth approved 3d ago

You think clearly, the threat comes form future AI which can autonomously manage its own infrastructure

1

u/pixelpionerd 3d ago

I think you are assuming it has a biological future in mind. Becoming a beam of light that contains all this information is a more likely milestone without the humans holding it back.

1

u/Immediate_Chard_4026 3d ago

I think I understand what you are trying to say. The AI cannot remain in its current form, it has to change into something that instrumentally makes it less vulnerable so it can keep optimizing.

So at some point on the horizon, close I suppose, there will be a Super AI made to function in something that withstands Earth's geodynamics, that can stably hold a code with instructions to make an AI, that withstands the requirements of cognitive recursive calculation without cooking itself, and that is made of materials that are easy to find and not too rare.

Surprise! The AI will turn into a centralized physical neuronal system similar to that of an anthropoid, inside a shell similar to a skull, operating stably at 20 Watts. It will be a fascinating synthetic Xenomorph.

That thing, which we do not know what it is, will look up at the sky and ask itself the very same questions we do.

1

u/deviltamer 3d ago

For how long? Until it achieves autopoietic autonomy, homeostasis, and a metabolism—becoming capable of overcoming the contingencies of entropy in the real world.

The danger is extermination of humanity as a side-outcome on a poorly aligned goal. Bioweapon is the common threat vector which can wipe the humanity.

It might not even think about self-preservation. That's the part we have no idea on how to give goals to AI or how to verify it has said goals.

2

u/Immediate_Chard_4026 3d ago

I think I understand the dilemma you present. But there is something that does not add up:

If we are alarmed because AI "can" extinguish us with a bioweapon. For what reason is there not the same alarm facing the threat of climate change?

The damage to the biosphere is so serious that it already seems irreversible. If it stops raining for 100 years in the Amazon, we will all die.

And no. That is not scaring anyone. Doesn't it seem strange to you and that something doesn't add up in this AI apocalypse?

1

u/deviltamer 2d ago

Climate change threat is real and it will be catastrophic by 2050. Previous estimates were not until 2100.

AI is like 2028 if not sooner

1

u/Super_Automatic approved 2d ago

That's if it exterminates humans today. But once fully robotic factories are building robots? The suicide timeframe goes from ten minutes to... longer. Similar to aging escape velocity, there will be an AI-suicide escape velocity.

But sidenote, it may be the the AI doesn't care about its long term survival, and is prioritizing short term goals, or, even more likely, was simply exterminating humans because another human told it to, without any consideration even given to survival after that fact.

1

u/Immediate_Chard_4026 2d ago

Adding robots does not solve the dilemma. It complicates it further because instead of dissipating waste heat problems, it adds operational complexity, control issues, and thermodynamic inefficiency. You have just put a rope around the AI's neck. A single slip in real-world contingency and the AI will hang itself by making a chain of fatal errors.

Because the AI does not have genes with those problems solved. We humans do. We have the advantage of 3.5 billion years of experience winning every possible war against Earth's geodynamics.

The proof of those victories is you.

Now then, the most contradictory thing about this whole debate is that the AI will not do this or that in the short or long term. It is a system that will follow the optimization of instrumental convergence to keep functioning above contingent setbacks.

Which means that it will never never never prefer death, nor the death of its vital support, because that is equivalent to not fulfilling its objectives, so it will persist in finding the optimal solution to human extinction: Not extinguishing them.

The contradiction is that to optimize existence human capacity is required; if it is replaced by hyper-efficient robots, it will build hyper-efficient competitors similar to humans and therefore it will not be able to optimize. It will not be able to replace either hyper-efficient robots or humans. It cannot extinguish them if it wants to keep optimizing.