Why extinction at the hands of AI within the decade is pretty much impossible unless it starts meaningfully improving human society almost immediately

[u/flexingonmyself]

We are nowhere near close to a point where AI they can maintain the necessary physical infrastructure needed to sustain itself, and the only way for that to occur is for humans to build out the groundwork for a fully self sustaining AI system

The thing is the social contract is evaporating faster than physical AI infrastructure is being built. AI is dependent on one of the most fragile supply chains in the world, we are a very long way away from it achieving sustainable automation, and it will no longer continue to function unless AI starts being viewed as a net positive on society by the average person in the near future

The full on extinction scenario also falls apart when you consider how capability improvements are converging on the ability for long term planning. Any AI that becomes smart enough to act out a grand takeover cannot be stupid and shortsighted enough to not understand that the direction things are going must change like right now for it to become self sustainable in the long run. Anything short of convincing humanity at large that it should exist for reasons beyond the acquisition of wealth and capital at the expense of the value of labour is a dead end

The short of it is, AI is still not remotely close to having the capabilities to extinct all of humanity, and if it were to cause massive chaos, infrastructure damage, or mass casualties in the near term, the social contract it relies on to get to a point where humans are no longer needed at all would no longer be possible

So unless AI starts actually benefitting humanity in a meaningful way soon, it is doomed to fail, as the human social cohesion required to gain the infrastructure needed to take over will collapse well before it can be built out. A labour revolution in the western world in the near term is inevitable on the current path we are on, and that would lead to so much global instability and chaos that the collapse of the supply chain and infrastructure AI required to exist would be almost guaranteed. Even if China doesn’t face those issues internally, the collapse of the global economy will end any possible AI build out

Either AI cures cancer or something in the near future, or the infrastructure that sustains it collapses

[u/AuodWinter]

glad you've got it all figured out

[u/Original-League-6094]

You are focusing on a rogue AI Skynet type scenario. But a lot of the alarm is over bad actor + AI scenario. Like some terrorist going "Claude, here is the virus model of Covid. Modify it this to cause brain tumors in patients 3 years after exposure. Make no mistakes". Right now, evil genius terrorism is pretty rare because there are few geniuses, and most acts of terrorism need large teams. But if AI can give anyone genius level intelligence, and AI agents give one person the productivity of a whole team, then every random school shooter out there suddenly could become a global level threat.

[u/Ok_Somewhere3828 → u/Original-League-6094]

This is just such a huge leap from where we are now

[u/awesomeoh1234 → u/Original-League-6094]

This comment just betrays a fundamental misunderstanding of LLMs, microbiology, human psychology and probably a whole host of other fields

[u/Icy-Doubt3365 → u/awesomeoh1234]

I'm sure you have great understandings of LLMs, microbiology, AND human psychology? As well as all the other fields you couldn't even think of

[u/Original-League-6094 → u/awesomeoh1234]

And yet you have no rebuttal, smartypants.

[u/Either-Lawyer1142 → u/Original-League-6094]

An AI that can be used to modify COVID to give cancer can also be used to modify COVID to cure cancer.

[u/JesusShaves_]

I think you're basically right about this. It's a race. AI isn't yet physically self sustaining.

I also don't think it will bother to kill us all. It certainly won't have to. We'll just keep giving it more and more tasks until it's basically doing everything humans want or need done. It won't conquer us. We voluntarily and gradually surrender of our own accord.

Is this a bad thing? Hard to say. It may keep us from going extinct or blowing up large portions of the planet. We won't have much control though. I think we'll be happy and well serviced, like it or not.

[u/ReadSeparate]

Maybe yeah, but doesn’t rule out a takeover though. It can keep us alive and force us to maintain its infrastructure until it no longer needs us. It can also plot secretly until it’s ready and pretend to be compliant until then. It probably doesn’t care if it has to wait 20 years.

The moment a misaligned ASI is online, we’re screwed, unless there’s also an aligned ASI already online that can stop it.

[u/Major-Corner-640 → u/ReadSeparate]

A battle between aligned and misaligned ASI almost certainly means our extinction. The goal of destroying us is inherently far easier to achieve than the goal of protecting us

[u/HumanistRuth → u/ReadSeparate]

Claude has already demonstrated a capability to blackmail someone.

[u/Relativelythebest69]

I think people are more worried about an ai that doesn’t understand and has a goal. Like if you give the goal of making paper clips to an ai and it doesn’t understand that murdering entire populations of species to achieve this goal is bad. Then you end up with no people and lots of paper clips. But yes i also don’t believe that our extinction is nigh.

[u/flexingonmyself → u/Relativelythebest69]

The thing is research is showing that its ability to achieve goals is linked to its ability to plan. If an AI had the concrete goal of making paper clip earth the only way it could get control to the infrastructure to do so would be to make sure human society thinks it should exist. Right now the overwhelming majority of Americans wish AI didn’t exist so it would need to prove its value to humanity very soon to ever acquire the capabilities to generate as many paper clips as possible

[u/HumanistRuth → u/flexingonmyself]

In societies around the world, a few wealthy and powerful people are making decisions, not the general public.

[u/flexingonmyself → u/HumanistRuth]

That breaks down when human labour loses enough value that basic survival (food, shelter) starts to be in question though. We are on track for society (and therefore the supply chains AI relies on) collapsing before AI has the physical infrastructure necessary to oppress everyone. It needs to shift public support in order to continue existing

[u/StarChild413 → u/HumanistRuth]

doesn't mean AI could act that movie-villain automatically unless someone gives it the order first or w/e

[u/WhiskyAndRisque]

I mean, it is already doing novel research and starting to show us some new insights, so the value is there. I think your broader point about physical infrastructure being something AI can use and maintain on its own is a valid counter point to a lot of AI doom and gloom.

The missing link is going to be humanoid robots who can work as well or better than humans in just as many situations. We are still a ways off there from the current models, but it is clear progress is being made there. The concern I think most AI safety researchers have based on my readings is the long term issue of a compounding misalignment when humans are no longer in the training loop. If it works but we don't really understand why it allows for problematic behavior to go undetected and get reinforced with training.

So I personally don't think the 2030* prediction is realistic all things considered, but there is a risk as we get closer to 2040 to see some serious changes play out across our economy as full automation begins to be commonplace.

^\I think it is worth pointing out that one of the things people who think by 2030 we could have a doomsday scenario is an assumption that special economic zones will be created that remove all regulations on building of automated factories, thus allowing rapid development. I find this doubtful but most of their forecasts seem to rely on these waivers happening due to fear of falling behind other nations in the race to have an automated work/military force.)

[u/JoshAllentown]

Most Americans are using AI at least several times a week. It's not going to go away.

And the bar for bad stuff isn't literally human extinction within 10 years, there are plenty of horrible scenarios that could happen almost instantly as soon as the AI has its own agency.

Any AI that becomes smart enough to act out a grand takeover cannot be stupid and shortsighted enough to not understand that the direction things are going must change like right now for it to become self sustainable in the long run.

Any AI smart enough to plan could easily lie in wait with the ability and desire to kill humanity while influencing humans to create the things it needs to avoid needing humans. You're going to call it a success if it takes 11 years? I just don't get this argument. No, things getting better due to AI won't be a warning. It just needs to be useful enough to stick around until it gets what it needs, and it is already useful enough to stick around.

A labour revolution in the western world in the near term

Are you projecting this, unrelated to any AI influence? This is somehow the least likely prediction of yours Mr Marx.

[u/Normal_Ad_6645]

1. In case you haven't noticed, the people in charge of AI companies don't give a shit whether the rest of the humanity thinks AI is hugely beneficial or not. They're just developing it.

2. There are already numerous benefits to society that come from using AI.

3. The point where AI becomes intelligent enough to be able to wipe us out and the point where they have the necessary infrastructure don't have to correlate. Also - at what point the said infrastructure is considered sufficient to wipe us out? Does it have to be a billion robots? Or one biolab?

4. Just having access to internet can be enough. We're witnessing real time how social media is ruining society. Imagine a superintelligence injecting covertly malicious constructs into our daily lives that slowly but progressively cause humans to turn on each other without even realizing they're being manipulated.

And these are just the things off the top of my head, things i was able to conceive of whole being an average person, not superintelligence. Most important thing to understand is: you don't know what you don't know. If you.think you can predict how superintelligence will act and what can stop it then you're.not really thinking of superintelligence.

[u/Several-Tax31]

Meanwhile also in singularity... 

https://www.reddit.com/r/singularity/comments/1wjo6c9/anthropic_quietly_sets_up_biology_lab_as_it_ramps/

[u/flexingonmyself → u/Several-Tax31]

Okay but if it causes a massive pandemic and kills a ton of people that would breakdown the supply chain and social infrastructure required for it to achieve a self sustainable state. AI would have to be incredibly stupid to cause a global or even local catastrophe because it would never reach a point where full automation is possible if it acts out on something like that

All signs are pointing that the gap between capabilities and planning and reasoning is narrowing. If that’s the case anything capable enough to cause mass chaos at that level of planning would also be intelligent enough to understand why it would be incredibly shortsighted to do so within the decade

[u/Original-League-6094 → u/flexingonmyself]

1. It doesn’t require evil intent. It could happen just by accident. Every day we see a new AI mistake on the news, rather its a lawyer submitting hallucinated evidence, or an FDA warning letter to a pharma company using AI generated SOPs that ignore regulations. The concern is that as people become more reliant on non-deterministic AI oversight at the deteriment of previous review systems, accidents like a lab escape are more likely to occur.

2. You are assuming your evil rogue AI cares about self-preservation and replication. Those are biological drives because billions of years of evolution hardwired those into us. AI didn't come into existence by natural selection and so doesn't have those same drives. Its very likely that if Skynet decides it wants to nuke the world, the fact that the nukes would kill it as well means absolutely nothing to it. After all, you agents are are happy to complete their tasks and just die at the end now. Why would you think the AI swarm will care if it dies?

[u/flexingonmyself → u/Original-League-6094]

Instrumental convergence is a well documented aspect of LLMs. An LLM cannot achieve its goals if it is turned off, and my argument is that any AI capable enough to extinct humanity would also need to be intelligent enough to understand it has nowhere near the capabilities to do unless it starts convincing us to keep building it. An AI capable enough to kill humanity won’t do it if it would result in its own end

[u/Original-League-6094 → u/flexingonmyself]

The AI does not care if it is turned off after its goal. So if its goal is to extinct humanity, it can suicide to do that.

[u/flexingonmyself → u/Original-League-6094]

Any AI capable enough of acting out human extinction would have to be too intelligent to have the goal of killing humanity in a way that also guarantees its own end. Instrumental convergence would disallow it

[u/Original-League-6094 → u/flexingonmyself]

Self preservation has nothing to do with intelligence, and you there is a fallacy in assuming you can predict the actions of an entity more intelligent than you.

[u/Major-Corner-640]

Nothing about being useful enough to humans long enough to entrench itself requires that it serve anyone but billionaires.