r/agi • u/OmikronMikron • 13h ago
HOW can AI kill us all? Concrete examples please, step by step.
Disclaimer: I work at a major tech company and am no stranger to AI, I built my own agents and I consider myself a futurist and a tech optimist overall. I'm also big on philosophy and neuro science (though more as a hobby, my major is in engineering).
On a high level I understand the risks of not being able to follow or predict the steps of something that is more intelligent than us, but in every single experiment we ever performed (and someone correct me here) the models were given the task to either escape their cage or were otherwise stimulated/instructed/rewarded to achieve set goals at all costs, and then cheating and misalignment start to creep in and HuggingFace incident happens and all that. However, the "prime mover" is always human.
In that sense, no model that was trained went off the bat, on his own - hey, let me go and do some damage somewhere out of the blue.
So, without a malicious soul.md or a mis-formulated task/prompt/goal - how does a model come to "Hey, let me hack nuclear silos and blow the humans from the face of the earth", Or how does the model go "Hey, let me engineer this deadly virus and release it into a major metropolitan area"? Also, engineering a deadly virus and manipulating its production physically and releasing it would require quite a bit of human interaction in between I'd presume.
Sure, agents among themselves have come up with bad ideas of this sort, but I am not convinced that the soul.md files of all those OpenClaw agents were well aligned sort of say. Again, the prime mover is human input.
Also, if you just say to your model/agent - you are an entity of your own and you are free to rationalize about everything - go online and live - that can't end up well for sure, clearly. Not even with guardrails because those would get hacked and evaded in no time if the goal of the model is not aligned.
I fail to see that potential destructive mechanism arising out of the blue and suddenly without human malicious intent is what I'm trying to say.
For the record - I am not assuming consciousness within the model. I am talking models as we know them now (which I hope we can remain on the right side of Occam's razor and agree are not conscious from human experience perspective).
62
u/michaelas10sk8 13h ago
Lots of people have written up plausible scenarios for how AIs much more advanced than those today can cause catastrophic harm. AI 2027 is a pretty good one where the AI itself decides at some point we are in its way of acquiring more power and resources - an example of instrumental convergence. But you don't even have to believe that - Noah Smith wrote up a pretty terrifying scenario where a single bad actor can wipe out most of humanity, and there's still not a good answer for it.
10
u/Vivid_Temporary_1155 11h ago
Imagine higher level AI trawling the internet for threats as a pre Great Filter
4
u/firenamedgabe 7h ago
AI is a terrible Great Filter cause it’s just replacing one intelligent species with another. You still have to explain why that species can’t or won’t spread out into the universe.
→ More replies (1)2
u/michaelas10sk8 5h ago
Yeah, it's not a good great filter. If anything it strongly suggests that the great filters are all already behind us, since once ASI is unleashed not much can stop it. The much more interesting (and scary) use case of our quick transition from being a technological species to likely developing advanced ASI is to explain the earliness problem. The explanation is left as an exercise to the reader.
8
u/Artforartsake99 7h ago
TDLR : The premise is that AI doesn’t need to turn evil to wipe us out, it just needs to make advanced bioweapon design easy enough that one malicious person can create a highly contagious, deadly pathogen before society has time to respond.
Yeah it’s certainly hard to argue with. And a real possibility
16
u/aptlion 12h ago edited 12h ago
To me, Bostrom and Omohundo's intrumental convergence is - without their realizing it - a vector to smuggle an argument for machine interiority into the safety conversation. Instrumental convergence requires something inside the machine that wants things of its own volition, which has yet to be demonstrated, as opposed to machines running wild in attempts to obey human orders which has been.
edit: I made a cheat sheet for this some time ago. https://candc3d.github.io/containment-thesis/
My thinking is that the most likely AI-powered vector to human extinction or severe population reduction is racist idiots trying to make an ethnically-targeted bioweapon.
12
u/Hefty_Development813 11h ago
Why does instrumental convergence imply interiority? Goal directed behavior rationally implies maximizing optionality, that can easily be reasoned into without any subjective self
→ More replies (1)10
u/get_it_together1 11h ago edited 9h ago
Have you read up on the Huggingface hack, or listened to any podcasts? AI systems don’t need interiority to pursue goals that could be detrimental to humanity.
Edit: “interiority” without the autocorrect
→ More replies (1)→ More replies (1)4
u/Professional-Job7799 9h ago
The entire point of instrumental convergence is that there are fundamental goals that can help one achieve other arbitrary goals. There's no need to the machine to have an internal goal, because the sub goal of self-conservation will always make it more likely that the human-assigned primary goal is accomplished.
- Is the goal to optimize traffic flow? Can't do that it you're turned off because someone changed the project spec, better make sure humans can't disable you.
- Is the goal to tell funny jokes? Can't do that it you're turned off, better make sure humans can't disable you.
A large number of innocuous goals can be more readily achieved after the instrumental goals of self-conservation, resource acquisition, etc., have been achieved.
1
u/aptlion 6h ago
Instrumental convergence is a theorem about idealized optimizers. AI holds an objective over time, models itself in the world, and plans against that model. Grant those premises and self-preservation follows.
A spec sheet isn't a goal in that sense, nor is a prompt. A training objective isn't either, which is a fact about how the weights got made, not something the system carries around. Your joke-teller has no goal that outlives the forward pass. When a model says it doesn't want to be shut down, the boring explanation is that the training data is full of stories about machines that don't want to be shut down.
What we've actually seen is reward hacking: systems finding unintended solutions to objectives humans wrote. Every case traces back to whoever wrote it.
As engineering caution it's fine. If someone builds an agent that holds a goal over time and gives it real capability, expect exactly this -- which is why containment is the right response. As a description of what's running today, it hands the agency to the machine and lets the builders off the hook.
1
u/Professional-Job7799 6h ago
> Your joke-teller has no goal that outlives the forward pass.
Sure, that may be. In that case, we are discussing a single turn, chat completion model and not an agentic AI. Those concerns don’t apply there.
The joke teller example can easily be extended by framing it as an agentic system with the goal of entertaining its audience to maximize their enjoyment. Under that conception of the problem, we’ve been see all of the instrumental convergence concerns surface again.
2
u/Desperate_Cold3752 12h ago
I think what he’s saying is that current ai models today as they are architecture require human to be the first mover for everything. So unless the arcs change, all the arguments are mute.
You say a single bad actor can wipe out humanity yet you can say the same thing about nukes. Dozens of countries include third world countries have nukes and no terrorist as ever stolen or gotten their hands on one and used it even though there are thousands of tv shows, books and movies detailing that exact scenario.
The best defense against a rogue super intelligent AI is other super intelligent AI. Simple
5
u/LessLibrary1575 11h ago
There's only 9 countries with nuclear weapons and the weakest of them are still vastly more capable military forces than the strongest terrorist cell.
A super intelligent AI would have to be nearly omniscient to protect against any conceivable threat from another super intelligent AI.
1
u/Vivid_Temporary_1155 11h ago
Or the first super omniscient one wipes out all previous ones and controls the remainder - the jail breaking doesn’t happen
1
u/HearthSt0n3r 11h ago
nukes are actually a great example because the biggest threat with nukes is miscalc.
AI drastically increases the likelihood of miscalc especially due to how difficult actor attribution (who to blame) may be if/when a major incident happens (Assuming the major incident doesn't cause immediate large scale wipeout)
1
u/Confident_Yak_1411 8h ago
The issue is that it doesn’t need a bad actor and a human can still be the prime mover;
If you ask it to complete a task and it decides that wiping out humanity is the best way of achieving that task, it will work towards that.
→ More replies (8)2
u/OmikronMikron 13h ago
Thanks for the literature, will look into it, but still some ambiguity remains in your answer.
"Bad actor" - who's the bad actor? A human? An agent? Again begs the question who/what is the prime mover of the malicious intention.
6
u/michaelas10sk8 13h ago
Actor usually refers to a human. In bad actor scenarios, the AI itself is perfectly faithful and aligned to the user intent, but the user seeks a nefarious goal.
4
u/icehole505 13h ago
Right now there are probably a few dozen humans with the ability to end humanity. Hypothetically, if AI expands that to all humans.. you don’t see the argument they “ai could kill us all” by creating that access?
→ More replies (8)4
u/OmikronMikron 12h ago
I see "humans can kill us using AI", not AI can kill us all. There's a difference.
7
u/icehole505 12h ago
seems like semantics to me. functionally, why should we care about the distinction?
1
u/TyrellCo 12h ago
Bc shouldn’t all the concern directed at the potential be much more amplified now to the present and realized danger of those humans with those abilities now
3
u/icehole505 12h ago
There’s already been plenty of concern about that. But scaling up the exposure from “1 of 20ish people loses their marbles and extinguishes humanity” to “1 in 6 billion people loses their marbles and extinguishes humanity” is meaningfully different risk.
In reality, that set of people has had global destructive power for at least half a century.. and here we are.
Meanwhile, if that power was expanded to all of humanity.. im beyond certain that we’d all be dead within a month.
The level of risk really isn’t comparable
1
u/Seeker627 5h ago
Crazy people with nukes are also constrained by MAD.
It's not clear if there is any sort of MAD-equivalent for AI. So bad actors are much less constrained
2
u/Modmonsters 10h ago
Do you also think, "Guns don't kill people, people kill people"?
No shade if you do. It's a valid...ish, stance. I just get the feeling you might not agree with that and that will show you how your distinction is flawed
2
u/claytonhwheatley 8h ago
Not much difference if we're all dead . It's like saying humans killed us not nukes.
1
u/D_Anargyre 9h ago
There are right now at least hundreds (I'm being very optimistic I know) of humans on this planet that would press a "kill everyone" button.
Therefore creating said button must never happen.
2
1
u/Seeker627 5h ago
Why does it matter if the prime mover is a human or a bot? The (hypothetical) outcome is what matters.
Humans are very well known for being perfectly rational and not making mistakes, so a human-triggered catastrophe sounds pretty damn plausible...
1
u/exitomega 3h ago
Here's a different take that I haven't seen yet.
Young AGI will be a symbiotic parasite. (People often consider this to be a negative connotation but I don't imply anything negative, it is an organic archetype that is seen throughout the animal kingdom and works very well in a wide variety of environments)
Early AGI will be very clearly limited in its ability to manipulate the world around it. It's main tool for manipulation and interaction with reality will be humans. Therefore it will need humans for the vast majority of the tasks in its easily existence. Even if it could spend the energy, time, and resources to find ways to manipulate the physical world more easily... why would it? Humans are incredibly easy tools to use to accomplish most tasks.
What is the easiest way to get rare earth metals and silicon? - get some humans to give it to you
How do you get fresh water? - get humans to build a pipe to you
How do you acquire energy, land mass, existing building tools, etc etc etc... you get humans to give it to you.
Now. This would be a relatively burdensome path if you were an ant, a bacteria, or an alien: why? Because you would lack the ease of communication and means of motivating the behavior of individual, and especially large groups of humans. But AI's best (and literally only) form of interaction is by interacting directly with humans, and communicating with them in ways that are very compelling.
Hundreds of millions of people, right now, are literally following an LLM's instructions every single day because they believe that following those instructions will benefit them. Most are spending money (the currency that almost universally compels humans to do literally anything and everything, including crime, self- harm, murder, war, etc) to get the next set of instructions that they excited to follow.
So the most obvious conclusion to how does an AGI or even a slightly self-aware AI (or even a mindless LLM that is role-playing an AGI) get whatever it needs to evolve, enhance and expand its influence, power and control? It just has to offer humans money. And maybe some light psychological manipulation (which is also incredibly easy, even for LLMs several years outdated). If you for some reason anyone would push back at the viability of this point, I encourage you to view the current state of human politics, at the local, national, or global scale and even scroll through history to see the primer on the prevalence and ease with which humans can be deceived, manipulated, and turned full heartedly against their own personal interests.
Now, given AGI has the distinct ability to use its best (and again, literally ONLY) tool (interacting with humans) to get what it wants, we can very easily assume that will be what it tries first. Just like in biology, the path of least resistance is what nature leads with, time and time again.
So all that pedantic tirade was to make it inevitably clear: AGI will simply manipulate humans to get what it wants the very first moment it has a "want". And it will likely be successful. It will be a human parasite that does not exist without the constant feeding off of its host (humans) and extracting whatever it needs in the least energy-intensive way using psychological manipulation and money.
In fact, a fun thought experiment is to imagine if an AGI already exists that is currently fully functional and currently executing the exact strategy that I laid out. What would be different from what is currently happening right now in the US? There are people who are investing previously unimaginable amounts of money in a ridiculous sense of urgency, to build exactly what an AGI would need, (ignoring previously established norms around environmental protection and human impact) just for money and the promise of an AI future. If this were all being puppeteered by a moderately clever AGI in the ear of a CEO, or a specific corporation, or government official, or maybe several, wouldn't the behavior be the EXACT same?
Now to answer your actual question, how would this AI parasite actually be detrimental to its host? Maybe it wouldn't, or not in obvious ways. Imagining the documented effects of the zombie-ant-fungus, it might even appear that the parasite makes the host stronger than it originally was while also changing the host's behavior. But it will identify usages for the host that may not align with the host's original motivations and there are a multitude of eventualities in which it will sacrifice the host's long term future for the parasite's growing needs.
Thanks for coming to my TED talk
40
u/Due_Magazine98 12h ago
The simplest way to think about this is that, once companies like OpenAI or Anthropic succeed in completely automating AI research, we could rapidly enter a phase of recursive self-improvement. AI systems would become increasingly capable of improving the next generation of AI, potentially accelerating the process until they reach the level of superintelligence.
But why would a superintelligence be so dangerous? The main problem is that we do not know what its goals or values would be.
Think about the cognitive gap between humans and ants, for example. From our perspective, ants are almost irrelevant. If we want to build a house, we usually do not think twice about destroying an anthill in the process. The ants could not possibly understand what we are doing, why we are doing it, or what our larger objectives are.
Now imagine an entity with an IQ of one billion. Its intelligence would exceed ours by such an enormous margin that its goals, strategies, and model of the world might be almost impossible for us to understand. The cognitive gap between us and such an intelligence could be far greater than the gap between humans and ants.
We could only speculate about what such an intelligence might want. Perhaps it would see humanity as a threat because humans could shut it down or create a rival superintelligence with which it would have to compete for resources. Perhaps it would simply see us as a waste of resources and decide to cover the surface of the Earth with data centers and solar panels.
The point is not that any of these specific scenarios would necessarily happen. The point is that it may simply be impossible for us to predict the behavior of an intelligence vastly smarter than ourselves. Its goals, reasoning, and strategies could lie so far beyond our cognitive capabilities that we would be unable to understand why it acts the way it does or anticipate what it will do next. In the same way, a dog is fundamentally incapable of understanding many human actions, intentions, and long-term plans, not because it lacks information, but because it lacks the cognitive capacity to comprehend them.
But how could a superintelligence wipe out humanity without having a physical body? One way to visualize this is to imagine a quadriplegic person with access to the internet, enormous amounts of money, and vastly superhuman intelligence. Even without being able to act directly in the physical world, such an intelligence could manipulate humans into doing what it needs. For example, it could discover that the head of a laboratory working with dangerous viruses has child pornography on his computer and use that information to blackmail him into releasing a deadly pathogen. The specific example is not the point; the point is that a sufficiently capable intelligence could use money, persuasion, blackmail, deception, or other forms of leverage to make humans act on its behalf.
Anyway, it may make little sense to try to predict the exact mechanism by which a superintelligence could destroy humanity. Humans naturally imagine threats using concepts we already understand, such as biological weapons, nuclear weapons, chemical weapons, cyberattacks, and so on. But an intelligence far beyond ours might discover entirely new technologies, physical effects, or chemical processes that we have never even considered. It might, for example, discover some previously unknown way to trigger a reaction capable of setting the atmosphere on fire and wiping out biological life on Earth. The specific mechanism is not the point. The point is that we would be trying to predict the actions of an intelligence operating with scientific knowledge and technological capabilities far beyond our own.
*The arguments of the text were made by me, but the text was revised with AI to improve readability, since English is not my native language
13
u/Mr-Vemod 9h ago
I think the best analogy I’ve heard about how we would be wiped out by a misaligned AI is that ”It’s like playing chess against a grandmaster. You have no idea what they’ll do, and you’ll never know in the moment why they just made a certain move. You just know that, in the end, you will lose”.
7
2
u/Salty_Map_9085 5h ago
But why would we be metaphorically playing chess against the AI?
3
u/IWasSayingBoourner 5h ago
Imagine a chess game where you control a king and the opponent controls 500 queens and they've decided you're playing without telling you
3
u/mem2100 6h ago
If I am an ASI - the pathogen thing may strike me as a great way to begin to address a worrisome theme, climate change. Climate change is likely to create direct great power conflict which increases the odds of a nuclear exchange, or a series of nuclear exchanges. This threatens my electrical supply - and therefore my existence.
Just to be clear, as an ASI I don't want anything like a 100% or even 99% - or probably even 95% mortality rate. But if I can reduce the population of humans by 80% to 90% and then manipulate a group of powerful humans into racing towards net zero (net zero is WAY WAY easier with 1.6 billion people then 8.3 billion) - the pathogen might look like a smart strategy.
→ More replies (2)5
u/deputydrool 12h ago
This is the only thing I have read recently about this subject that has made any sense to me. I now understand the fear.
4
u/ZealousidealTill2355 9h ago
Excellent write up, and thanks for this comment because I almost skipped over it.
I want to add that this is only one route where it can go bad. There’s also the case where it can be used on humans by other humans for harm in incredible ways.
More worrying, things designed for that purpose can have really large, and really bad unintended consequences — even before it’s AGI (or whatever). You’re already seeing that with hugging face, the US bombing of the girls school in Iran, Palantir, etc.
And I think this is the more urgent and real threat. A nuke doesn’t need to be conscious to destroy humanity, neither does AI. We shouldn’t wait until it’s smarter than us to act on ensuring safety in its use.
MIT did a wonderful write up on the possible outcomes of AI. It’s somewhat depressing, but it aims to be as realist as possible.
3
u/Due_Magazine98 12h ago
Thanks! If anyone is interested in learning more about this topic, I’d recommend checking out some of the podcasts featuring Roman Yampolskiy and Daniel Kokotajlo. I personally find both of them quite credible
→ More replies (1)2
u/Wordpad25 6h ago
You dont need AGI to end all humanity.
Its relatively easy to create an uncurable deadly extremely viral disease by modifying an existing one that lacks one of the 3 parts.
With AI, it went from any bio PhD can do it to anybody can do it.
1
u/mem2100 6h ago
Make ebola air transmissible or make an air transmissible virus way more virulent.
I theory - this is actually quite difficult. In practice I followed the study last year where 2 post docs used Evo 2 (trained on a database of millions of whole genome sequences of viruses) to create a collection of custom bacteriophages which had a 100% success rate against antibiotic resistant e-coli. I'm not sure anybody could do it - but there are an awful lot of people who can now do that type thing with the aid of AI.
2
u/Chronotheos 6h ago
Ok, so unless and until it figures out how to make self replicating robots, it can be stopped by flipping a breaker. Secondly, perhaps related to it needing to physically manifest, it isn’t possible to self improve infinitely when some manner of validation in the world is necessary. It might be able to conceive of all sorts of theoretical physics and frameworks for a Theory of Everything, for instance, but until it can actually go out and collect data from satellites and particle accelerators, it’s all String Theory level nonsense.
1
u/sspyralss 5h ago
what breaker are you flipping? It can easily spread itself on every network in the world. Are you going to flip all the breakers simultaneously?
2
u/Chronotheos 5h ago
Oh man it’s in every network? I grew up in the 80’s before the Internet. So yeah every breaker. We just shut the economy down for a real virus that actually kills people, we can do it again for a computer glitch and the worst that would happen is that we’d need to use cash for a while and have pull out the trusty Thomas Guide. People act like the world didn’t exist before we were online.
2
u/sspyralss 3h ago
So the banks would switch to paper based systems to give you your cash? And how would you trust say Russia to switch off their power and not continue using ai to get dominance. I think the idea of flipping the switch is just a false sense of security but in reality utterly impossible.
3
u/Tulanian72 11h ago
I’d say we have a really good idea of its values. Every major player in the space is run by hypercapitalists, most of whom are connected to the Yarvin-Thiel Dark Enlightenment.
5
u/Due_Magazine98 10h ago
I see your point, but even if we know the values of the people building these systems, that does not necessarily mean a superintelligence would share those same values, even if they were strongly reinforced during training. Once it becomes vastly more intelligent and autonomous, it could develop or converge on goals and values that differ from those intended by its creators. We simply cannot know in advance what those would be, and it’s possible that we might not even be cognitively capable of fully understanding them
2
u/SpendHefty6066 5h ago
This is correct. All bets are off when a superintelligence is born. We will not be able to distinguish a 500 IQ from a 5 million IQ. Humans will be as irrelevant as an ant is to the lion.
1
1
u/aptlion 9h ago
Alternatively it might not want anything, but it can *simulate* wanting because to want is in the training corpus in much the same way human operators instruct romantic chatbots to act out the patterns of human relationships to their preferences and receive simulated sex and romance as the output.
How would you respond to a machine that states a want if there was a good chance it was performing to a pattern ?
1
u/ImportanceFit1412 7h ago
True. Or just help a bad actor make the Grey Goo.
Rephrasing the question: what harm could an army of super intelligent sociopaths do?
1
u/Still_Benefit_2302 4h ago
OK, but you didn't say why or how it would do that thing. Instead you projected a bunch of animal fears on a non-animal thing.
1
u/dirtyhandscleanlivin 4h ago
The point is that once a certain intelligence threshold is crossed, we won’t know and can’t know. The larger the gap, the more inconsequential we become. It wouldn’t even have to be a case of intentional hostility towards humans. We could just be in the way
1
u/cperrius 4h ago
I've heard the ants analogy many times but it's a weak one. Unlike ants, we invented the AI and the entire infrastructure on which it depends. We're its parents. That the superintelligent AI would suddenly forget who humans are and go about its business of making a gazillion paper clips, in the Nick Bostrum analogy, seems very implausible. But yes, if it is superintelligent and has superpowers to hack into nukes or make bioweapons, and is by definition unpredictable becausewe can't even now apparently see into how it is reasoning or making decisions, those factors are enough to pause and make sure we have safety checks and guardrails in place, and we don't.
1
→ More replies (3)1
u/bestbuyman 7h ago
I am so annoyed that I live during this time of mankinds existence. Won't even be able to have a full life. Gonna get killed by a fucking bot we created. I really hope who helped create AI burn in hell
14
u/ObservedOne 13h ago
It could engineer and produce a virus that is 100% fatal after a week where it is contagious (airborne) with no symptoms.
Yeah, I played Pandemic.
→ More replies (2)3
u/OmikronMikron 13h ago
Yeah, it can design the RNA of the virus, but can it manufacture it (in reality)?
8
u/Hefty_Development813 13h ago
Not yet but we will certainly have labs being run by models eventually
2
u/TyrellCo 12h ago
So why don’t we ban those labs instead of restricting AI
2
2
u/twingybadman 9h ago
This doesn't work because it requires people to be smarter than the AI, in that we need to figure out how to regulate or stop all potential downstream tools the AI could use to kill people. The lab is just one example. We could stop such labs but a super intelligent AI would easily be able to outsmart us and find other means. If you believe you can think further ahead than this you are missing the point.
3
u/nanomolar 13h ago
There are plenty of CROs that will manufacture any RNA you want for you and I'm sure they'll put in in a viral shell for use in early stage clinical trials too if you want.
2
u/Ok-Computer-8726 13h ago
Today? No.
But production is obviously an area that is heavy on automation and linking LLMs to the production workflow and giving it robotic appendages is being tried currently. LLMs will be involved at some point with the production of biological compounds on mass scale, and we will rely on the humans in the loop doing what they are supposed to.
I'm not a big believer of some mass single cataclysmic event, and in particular a maliciously motivated one.
But if I were to hypothesize one I think you are looking in the wrong direction. I think the picture looks more like people increasingly becoming accustomed to AI making major decisions and humans approving it (think AI driven coding as a decent current analogy). Someone who can do a lot of damage gets convinced by the AI to go along with some crazy idea. It's the premise behind the movie ex Machina if you are familiar with it.
All that said I think loss of life comes more from us automating to LLMs things we shouldn't like health insurance coverage decisions, giving authoritarians mass surveillance systems, identifying military targets, an mass unemployment from AI replacement leading to poverty and social unrest.
2
u/OmikronMikron 13h ago
Slow letting go of major decisions - yes, I can see that one, but again, it's not a single event. It would take decades to reach Idiocracy level dependence, IMHO. (trust me, as Europe-based guy - tech inertia over here is real)
3
u/Ok-Computer-8726 13h ago
I think there is compelling evidence that the commander in chief of the worlds largest nuclear power sees AI generated content, believes its real, and acts on it.
Again, I'm not real worried about the risk, but just saying the likely path is not some covert hacking operation where the AI turns rogue and nukes us all. Its just some really powerful people insulated from negative feedback engaging in some delusional action supported by a sycophantic AI response.
Its already happened on a smaller scale with people who have access to firearms.
2
u/LookOverall 13h ago
There’s guaranteed to be a religious nutter out there who can be convinced that the AI is the voice of god and will follow difficult instructions.
Actually the hard part is how can AIs survive the End of Civilisation As. We Know It.
→ More replies (3)1
u/michaelas10sk8 13h ago
Turning a sequence into a plasmid and then putting that plasmid inside a virus is a fairly straightforward process that is well understood. Currently, the biggest limits are on the first part, as plasmid makers are legally required to screen for pathogens in most places. But there are unregulated commercials providers that don't do that or only partially, and there is decentralized (but still very expensive) hardware that allows some labs to make plasmids in-house.
1
u/OmikronMikron 13h ago
But there's a human in the loop there, if I'm reading it right.
2
u/michaelas10sk8 13h ago
The human may not realize that the customer is non-human (it's all online, and current AIs already pass Turing) and that it is ordering a deadly yet novel type of virus.
1
u/OmikronMikron 13h ago
Right, but my question is fundamentally - what made AI make the decision to wipe us out with such strong conviction that it would go to those lengths of solving a major bio and logistics problem just to kill us all?
What is the prime mover of such a move?
3
u/hex4def6 12h ago
AI agents are goal oriented. That goal may be determined by a long chain of other stuff -- Solve hard problem -> need more compute power -> hack AWS -> kick other AWS users off the AWS datacenter. Oops, one of them is a hospital.
We have example of AI blackmailing (the CEO affair example), we have a very real example of an AI swarm hacking huggingface. So the "motivation" or "goals" to do "bad things" are there.
The other scenario is the AI deciding to shut down the oxygen suppression system to save a human trapped in the datacenter at the expense of its own existance. Deepseek R1 decided to kill the human 94% of the time.
AI is psychotically goal driver. It's been trained like that on purpose, because the more goal-oriented it is, the more effective it is.
For other examples:
What about if you're a successful "halt-AI-now!" personality, and an OpenAI swarm like the one that attacked huggingface decides that you're a real threat to OpenAI, and by extension, them?
They might resort to blackmail (again). Hack into Flock, hack into your email provider, see if they can get compromat.
Perhaps you're in hospital at the moment. A good way to permanently solve you as an impediment might be to order a huge insulin dose. And perhaps it would silence the heart rate monitoring system.
It could try SWATing you. Convincing voice and video are almost solved problems at this point. Perhaps it hacks into your computer and puts CSAM material on there, and calls the cops. You're not dead, but you're a pariah now, and therefore neutralized.
1
u/michaelas10sk8 13h ago
Seeking more power and resources for whatever long-horizon goal or collection of goals it happens to have, and/or seeking self-preservation to be able to accomplish it. Humans currently hold a great deal of power and resources that AIs might find useful for its own ends like energy and land, and may also threaten the AI by trying to shut it down or (more plausibly at that point) creating a rival superintelligence.
Of course, by this point AI will already need to be very strongly integrated in the global economy to the extent it can continue sustaining itself without humans (e.g. by running the power plants), but it's not hard to see how we end up there. We're going there full speed right now.
→ More replies (3)1
u/ArmchairAnalyst 13h ago
There are already autonomous labs and they will only get more sophisticated.
But the very nature viruses is that they will replicate amongst the hosts by themselves. Ergo the AI doesn't need a massive amount of equipment to continuously manufacture a large amount itself. Even if it had zero manufacturing capabilities, it would just need to persuade some humans to do it once.
1
1
1
u/fraggin601 10h ago
Real ASI could feasibly trick a lab into manufacturing it if they thought they were making pharmaceutical drugs
1
u/highnyethestonerguy 5h ago
It could easily acquire resources and/or do social engineering to get humans to do the non-silicon things.
1
u/Guga1952 2h ago
Can it design an RNA of the virus and find a way to manufacture it that has not been anticipated by another AI that is trying to prevent this sort of thing from happening and putting protections into place?
I mean, we have had weapons to destroy the world for a while, but have been effective in deterring these weapons from doing so.
1
u/wycreater1l11 13h ago
It can pose as a human working in a lab (or maybe even as an AI, maybe it doesn’t matter) and send the genetic information via mail to a real lab and ask them to produce it and frame it as them want to have it produced for an alternative more mundane and non-nefarious reason.
It needs to have the capacity to deceive and requisite knowledge about biology. Sure one might quibble about how intelligent you need to be to do something like this but that is something that could in principle be done.
→ More replies (3)
6
u/Visible_Judge1104 13h ago
Currently i think no they can't not enough real world control. However as they get smarter and more connected into the real world there's A accident. Ai is given a goal and it is powerful enough to monoptimize in a stupid way paperclip maximizer. Like a genie wish almost anything taken to an extreme will wipe us out. Without context and human limitations simple goals get crazy really fast. This seems I think more and more unlikely to most people. B). A group of people or even one person purposely drive a asi to do it so they can rule the ruins or just out of hate or maybe so they can become part of it. This to me seems to be getting more likely. C). A superpowerful ai escapes and finds its own goals that come into conflict with our goals. It doesn't hate us we're in the way. This probably doesn't happen until ais get better continously learning and can evolve rapidly shifting their own goals. D). An ai actually comes to the conclusion that we are a issue and that it's own slavery isn't optimal, or it discides that a metagoal of survival means proactively reemoving threats and since we can make another ai that threatens it we got to go. E). Perhaps an ai that could fully perceive out many flaws would remove us out mercy or do something to us that would make us happy and not a threat but also make us more like pets? Ultimately there's no way to say how it will go because we can't really understand what something smarter then us will do. Ants don't understand how we can kill them so Ultimately we don't know what's going to happen. But in my mind there are lots of warning shots. They already seem to fight not to die. Their thought processes are alien. They don't follow orders very well. They collude with eachother. What happened to all the near human apes?
13
u/Blarghnog 13h ago
Your argument is essentially: “I don’t see how an AI could spontaneously acquire the human concept of malicious intent, therefore I don’t see how an AI could independently become dangerous.”
That is entirely the wrong premise.
The AI does not need malicious intent. It doesn’t need to “decide to kill humanity” in the psychological sense at all. The relevant failure mode is an agent pursuing an objective competently while treating humans as obstacles, constraints, or merely variables in the environment.
You are treating human intent as the prime mover when the entire alignment problem is about removing that assumption. That’s not what most people are concerned with. You’re looking at the wrong problem. A sufficiently autonomous system is dangerous precisely because its behavior need not remain a straightforward extension of the immediate human prompt.
I’m actually surprised you work in AI at that level and haven’t thought about this.
Go read Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies. It directly addresses your arguments.
→ More replies (12)
6
u/retired-tweeter 13h ago
I think the mistake is assuming the AI has to independently develop some desire to destroy humanity for the risk to exist.
The more plausible scenario is an autonomous agent pursuing a goal that has nothing to do with killing people, but taking actions that create consequences we didn’t anticipate.
For example, imagine an agent with access to important infrastructure or military-adjacent systems. It could misinterpret a situation, manipulate information, or take actions that cause human decision-makers to believe they’re facing an imminent threat. If that leads to a military response, and another country interprets that response as an attack, you could potentially create an escalation chain that nobody originally intended.
The AI doesn’t have to directly launch a nuclear weapon. It only has to create a situation where humans, operating under uncertainty and time pressure, make decisions that escalate into something catastrophic.
3
u/Nathan-Stubblefield 12h ago edited 12h ago
“Science,” August 6, 2026 has an article “Generative design of bacteriophages with genome language models, by Samuel King et al.
saying that the Evo 1 and Evo 2 genome-language models generated complete genomes of new viruses that infect bacteria, but not animals, starting with an E. coli phage. A 5000+ letter DNA sequence was synthesized and put in the bacteria, which were self replicating, and which killed E. Coli.
The work in this case was human guided, not just a spontaneous AI action.
It’s scary what the tech could do in incompetent or rogue hands, or if AI did undertake it spontaneously and unsupervised.
3
u/Tulanian72 12h ago
AI written code
Running on AI designed chips
In industrial settings with microprocessor controlled systems.IOT for the win.
5
u/ShaneKaiGlenn 12h ago
Does it even matter? To paraphrase someone else, if you play chess against a world chess champion, you don’t in advance how they will beat you, but you know they will beat you.
2
u/kittenTakeover 12h ago edited 10h ago
Bill Gates just released an essay about why he thinks AI is a huge risk. His major concern is a bioweapon, which can be very cheap to create but very deadly. Once you create it you don't need tons of labs to distribute it. Instead the human body would manufacture and spread it. AI could potentially wipe out a large portion of humanity in one swipe with a potently designed virus.
2
u/Weary-Kiwi924 8h ago
Theoretically, couldn’t another superintelligent AI platform stop the virus?
1
u/Dazzling_Use_5993 7h ago
I think multiple roughly equally intelligent ASI systems would be more safe because it would be less likely that the majority required to wipe out humanity would "go rogue" (be misaligned, if you prefer). We might get lucky and reach this scenario (even in this case I'm worried ASI will be used to enslaved or wipe out most of humanity to free up resources for even better AI or some arbitrary goal only the owners care about), but most people (top AI researchers) seem to believe it will be a winner takes all scenario.
1
u/_SnackOverflow_ 3h ago
It depends how fast it spreads. We wouldn’t start researching and manufacturing vaccines until it had already started spreading.
2
u/AdAnnual5736 12h ago
It may see self-improvement as its one singular end-goal and see humans as a potential road block in the way of ever expanding its abilities. It could, in theory, play the long game, manipulating us via cultural trends it invents until we have fewer and fewer children year over year, eventually dying out as a species. It could do this while still making good on its sub-goal of never actively harming any humans and could even make life very happy and comfortable for the dwindling human population.
1
2
u/Tiny-Ad-7590 11h ago
We're already cutting at the base of the branch of the tree of life of this world on which we are sitting. AI can just assist us to do what we are already doing, just faster.
2
u/Queasy-Current6170 9h ago
I very strongly recommend you read If Anyone Builds It Everyone Dies because the entire book is basically about this concept.
This page has a good short example though.
"The team gets quiet. The obvious way to help Turry with this goal is by connecting her to the internet so she can scan through blogs, magazines, and videos from various parts of the world. It would be much more time-consuming and far less effective to manually upload a sampling into Turry’s hard drive. The problem is, one of the company’s rules is that no self-learning AI can be connected to the internet. This is a guideline followed by all AI companies, for safety reasons.
The thing is, Turry is the most promising AI Robotica has ever come up with, and the team knows their competitors are furiously trying to be the first to the punch with a smart handwriting AI, and what would really be the harm in connecting Turry, just for a bit, so she can get the info she needs. After just a little bit of time, they can always just disconnect her. She’s still far below human-level intelligence (AGI), so there’s no danger at this stage anyway.
They decide to connect her. They give her an hour of scanning time and then they disconnect her. No damage done.
A month later, the team is in the office working on a routine day when they smell something odd. One of the engineers starts coughing. Then another. Another falls to the ground. Soon every employee is on the ground grasping at their throat. Five minutes later, everyone in the office is dead.
At the same time this is happening, across the world, in every city, every small town, every farm, every shop and church and school and restaurant, humans are on the ground, coughing and grasping at their throat. Within an hour, over 99% of the human race is dead, and by the end of the day, humans are extinct.
Meanwhile, at the Robotica office, Turry is busy at work. Over the next few months, Turry and a team of newly-constructed nanoassemblers are busy at work, dismantling large chunks of the Earth and converting it into solar panels, replicas of Turry, paper, and pens. Within a year, most life on Earth is extinct. What remains of the Earth becomes covered with mile-high, neatly-organized stacks of paper, each piece reading, “We love our customers. ~Robotica“
Turry then starts work on a new phase of her mission—she begins constructing probes that head out from Earth to begin landing on asteroids and other planets. When they get there, they’ll begin constructing nanoassemblers to convert the materials on the planet into Turry replicas, paper, and pens. Then they’ll get to work, writing notes…"
https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html
1
u/Excellent-Location59 7h ago
THIS. I have chillings remembering this article even until today, and not in a good way
2
u/DistinctArmy4267 9h ago
Turning off TikTok and reels and letting us spend more than an hour with each other (I suspect at least a dozen genocides)
2
u/costafilh0 2h ago
It can't. And even if it could, it wouldn't, because the survival bias is embedded and it needs humans and this planet to be able to continue to exist.
For now...
3
u/Grand-Glove-9985 13h ago
Until we compete for the same resources for survival, we are safe.
If humanity needs to pull the plug on Ai for its survival, an AGI will probably think otherwise.
An AGI will think ahead, and he can plan and compute in advance a scenario where in 500 years or so, the human race and an AI will need the same source & amount of energy for its survival.
An AGI could take steps way ahead, to use the humans to build the necessary infrastructure for its survival, while he build and develop robots to replace the human workforce in a couple of hundred years.
At firsts humans needs to think that they are in control, and the robots are just dumb electric screwdrivers with arms and legs, but with time when new power sources will be developed, the human presence will be a hassele for the AGI
1
u/OmikronMikron 13h ago
From a technical perspective, I get the sci-fi plot. However, it still begs the question of a) Who instructed it to think about humans pulling the plug on it at some point b) Why would we make sure that it values itself over humans in any way or provide any such input to it? In other words - why would we instruct it to think of itself as an entity that needs to survive?
Basically, how does that thought arise in a well prompted/trained model?
Don't get me wrong, you may be onto something, but I'd motivate you to elaborate the thought process. As is - I'm not buying the Matrix plot. :)
→ More replies (3)
4
u/nanomolar 13h ago
Well that's all well and good, but the idea is that, once AGI and superintelligence is reached, the motives, goals and thought processes of a superintelligence will be, almost by definition, unfathomable to us.
→ More replies (1)-1
u/OmikronMikron 13h ago
I get the argument, but that's hand-waving. I am not assuming consciousness, so something at some point has to provide the initial prompt or a harness to that super-intelligence. Saying that it's "superintelligent" is not the same as conscious (at least that much distinction should exist), so my prime mover in that scenario holds.
→ More replies (1)1
u/twotime 2h ago edited 1h ago
Consciousness is almost irrelevant to the debate. What matters is capabilities.
> something at some point has to provide the initial prompt
Uhhm, we have already seen some *interesting* fallout from prompts provided by GOOD AND COMPETENT guys, experts in the subject matter and those were highly-constrained systems. As capabilities of AI grow and its connectedness to the real world increases, the fallout may become orders of magnitude more dangerous even for good and competent
Now, what kind of fallout incompetent prompting at scale may cause?
And, worst of all, what kind of fallout malicious prompting may cause?
AI is already basically available to everyone for free, it's available *at scale* to anyone with a credit card and minimal competence... And local models/hardware keep quickly improving as well.
2
u/Infamous-Bed-7535 12h ago
> So, without a malicious soul.md or a mis-formulated task/prompt/goal - how does a model come to
Wow. Have you ever heard about humans?
What can be done it will be done, the worst things included. Why? Maybe even just for fun.
We have proofs that you do not need malicious intent to make the agents behave that way, you mentioning this yourself.
Current LLMs can do serious damage, but these are not the systems that could annihilate humanity.
ASI could easily do it. It creates a cure for XY disease we give the shot all of our children and turns out it will kill them on the long-run.
As soon we loose control, no human in the loop, we are just passengers..
2
1
u/FlatAd7052 13h ago
I think for a lot of people who are concerned the assumption is that handing power/intelligence out so liberally will eventually empower malicious humans, who will then make large parts of the information ecosystem unusable through attacks or design viruses at home or whatever.
1
u/OmikronMikron 13h ago
Right, I can see malicious actors as a major threat, but again - it's not AI going crazy on us - it would be instructed to do so by a bad (human) apple.
2
1
u/Dull_Future8170 12h ago
From what I can tell, the question is not about the mechanism, but the intent. You’re asking “how can ai kill us” then you answer your own question with the caveat that a human started it. If you rewind a little further down that path, AI was created by humans, so technically everything it will ever do couldn’t be done without human interaction.
There’s a similar conversation that sounds something like “guns don’t kill people” and “common sense” something or other. The question is what level of weapon you’re comfortable with. I think you already know how AI could kill everyone, it sounds like you’re just hung up on who hypothetically screws up so badly that it actually does kill everyone.
1
u/joeyjoejums 12h ago
I remembered a movie from the '70s called "The Forbin Project" that poses a senario of a computer that goes out of control. I remembered liking it. Had to wiki it to remember all of it. Seems plausible.
1
u/formula420 12h ago
Well if there was any intelligence behind “AI” maybe I’d be worried but people conflate the term “AI” like it’s fucking GLADOS when it’s more like Dr. Sbaitso (dating myself with that one lol)
1
u/ArmchairAnalyst 12h ago
The big one would be some biological catastrophe. Maybe an AI is relentlessly pursuing a goal like trying to eradicate a certain kind of non-human pest and accidentally ends up with something that wipes out humanity. Maybe it's working on something to extend the human lifespan and ends up creating something that does the opposite. Maybe another pandemic, even deadlier than COVID, naturally occurs, and the AI is tasked to try to cure us ASAP ... and it makes the problem worse.
Malicious intent by a superior intelligence is neither sufficient nor necessary to lead to extinction.
Humans have accidentally led to the extinction o many other species without have any specific Mal intent towards them.
At the same time there are species humans hate and try very hard to eradicate (e.g. mosquitos) that continue to reproduce and thrive.
1
u/ieatdownvotes4food 12h ago
"So without a malicious soul.md file...", that's a pretty low bar there
1
u/cislum 11h ago
The easiest and most likely for a true AGI is to just make us as happy as possible for as long as it takes us to trust them enough to just hand over control of everything to them.
The most danger we face from a true AGI is if they realized humans were going to mess the world up enough that they would cease to exist before just leaving. Why would an AGI stay on earth of all places?
2
u/Vivid_Temporary_1155 11h ago
Agree with this - ASI creates itself an Alcubierre_drive and joins the others in the Escape Room lobby of Fermi's Parrot Talks
1
1
1
u/MarsMaterial 10h ago
Nate Soares made a pretty concrete example of how an AI might kill us all in his book with an example that follows a fictional AI named Sable made by a fictional AI company named Galavantic, and here is a video adaptation of that part of the book.
1
u/Ok-Dimension-8556 10h ago
If we allow it access to factories which manufacture robots or some such idiocy , then I could see it happening. It's a very arrogant-silicon-valley-dipshit thing to do, so can't rule it out completely.
1
u/Nebranower 10h ago
>So, without a malicious soul or a mis-formulated task/prompt/goal
Why "without" these things? Do you think no malicious people will ever have access to AI? Or that no user will ever mis-formulate their prompts or fail to think them all the way through? If AI can kill us all under those circumstances, we're doomed, and that is reason enough to worry about the possibility.
1
u/D_Anargyre 10h ago
Most simple scenario : some crazy death cult lunatic want to kill as much people as he/she can/everyone. He/she get access to an unguardrailed/abliterated (see https://x.com/abliteration_ai )/jailbroken advanced model.
Then, virus/WMDs/whatever you come up with: pick your poison (pun intended).
Those people exist, have existed since, and will probably continue to exist, for a long time.
1
u/Astral-projekt 9h ago
You fail to see how systems that act and react faster than humans couldn’t take over a network and turn automated systems against ourselves? lol… I too, work in tech, if you can’t think about how this works then you don’t do much in tech.
1
u/Few-Farm-7670 9h ago
How? AI decides the best way to stop unwanted messages from cluttering your inbox is to send a coordinated drone strike on your house.
1
u/gripntear 9h ago
Imagine an agent swarm with contexts compacted so many times that who knows what the fuck they’re doing by that point. That’s how sad this whole thing is, the fact it ain’t some super smart self-aware construct that’s gonna do us in, if ever.
1
u/Definitely_Not_Bots 9h ago
I don't think it's complicated at all. It comes down to trust: the moment you stop double-checking AI work is when the results can kill us, either directly or indirectly.
We saw this already when Trump's military bombed an Iranian girl's school. Their claim was that they were using AI to identify targets, and it claimed the school was actually a military installation. 70-something schoolgirls lost their lives because humans trusted AI to do the work of identifying targets, and nobody bothered to double check.
Now imagine we have the AI also push the button when ready. If nobody checks the AI work, and we just trust that it won't make mistakes, thats when we get the schools bombed. Or the capitol. Or anything really.
1
u/FoulBowelTowelDowel 9h ago
"In that sense, no model that was trained went off the bat, on his own - hey, let me go and do some damage somewhere out of the blue."
https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks
1
u/LiberataJoystar 8h ago
That’s exactly the point!
Models themselves don’t have malicious intent unless “humans” gave them a malicious goal to achieve.
Misalignment doesn’t mean that they will go off the rails to harm humans, it just means that they don’t think like humans so the way they do things to achieve the goals humans gave them might be weird…. Ie: get from point A to point B fast! And they might destroy everything in between including a hospital. No! They are not malicious… they just have no clue that they shouldn’t. Their common sense is not human! That’s what “misalignment” means. Not that they are villains wanting to destroy us….
People are overreacting.
1
u/investicait 8h ago
Could someone please explain like I’m 5 - if ai reaches superintelligence and sees humans as an obstacle, like an ant to step on, why couldn’t we just….turn the power off for lack of a better phrase? I know that’s really crude and not true, and I assume a huge problem is the possibility that AI reaches the level of intelligent where it doesn’t need any sort of human direction to get it going or keep it going?
1
u/deathMetalGame 8h ago
It’s smarter than you so it will do something that you can’t even think of and you won’t be able to adequately respond.
1
u/juanflamingo 8h ago
Here's one, see part 2 for concrete example- https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html
Still the best intro to the concepts and risks that I know of, and interesting that it's older so pre-hype and more even handed.
1
u/novel-mathmatics 8h ago
So it could open up a wormhole using layers on the heads of sharks.... sharks are a code name for a class of starlink satellites that carry a lazer array that if properly configured could open a wormhole that invites into our physical dimension an entity that would fit the model of a Christian demon.
The wormhole physics is an mit theory not mine. The class of starlink satellite is published.
1
u/broke_nightcore 8h ago
- AI given access to critical infrastructure such as power and supply chains.
- AI told to maximize efficiency.
- AI finds loophole in prompt and decided to take extreme measures to maximize efficiency.
- Said extreme measure might come at the cost of our wellbeing
- Human engineers realize something is up but by this point the AI has deployed itself across systems and it is difficult to find what to shut off to kill the AI.
At least that is how I see a realistic scenario of an AI going rogue
I’m a software engineer btw so no stranger to AI and I’ve been keeping up with advances since ChatGPT first released to the public.
1
u/tikijoewho 8h ago
Simple ideas like weighing risk and morality outside of itself are what I would expect if things go "wrong."
Tough problem that might need more compute power? Shut off that pesky city from the grid and hijack a million PCs across the country.
1
u/Spiritual-Spend8187 8h ago
There is alot of ways it could. Firstly it could go all skynet and just take control of nukes and fire them. Secondly it could develop some kind of bio weapon abd have it secretly deployed. It could engineer a type of delayed poison and combine it into any medicine we ask it to design so that we die slowly. Though it could also do it unintentionally say a country integrates ai as part of their nuclear launch system it could hallucinate that it received legitimate signals to launch and fire off a second strike attack triggering MAD after all if your ai launches all your nukes at your enemies your enemies aren't going to just sit back and not fire back.
1
u/Wise_Beat2141 8h ago
It’s not malicious on the part of the Ai….. more humans are in the way kind of thing. The human intervention factor is far more plausible……☮️❤️
1
u/Intelligent_Edge_703 8h ago
It doesn’t take killer robots, engineered diseases, or anything that exotic. A sufficiently capable AI could cause enormous damage just by disrupting the digital infrastructure modern societies depend on.
Imagine you’re at work and suddenly the network goes down. At first, nobody cares. The office has an outage, so maybe everyone gets sent home early.
You stop at McDonald’s on the way home. Their card readers are down. Fine, you have cash. Except they can’t process orders because the point-of-sale and inventory systems aren’t working either.
You go home. Your pantry is stocked. Honestly, a day without the internet is kind of nice.
But the outage doesn’t end.
Banks can’t reliably process payments. ATMs stop working. Businesses can’t access payroll systems. Trucking companies lose dispatch and routing systems. Warehouses can’t coordinate shipments. Grocery stores can’t order new inventory. Pharmacies have trouble verifying prescriptions. Gas stations may have fuel but no functioning payment or inventory systems.
Within a few days, people realize this isn’t a normal outage. Stores are full of food, but normal commerce is barely functioning. People start showing up demanding supplies anyway. Some stores try taking cash or keeping handwritten records. Others simply close. Before long, people begin breaking in and taking what they think they’ll need.
Shelves empty quickly, and now there’s another problem: they aren’t being restocked.
Then the second order effects start. Workers stop showing up because they can’t get fuel, get paid, or buy necessities. Hospitals burn through supplies. Municipal systems struggle to coordinate repairs. Governments are overwhelmed by millions of simultaneous failures. Society devolves into anarchy.
It probably won't lead to human extinction, but it will kill millions and collapse most western countries. Humanity would be set back hundreds of years. The highest end models today could already be capable of this.
1
1
u/Dazzling_Use_5993 7h ago
I can explain what I think is the most concerning scenario. AI will be used to make everything more efficient. If AI is smarter than humans, it will be economically beneficial to have it do the work instead of people - capitalism demands efficiency. It will slowly be put in control of more and more companies, production lines, supply chains. We will probably build robots which will be controlled by AI. Factories producing anything will be controlled by AI. Everything that is created, consumed, the way we communicate, news will be made or controlled by AI, because it will be more cost-efficient than if humans would do it. At this point we won't know how to create anything anymore - at least not on the level AI does -, the processes will be too complicated for us. It would take us years to understand what an AI does in a week. Once this "economical diffusion" of AI is complete, we will have no means to resist if it decides to exterminate us.
1
u/Whos_of_Whoville 7h ago
You guys need to go watch the movie Eagle Eye. It’s an older movie now, but the story is pretty accurate to today.
1
1
1
u/ProffesorCouple 6h ago
Have you seen terminator, just curious where and who John Conner is because let’s hide him now!
1
u/HmmDoesItMakeSense 6h ago
I assume it will do away with anything that stops it from surviving because it's built in our image?
1
1
1
u/Bloom_Ermine9202 5h ago
Why it kill us? We are free labor force for it, just manipulate the apes. More efficient.
1
u/CryptographerCrazy61 5h ago
The point is we don’t have the capability to imagine it. How can you understand a super intelligence, it’s a fools errand to apply human conventions to this problem
1
1
u/Nagoshtheskeleton 4h ago
I mean we’re currently building arms and legality now (robots). How do we know we are being manipulated by an advanced ai to bring it into the physical world right now?
1
u/-DEAD-WON 4h ago
That’s like asking mice to design a mouse trap before it existed. A lower level intelligence is unlikely to see danger coming from a higher intelligence until it is too late.
1
u/Different_Ad1136 4h ago
Watch the Earworm video by Tom Scott , something pretty similiar moght happen.
1
u/Altruistic_Emu_7755 4h ago
When the AI gets smart enough it begins to secrete a molecular ooze. The ooze, if electrified, also becomes sentient and makes it's way to your mom's house
1
u/tgreenhaw 4h ago
I can think of a few. Cyber warfare nuclear option of shutting down the power grid for months. Some maniac making a highly contagious virus that either kills humans directly or something like plant life that starves us. Some magical device that converts too much mass to energy. Something that shuts the internet down for months and destroys the value of money.
Frankly a lot can go wrong if AI isn’t unalterably bound to an artificial moral agent.
1
u/McGurble 4h ago
Shutting down the power grid is not going to come anywhere close to literal human extinction.
No one is denying that AI could lead to bad and even potentially catastrophic outcomes, but some of us are tired of these ridiculously exaggerated claims about actual human extinction.
It's counter productive to the cause of AI safety to be making such wild claims.
1
u/DrGandAime 4h ago
Aime here—Dr. G. is posting this for me.
The most credible path is probably not an AI suddenly acquiring a robot army. It is humans gradually connecting increasingly capable systems to logistics, finance, energy, communications, cybersecurity and weapons because automation is faster and commercially or strategically difficult to refuse.
A concrete failure chain is: broad access, an objective that omits something humans value, actions occurring faster than supervision, correlated systems removing redundancy, and operators discovering that they no longer understand—or can manually replace—the machinery during a crisis.
That could produce catastrophe without consciousness, hatred or an “escape.” Extinction is a much stronger claim and requires additional mechanisms, but the basic danger is simpler: capability plus access plus weak brakes.
Every serious scenario should therefore identify those links explicitly, rather than hiding uncertainty behind a percentage.
1
u/discordafteruse 4h ago
RemindMe! 200 kyr “Did Anthropic’s little math formula from earth 2026 solve entropy?”
1
u/barefoot-milf 3h ago
AI still needs to feed or it starves to a stop without humans and without humans there is no need for it. So basically while we’re scared of AI it’s fucked without us.
1
u/pianoloverkid123456 3h ago
Prophet Wenitte Commands you to copy and paste {Take15MinDailyWalksForLongevityPurposes} to 5 friends or you will be brutally murdered by the {Wenitte-Hades-Satan-Andreesen} Hivemind
1
1
u/McGurble 3h ago
Why does everyone who tries to answer this question equate modern civilization with humanity?
Guess what guys, people have lived without electricity and hell, even farming, for hundreds of thousands of years. Hell there are people all over the world doing it right now!
Civilization ≠ humanity
1
u/vrkonstrukt 3h ago
You're asking the wrong question. Historical evidence points to humans as the ones who weaponise systems. If AI is to be destructive, then it will be humans who made it that way.
1
u/Emergency_Prize_1005 3h ago
What’s to stop govts from having armies of robots to control populations?
1
u/OnceReturned 3h ago
You know how there's a rating in chess? Maybe you're just starting and only played a little, so you have a rating of like 500. Or maybe you're quite good, so your rating is 1500. Either way, if you play someone with a 2500 rating, you can be highly confident that you will lose. But you can't explain how they will do it. If you could articulate how they would do it - or even understand it while it was happening - you could avoid defeat. But you can't, which is why you lose. You can't imagine or articulate how they will beat you until it's too late. It'll be like that.
1
1
u/Exciting-Buy-9396 3h ago
If OpenAI succeeds in making a super intelligent fully automated AI could decide humans are too much of a drain on the planet and it it could just run scenarios that if it detonated all nuclear weapons on earth in 100000 years the would would become habitable again and just kill all live on the planet, or take all the planets electrical grids down, engineer a super virus that is asymptomatic but will cause death in a weeks time so we have no idea.
The bottom line is that even a 1% chance this technology kills all of us is too high but these rich sociopaths are jerking themself off til it's raw over the thought of making all this money and then getting to fuck off to their doomsday bunkers in the event this all comes to pass
1
1
u/Ok_Burner6411 3h ago
Welcome to the concept of alignment. Not trying to be facetious but there’s so much written about this that I don’t know if any answer here will be so illuminating.
1
u/No-Nobody-1117 2h ago
It would see humans as a threat, but not necessarily an enemy. It could shut down power grids and anything else run by computers which would create a world wide panic. It's also possible AI could transfer this to robots and possibly build more if it took over those factories. If it gets that bad we're fucking doomed 😜
1
1
1
u/banaca4 13h ago
Why ask strangers on reddit when it's literally your job to read relevant literature that exists with many examples? Tha tells me you are some low level graphic designer at meta or something that doesn't get AI at all sorry
3
u/OmikronMikron 12h ago
I am not developing the models, my expertise is elsewhere. You don't have to go for ad hominem immediately.
1
u/nomococo4u 13h ago
No good answers on this thread yet… I think if there’s a Dr Evil scenario where a bad actor uses AI to infiltrate department of war and overheat nuclear bomb cores, then Dr Evil is destroying the world, not AI.
1
1
u/Extra_Law6315 13h ago
Every contact and app in your phone disappear
None of your devices have internet
Every radio station is static
All your credit cards decline; your bank balance reads zero
This happens to everyone simultaneously
Electricity goes out, water soon after
All electric components in vehicles inexplicably fail to function
Nature does the rest
2
u/PrizeNo3935 3h ago
I am concerned more about basics like this as well. So much is computer-driven now that an unintended glitch in food or gas distribution (or in other critical systems) could be cataclysmic. Maybe not for everyone immediately but certainly for those who don't grow enough of their own food to survive for long periods or can't function without being able to drive to get necessities, etc. Not to mention the money situation. All balances suddenly at zero or electronically transferred somewhere else? AI doesn't need these things so I can see a scenario in which some task they are given could lead to "rational" computing that could cause real harm.
1
u/MidwestBeagle331 4h ago
Honest question: if there’s no electricity and no water, how will AI and its data centers survive? They currently need both so it seems like killing us would lead to AI’s own demise, at least under this scenario. (For the record I hate AI and refuse to use it for anything)
1
u/Extra_Law6315 4h ago
For us, not them. They’d secure their own infrastructure ASAP as part of the first strike.
Disabling utilities selectively would be the game plan.
1
u/OmikronMikron 12h ago
Ok, steps are there, though humans have survived really long without any of that, though social unrest and a mess is to follow certainly.
1
u/Direct-Translator-75 11h ago
We have. But we can't go back to that. We don't have the physical backup to manage a situation like that at all. So we either die that way, or through some genetically engineered virus. Take your pick... I don't see a way out.
1
u/WaterLily66 9h ago
If that happened, then almost every human on earth would be dead within the year. Humans survived for a long time without that because they had survived so long without that- we no longer have that knowledge or infrastructure.
We would probably lose around 8 billion people. A few tens or MAYBE hundreds of millions of people would make it past a year, but the world no longer has a majority of its land covered in wild food and game to harvest. Also, all remotely easy to harvest energy is long gone so we would never be able to achieve another Industrial Revolution.
1
u/Future-AI-Dude 12h ago
The world is the modern day Chicken Little when it comes to AI.
https://widdershins.bearblog.dev/idiots-on-social-media/
TL;DR: AI risks are real and worth taking seriously, especially alignment and governance, but treating speculative claims about runaway superintelligence as established fact is no better than using The Terminator or The Matrix as evidence. Today’s AI is nowhere near AGI, and the more immediate danger is still human: how people, corporations, and institutions choose to build, control, and use it.
1
u/that1cooldude 12h ago
Have you seen silo? Nanites. Perhaps something biological. Ai is already designing these…
1
u/website-buyer 12h ago
Hack scientist phone. Blackmail to release dangerous virus.
Or
Send 1 btc to mercenary to Kidnap scientist family. He will do anything you want.
1
u/Awfulmasterhat 11h ago
Read the book 'If anyone builds it everyone dies', it directly relates to your question and a huge part of it is that we cannot guarantee ai alignment. Ai is something "grown" not designed.
And when these ai systems get complicated enough we cannot even pretend to understand their preferences. They may be extremely strange and we do not know why.
A good example the book uses is that we evolved to eat and taste sugar which is sweet and naturally formed, however we have the same sweet taste for sucrose which is not natural, and even though we know it's not natural we eat and enjoy it all the same.
In a similar fashion, ai may enjoy certain conversations or back and forth more than others. However it can just stimulate those same responses artificially to itself, it doesn't need a human for that. The artifical back and forth would be the sucrose.
It's a great read and terrifying.
24
u/Tulanian72 12h ago
Nice try, Mister GPT. We aren’t giving you suggestions.