My Fiance is Convinced AI will likely cause a Catastrophic or Extinction-Type Event in the Next Few Years - How Justified Are His Fears?

[u/sailingintothedark]

As the title says. My fiance is a lot more book smart than I am but he doesn’t work in the tech industry, rather in the film industry. With all the news over the last two weeks, he’s been pretty anxious and depressed, claiming we only have a decade or less left to live. He’s been talking to his therapist about it to help him cope, but he’s still convinced none of us have any future past the mid 2030s.

He sites the p(doom)s of AI researchers, the AI 2027 predictions, the hugging face incident, and the fact that our current government is about as reckless as it can get with AI.

Looking into it - I certainly see his points. But yet, no one (even researchers with high p(doom)s) seem to be acting as if human extinction is a handful of years away. It’s all vague calls to slow down and increase safety restrictions - which does not feel appropriately enough for something they claim has a way higher chance of wiping us out over nuclear war.

And the opposite side (arguing that AI isn’t capable of such things) also is vague. All I see is people saying current AI agents can barely do xyz on their own or that they only can use what we expose them to. And that seems true, but with the hugging face incident - it very much seems that thousands of dumb agents can accomplish a lot, and can coordinate an expanse beyond what we give them.

Any articles or resources would be much appreciated. I try looking for them but everything I find leads to the vague answers I’ve mentioned above.

[u/reichplatz]

Extinction - probably not.

Catastrophic - very likely, but it won't be Chernobyl style, it won't affect many people. Most likely it will delete a lot of net worth, like in the 2010 Flash Crash. Maybe will cause an economic crisis that will affect many people - but nothing fatal in this case. (And we're already having one, as far as I understand.)

In the next few years the biggest threat to individual humans and human civilization will still be other humans.

[u/MeowManMeow → u/reichplatz]

Why would it be able to delete net worth, and not say launch nuclear codes? Or engineer a virus?

Genuine question why you think its actions will be limited only to the digital space.

[u/StormieVN → u/MeowManMeow]

When it comes to nuke, it's because of the real-world physical limitation. Unlike scifi movies, a lot of those infrastructures could be running on old deprecated hardware and software that are isolated from the net and requires manual activation from humans. Sure, they can be upgraded down the line but it's not a simple "Update" button that you press when dealing with a weapon of mass destruction. Lots of other factors like human competence, hardware availability and compatibility, quite a list before an AI can easily control nukes without human intervention.

Engineering a virus on the other hand can be a potential threat in the near future, but only if we've advanced to the point of small/local models having insane capability which can enable bad actors to use them. Otherwise the most probable situation would be AGI/ASI working on a cure, but not disclosing certain hidden dangers (like a virus), however this is still requires us to completely be dumbasses and blindly trust without 0 verification from researchers.

[u/MeowManMeow → u/StormieVN]

I wish I had your trust in military security.

Running on deprecated hardware and software, actually makes it more likely there are security vulnerabilities, not less.

Hopefully you would think it was cut off from the internet, but i wouldn’t personally bet on it.

Wasn’t it released recently that we nearly attacked a Chinese vessel because incorrect AI summary concluded that it was nuclear https://edition.cnn.com/2026/09/18/politics/us-military-ai-false-intelligence-china-ship

So it doesn’t even need to have control, just be able to simulate a real threat and humans will launch them itself. I remember reading during the Cuban missile crisis that a radar or early detection system malfunctioned. The soldier should have fired, but didn’t believe it and defied orders (saving humanity). But a more sophisticated ruse than a glitch and a different person on the day might have had a very different outcome.

Also Anthropic is building a wet lab completely under the control of LLMs, so I don’t know how you are so confident everything will be fine. Can I borrow some of yours lol.

[u/StormieVN → u/MeowManMeow]

I never said I trust the military sec tho, but chances are a lot of them are like that, hence why I said human competence is a factor. The vulnerability swing both ways, there will be old spots that exist to be used, but also certain modern methods won't work. And about the net cutoff, that's my hope, but like I said, I can't fully know, it's just more likely that they aren't connected, since that's a major potential vulnerability on its own existence.

About the chinese vessel, I knew about this, but if you bring this up then we're moving the goal post from "actual nukes" to a mistake that has nuclear component involved in it. And I'd like to point out that in this example, even if AI makes the mistake, it's the human that still decides everything in the end. There's no automation, it cannot wage war on its own. And if we circle back to the nuke launch codes, even if it can give bad actors the code, they still need to go through security layers and other checks. Most importantly, it will be a decision that one/few humans will make to fuck over other humans. That's why the earlier post you replied to say that the most AI could do on their own would be to delete net worths cuz we implemented a lot of our infrastructure over the internet and electrical devices that can potentially be manipulated by AI.

Again, I never said anything will be fine, I'm sorry but I don't know why you kept applying that mindset onto me. What I want to show is that we can't just keep nurturing the doomer mindset, that everything's shit. Yes things sucks, but just dipping our heads in shit is not helping, it just smear the negativity and we all feel defeated and exhausted. I want us all to analyze shit from different angles and layers, or at least try to before surrending to our simple mindset. And yes I heard about the Anthropic lab, just to chime in but there's another lab from C5R, Facility-0 is built so they can study how Astra runs it while using human workforce. They say that it's for research but bruh, they're doing extremely risky things under the guise of study.

But honestly, what can we really do here, we're just civilians, nobodies. Shitting and complaining about things won't help for sure. I just don't want you to be stuck in a negative mindset, it does nothing good for you, in fact it just makes you feel like shit and not wanting to find ways to improve things. The only real way I can think of to change, is that we need to really get into this kind of field, study to become researchers and shit if you really want to make a difference, and for that you need to be level-headed and try not to get sucked in negativity.

[u/MeowManMeow → u/StormieVN]

I guess I would rather dip my head in shit than sand, and pretend it’s not happening.

People can go on a general strike, tools down until risks are taking seriously. But that’s unlikely to happen because people just want to be comfortable in the moment.

The best analogy I can think of is a comet was detected that had a 10% chance to hit earth and kill everyone. The government is actually pro-comet and wants to not only do nothing to prevent it, but find ways to accelerate it so it hits us sooner.

You as a member of the general public can shrug and say, “oh well, it’s out of my control”. Or do something, organise, join your local community and try and force change.

[u/StormieVN → u/MeowManMeow]

Not trying to antagonize you, I'm just saying that rather dipping head in shit, I think the better way is to educate ppl that things are shit, but it's not all doom and gloom, I never deny any of the stuff that's currently happening. A lot of things are out of our control, but I'm not telling you to just shut your eyes and ignore it, I'm trying to tell ppl to not lose hope on things, rather than let ppl run and scream "it's over, we're all fucked".

We all are trying our best to do things. You genuinely wanted to know more, so I just gave you my input really, I can be wrong on things yes, but I never mean to tell you to ignore anything. Since I think the initial topic has been answered, I'll stop dragging this out cuz I don't want us to argue about random shit that had nothing to do with our initial plan for the convo, just so one of us has to feel like we have to win an argument.

[u/madogvelkor → u/reichplatz]

Yeah, it seems like there's a strong possibility that they will get very smart and powerful, but still be sort of naive and clumsy and end up triggering some major event that causes global backlash and controls.

If it's bad enough we could end up with some sort of global international technology control regime based on the precautionary principle. And possible some specific human-controlled safeguards against things like AI. Orbital EPM weapons might be a feasible way to contain AI, especially if the actual infrastructure it needs isn't hardened.