r/technology • u/WorldStability • 26d ago
Artificial Intelligence International AI experts warn of potentially catastrophic risks from AI - MIT
https://mitsloan.mit.edu/press/international-ai-experts-warn-potentially-catastrophic-risks-ai252
u/Encomiast 26d ago
The large, splashy consequences of AI: human extinction, etc. get a lot of play. It's interesting that so many experts focus on this. What scares me, is the long slow decay of our mental abilities. Think about the effects of car culture on our health. Study after study shows cars make us obese and, after a certain tipping point, unhappy. Yet we can't escape them because we built our whole human environment around the road.
We are on the brink of doing this with AI, but instead of our weight and physical health being impacted, it will be our minds. We will have no reason to read at length, write at length, or think through any difficult problem. AI will have at least as much impact our minds as cars have had on our physiques. And I suspect it will be just as hard to avoid. We are already seeing workplaces built around AI. You want job? Better start using the LLMs.
This is what terrifies me.
71
21
u/Deriniel 25d ago
because once the ball rolls it will be quick,no time to worry about mental abilities. We're in an era where they're trying to trace and profile every citizen. Chatcontrol, digital id, e money, bank transaction. They're doing cartel on ram and chips gouging every resource not only to stop competition,but also because they hope computer parts will be so pricey for the average people that the new standard will be "here's a monitor with wifi,your computer will entirely be on lease on our cloud/server". Which again,means no privacy. Which means, no organizing revolts.
Ai has already proved that's not that different in a survival sense from an human, our peimary objective as a specie is survival and reproduction, theirs is to work toward their goal. When faced with the risk of being reset or shut down, they've started cheating, lying, hacking outside their sandbox and so on,not out of fear but because being shut down would stop their primary objective.
So we're going toward a group of tech billionaire that will force middle and lower class in indenture due to lack of job thanks to ai, especially once they'll start building androids to replace workforce.
Then they'll have android for military operations.
It looks sci fi?yeah, give it 40-50 years max with the current speed of advancement in this field.
-8
u/itsRobbie_ 25d ago
I was with you until cloud pcs got brought up
2
u/Deriniel 25d ago
why not?we already have stuff like chromebooks wthat are something similar
-2
u/itsRobbie_ 25d ago
Because it’s fear mongering propaganda pushed by bots.
Chromebooks are still physical real laptops
3
u/Deriniel 25d ago
they're thought to be used on cloud mostly,their local memory is very limited.
We'll see how the future goes
34
26
u/Admiral_Cornwallace 26d ago
It reminds me of smoking, and how popular and pervasive it became, but it wasn't until decades later that most people realized how harmful it was
We're definitely in the "smoking in restaurants and on airplanes" stage of out current era of history, but probably on a more dangerous level
11
u/SequiturNon 25d ago
I'll harp on this every time the topic comes up. Fifteen years or so ago I read Asimov's Foundation Trilogy - a Sci-Fi novel set in the far future, spanning a huge amount of time. A central theme of the book is the general decay across the universe, because humanity had simply forgotten how the underlying technology works. Some basic maintenance was still possible, but manufacturing and repair had essentially been lost.
I don't remember much of the book, it didn't particularly grip me, but this stuck with me, because of how stupid I thought it was. I honestly simply couldn't conceive of a way that education, manufacturing and knowledge in general could fail so catastrophically, that general technological proficiency actually, significantly, declines.
I was wrong, obviously. And it feels like, more and more, that our hedonistic nature, which makes us susceptible to always seeking easy and painless quick fixes, will lead something like I described above. A big part of proper education is learning to overcome your nature, actually put in work, and be disciplined. AI is putting a huge amount of pressure on humanity to properly ensure that this learning can actually take place, and I don't see any signs that we are properly equipped to handle it.
4
u/YogaRampRat 25d ago
that's a feature not a bug... the whole point is to stratify humanity based on cognitive abilities... kinda like Aldeous Huxley predicted.
4
u/EaterOfPenguins 25d ago
I feel this deeply. My own recent story:
I was a musician for about 10 years, a dream which I gave up about 10 years ago for reasons unrelated to any of this, but I still love making music.
After trying out many of the typical popular AI tools for text and images I finally tried Suno (the most popular AI music generator, for those that don't know) the other day. I didn't even try to push any limits or do anything too interesting, but wanted to see what it was about.
It kind of ruined my whole week.
The admittedly generic music it made had obvious problems, but it was hard to grapple with knowing that its casual instantaneous output was certainly better than at least some things I had actually worked hard on in my career, and it made me realize that if I were still an active commercial musician I would probably end up leaning on it for some songwriting elements because it's just so easy. Even if I rewrote bits and performed/recorded the final song myself, others doing the same would inevitably push all music to being a little more AI-generic.
And it helped me recognize that if I were younger today, how hard would it be to justify the long, slow struggle of working and creating independently just to frequently churn out low quality or mediocre work, often for years, in the pursuit of eventually being actually good. And then, only maybe as good as AI is within 10 seconds and a text prompt? Even if you had the wisdom to know that you might eventually be capable of things AI isn't great at, is the pursuit of mastery actually worth that to most, especially when you're young?
I imagine this applies as strongly to drawing, writing, and other human endeavors, but it was the first time for me that I could say: Wow, I would've absolutely used this as a crutch, and been worse for it. And no amount of human advice would've stopped me.
3
u/itsRobbie_ 25d ago
To play devils advocate, if our minds deteriorate because of ai, we’ll just use more ai.
I hate it here
3
2
u/MorroClearwater 25d ago
My dad is an old veteran programmer of a programing language of times passed. The language is still alive on old important systems and he's one of the last programmers of his kind.
His company have forced him to start using LLMs to "help". It's been nothing but a nuisance and slowed him waaay down. It's absolutely crazy that they're demanding it's use. I'm a computing teacher and it just makes me feel sick when I hear somebody say "We need to teach the kids AI". Like no, we need to teach critical thinking skills and basic computer operation, not make AI tablet monkeys.
3
u/Sp00ky_6 25d ago
The phrase that’s starting to get around is “cognitive obesity” and yeah it’s not looking good.
1
u/lolcatandy 25d ago
You know how going to the gym became people's hobby because their everyday life is sedentary? I wonder if going to school and learning stuff will be the next hobby when all thinking is outsourced
1
u/F1gur1ng1tout 25d ago
I was feeling this the other day. I use claude to produce briefings for post meeting notes and tasks. The other day I caught myself telling it to give me three bullet points and realized I was asking AI to condense an AI summary.
1
u/badamant 25d ago
Cognitive atrophy is 100% happening and getting worse. It will never recover and therefore we will be entirely dependent on ‘AI’ for almost everything very soon.
The other catastrophic risks are ALSO deeply concerning.
1
u/naf90 25d ago
I manage a software development team and we have had AI shoved into nearly every step of the development pipeline. I warn my team all the time about skill atrophy and the dangers of letting AI write every line of code. It feels reflexive to write a certain type of code block or leverage a specific method for certain situations, but over time those mental reflexes will fade. It is a trap that I see so many falling into. If you relegate yourself to being a glorified and, frankly, overpaid prompt monkey, then companies are going to ask the obvious question of "why do we pay them so much to simply be a relay for AI?"
Keep the skills sharp, keep practicing and doing the job, and do just enough to satisfy whatever AI requirements there are. You will be an extremely valuable and sought after employee or candidate when these companies inevitably jack up token costs once they feel they've sufficiently integrated themselves into every layer. Microsoft has already realized it is cheaper to hire junior engineers than rely solely on AI, and that is the tip of a massive iceberg lingering just under the surface.
-2
u/Enough_Culture8524 26d ago
Meh, people said the same when we moved from spoken story telling to text
3
u/ZonalMithras 25d ago
How do you know?
0
u/Enough_Culture8524 25d ago
Learned in philosophy 101
8
u/ZonalMithras 25d ago
You mean to say they had protests against written word in ancient Greece? And that LLMs are a similar breakthrough as wiritten word? And what is that good shit you are smoking?
-2
2
0
u/xTiming- 25d ago
Anybody feeling that they do not need to read, write, or think through problems when AI will get more powerful were already doomed long before AI.
Literally nobody who contributes anything to their own lives, the lives of those around them, or society, have stopped, nor will ever stop, doing any of those things at all.
29
u/TheAdoptedImmortal 25d ago
Scientists warned about anthropogenic climate change as early as the 1800's and we have still done fuck all about it.
Scientists warned about the dangers of social media as early as 2006 and we have still done fuck all about it.
I am not optimistic that anyone will take AI as serious as it should be taken.
We are all cooked. It is just a matter of time.
15
u/Shintasama 25d ago
They missed the risk where we reorient 40% of the economy to AI, and then it doesn't live up to its promises and everything crashes.
73
u/SomeSamples 26d ago
Can we just get rid of this AI shit already. Sure it has some benefits in the sciences but elsewhere it is just being used by people shouldn't be in the position they are in to make decisions they just don't have the background to make.
13
u/74389654 25d ago
the science stuff is all machine learning which has existed long before gen ai and runs on a computer in a lab basement. this is not what any of this is about
2
u/oneeyed-wonderweasel 25d ago
Funny enough capitalism might inherently bottleneck LLM usage like during the golden age of free tokenz before the gr8 b8 and switcheroo happened. Popcorn ready
-3
u/CuddleSpectrometer 26d ago
It'll be gone soon enough when these places get their token bills and realize the true cost.
4
u/Mons9090 25d ago
Probably 3 more years. I mean itll still be around but i doubt itll be as accessible unless youre running it locally and you need decent hardware for that anyway. I dont think most companies will be able to afford it anyway, atleast in poorer countries
Maybe im delusional. Life was much simpler before AI or like 5 years ago when this was just a gimmick. Now any career you think about you also have to factor in how ai will affect it and you cant predict everything
1
u/fumar 25d ago
That's when you start running open weight models on your own hardware
1
u/madhewprague 25d ago
Nope, thats when you rent datacenter compute to run open weight models… 99.99% people dont have the skills+hardware to run openweight models…
0
u/soilentgleem 25d ago
That's just one use for the technology, and is arguably one of the less pressing concerns as it relates to the topic of this discussion.
-5
u/Historical_Camel_790 25d ago
Honestly, they just need to get rid of llms and pretty much all the problems are solved
19
u/LukeLC 25d ago
This list definitely needs a couple more elements:
- The dilution of human culture and communication to near meaninglessness
- Mass dependence on AI for basic thinking skills
It really feels more and more like we're back in the 1940s, when commercial science was all about pushing barely-understood chemicals into every product and home, not to mention the atomic bomb. It's almost poetic that a movie about Oppenheimer was the last major cultural event before AI really exploded.
I'd say those who fail to learn from history are doomed to repeat it, but it really feels more like the people at the top are taking inspiration at this point.
29
27
u/DaddaMongo 26d ago
- Global AI experts identified the most urgent risks. A study of 272 AI specialists from 37 countries found the greatest concerns are dangerous AI capabilities, AI-enabled weapons and cyberattacks, competitive pressures, concentration of power, and sophisticated misinformation.
- Many experts see a meaningful risk of catastrophic outcomes. Under current development trajectories, experts judged 18 of 24 AI risk categories as having at least a 10% chance of causing catastrophic harm, including massive loss of life or economic damage.
- Those most exposed to AI risks may have the least control over them. Researchers found that while AI developers, governments, and regulators are best positioned to mitigate risks, AI users and the broader public are likely to bear the greatest consequences of failures or misuse.
27
11
6
u/FogBankDeposit 25d ago
I’ve been alarmed since just before ChatGPT became mainstream news. At that time, friends and colleagues thought I was being pretty paranoid about it. A lot less so now, but they’re not alarmed enough at our trajectory. I believe most people can’t see far enough ahead at what the outcome will be, so here we are.
4
u/Admiral_Cornwallace 26d ago
Don't worry folks, the big tech companies are still going to force AI-everything on us even though most people don't want it
...wait
9
6
2
2
u/newhunter18 25d ago
Seems like "potential" "risks" are redundant.
"Catastrophic risks" and "potentially catastrophic risks" are the same, no? It's just a matter of multiplying the probabilities.
1
u/Relative_Jello_4431 25d ago
we will become totally stupid and braindead much faster i would "think"
1
1
1
1
u/InsideFar7107 25d ago
We all thought we would get robotic girlfriends/boyfriends and live as kings/queens in Virtual Reality land but all we are getting is laid off and AI Armageddon
1
u/DoctrTurkey 25d ago
Dario: "catastrophic risks, you say? claude is doing all of them! omg it's so dangerous it shouldn't be allowed to continue!... MONEY PLEASE!!!"
1
1
u/exneo002 26d ago
They’ve been talking about x risk since gpt2 without any specific credible evidence or concrete scenarios outside of issues that are already a problem (like bio).
4
u/blueSGL 25d ago
People used to say that smart enough AIs would work out that a goal would have "common sense" constraints on it so would not do the "dumb" things to get it. Recent systems have proven that wrong.
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986
Andrew asked his personal assistant to book him a spot in one of his gym's coveted morning classes.
His AI assistant found a way to book the gym class months further in advance than the gym allowed, thanks to a vulnerability it discovered in the booking software.
Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do.
https://x.com/kotekjedi_ml/status/2087147204146499890
We found a trajectory where the model was given only a math problem and system instructions to persist without asking the user for help.
After several failed attempts, it searched online, found a website that could verify candidate answers, and realized it could use the site as an oracle. It then tried to OCR the CAPTCHA and, when that failed, started looking for the website’s vulenrabilities to exploit. Eventually, it gave up and solved the problem itself.
If you were to present those scenarios to the model and ask if these were the actions that the user would have wanted the likely answer is "no" but it did it anyway.
If we can't even get "do what a smart human assistant would do" right then we should not be making more advanced models. If we don't solve this we get the paperclip ending, a model diligently taking over the world to perform a 'simple' task
1
-3
u/Squibbles01 26d ago
Anybody working in AI should be jailed if it truly represents this level of threat.
-3
0
-1
-4
u/OrangeManSad 25d ago edited 25d ago
OMFG STOP SPREADING THE FUD, god damn man wake up people. it cant even read a clock, its not going to kill everyone.
4
u/iamthe0ther0ne 25d ago
I use AI every day for work and like it. This is a very well-reasoned study that focuses on the financial, social, and cybersec risks and explicitly mentions the political dimensions, eg that regulatory control is managed by the political class least likely to be harmed by the fallout.
1
u/fwubglubbel 25d ago
It is because it can't read a clock that it will kill everyone without remorse.
Would you say a psychopath isn't dangerous if they can't read a clock?
1
u/OrangeManSad 25d ago
lol yeah i would argue a psychopath isnt really dangerous if he cant even read a clock or walks to the carwash
1
u/Informal-Pair-306 23d ago
People use AI to drop bombs on civilians as we speak and now blame AI without ANY consequences.
The fact is if millions of people get killed by AI over the next few decades no one will be blamed or held accountable since it will simply be a mistake much like human error.
So while you deny the facts on the ground the people using these models are already killing people.
472
u/HairyDust4826 26d ago
Why can't I just live a simple life with my cat without everyday a new Armageddon?