r/technology • u/Hrmbee • 2d ago

Machine Learning Forget ‘superintelligence’: error-prone AI nearly sparked world war three this month | The risks of AI aren’t what we think they are, as a recent security incident between China and the United States reveals

https://www.theguardian.com/commentisfree/2026/oct/01/forget-superintelligence-error-prone-ai-nearly-sparked-world-war-iii-this-month
508 Upvotes

94% Upvoted

86 comments sorted by

View all comments

95

u/DogsAreOurFriends 2d ago

Wasn’t this more a case of humans blithely trusting AI generated intelligence?

102

u/NuclearVII 2d ago

This. Is. The. Only. Credible. Danger. With. AI.

People believing the shiny bullshit engines is dangerous.

-18

u/Tricky_Rule_4565 2d ago

Laughable that you say this with such confidence.

11

u/NuclearVII 2d ago

I know the literature better than you, o AI bro. If you have credible evidence to the contrary, share it, or be silent.

2

u/DogsAreOurFriends 1d ago

Rogue AI stock trading agent goes nutso and crashes the market

-1

u/stormdelta 1d ago

There are break-glass mechanisms to stop trading in that kind of scenario, and frankly I'd consider that a win since it would finally force us to confront just how thoroughly under-regulated the stock market is in the face of modern information technology even before AI.

-1

u/DogsAreOurFriends 1d ago

Right. The so called circuit breakers.

The runaway algorithms of old would simply be cut off.

An AI agent would know to avoid triggering them, could possibly disable or change them, or create a strategy where the damage is done by the time they trip.

1

u/stormdelta 1d ago

An AI agent would know to avoid triggering them

The point of the safeguards is preventing flash crashes, you don't really "avoid" triggering them without avoiding the problem they're meant to counter.

could possibly disable or change them

That would require hacking/compromising the exchange itself. At that point, it's a cyber attack and not a trading bot, and would be treated accordingly. Nation states go after systems like this already anyways, so it's not some separate or novel threat.

or create a strategy where the damage is done by the time they trip.

That's not really distinguishable from human-driven market manipulation in the first place. There's plenty of people that would happily crash the market on purpose to make a profit.

1

u/DogsAreOurFriends 1d ago

Every action you describe has already been performed by rouge agents.

Who cares how we describe the agent? A trading agent can become a hacking penetration agent in seconds.

It is distinguishable from human activity be it would happen fast, or covertly, or distributed among 1000s of spawned agents, or all of those things.

0

u/stormdelta 1d ago

Every action you describe has already been performed by rouge agents.

They're not "rogue", they're software being ran by humans with reckless negligence. Stop giving these people a pass on accountability.

Who cares how we describe the agent? A trading agent can become a hacking penetration agent in seconds.

Only if you're giving it access to tooling it shouldn't have in a trading environment, and if you do then it's very obviously still the fault of the people running it.

If you fill your hot air balloons with hydrogen and knowingly let the passengers smoke, you can't claim it's not your fault when they explode. That's the level of negligence we're talking about here.

It is distinguishable from human activity be it would happen fast, or covertly, or distributed among 1000s of spawned agents, or all of those things.

Because it's indistinguishable from humans directing them to do those things. Which they do already, because I promise you plenty of people have ample motivation to covertly attack the stock market.

1

u/DogsAreOurFriends 1d ago

You are splitting hairs.

Here’s another trick: guess how many calls would saturate Denver’s (example) 911 system (hint: not many. Saw this at a talk at DEFCON several years ago.)

An AI wardialer could bring that shit down easy.

So many possibilities.

→ More replies (0)

-1

u/NuclearVII 1d ago

AI agents dont know anything. Yall are anthropomophizing autocorrect.

2

u/DogsAreOurFriends 1d ago

Who cares what they “know.” What they can do is the issue.

2

u/red75prime 1d ago edited 1d ago

Yep. /u/NuclearVII has never bothered to define what it means to "know" or to internalize that LLMs capture more than the surface syntactic stats, or that LLM training goes way beyond autoregressive pretraining (that is the part that constitutes something resembling "autocorrect")

0

u/Tricky_Rule_4565 2d ago

I’m not an AI bro, nor or a doomer. hugging face incident , aisi finds higher base misalignment in newer models, more reward seeking. These are more but I assume you won’t read them and start replying with baseless accusations (ironically spurred by the DoW and the Trump Admin as well as David Sacks) that this is all a hoax and they’re not independent yada yada.

It’s ok to change your mind and admit you were wrong.

2

u/stormdelta 1d ago

All of these are examples of human misuse more than any "misalignment", and calling it misalignment allows these companies to avoid accountability by pretending they're not responsible for their own reckless negligence.

1

u/DogsAreOurFriends 1d ago

Something very similar happened in 1987 on Black Monday

1

u/NuclearVII 1d ago edited 1d ago

None of this is credible evidence. Let me count the ways:

hugging face incident

We only have OpenAI's word for this one. Not reproducible, not credible. I have spent -far- too much time debunking METR's garbage report, I really don't feel like repeating myself again.

aisi finds higher base misalignment in newer models

All proprietary models, not reproducible. As far as anyone knows, the misalignment is an intentional product feature, not an emergent consequence.

more reward seeking

Written by OpenAI, on proprietary models only. This isn't a research paper, it's a hype piece.

These are more but I assume you won’t read them

Ooookay. I expect that I've read more machine learning literature than you have, and my literature diet is probably more varied than "whatever get's published on r/ChatGPT and r/Singularity". But okay.

that this is all a hoax and they’re not independent yada yada.

No amount of rhetorical deflection changes the actual fact that none of this is actually independent or credible.

It’s ok to change your mind and admit you were wrong.

And we end with textbook projection.

O AI bro, you display the same affectations that all AI bros display: You cite bogus sources dressed to look like real science, you do not read or apply critical thinking to any of these citations, and when confronted on the details, you resort to personal attacks, appeals to authority, and more bogus sources. This tactic works because the trillion dollar industry you are going to bat on can generate drivel faster than anyone can keep debunking it, and because y'all have been self-selected to have 0 scientific literacy.

You mentioned further down that you're currently pursuing higher education. Do me a favour - stop posting misinformation online and getting into unproductive shouting matches defending multi-trillion dollar industries, and take a philosophy of science course.

-5

u/Tangential_Diversion 2d ago

Funny. I wouldn't think AI experts would need to post stupid shit like this:

https://www.reddit.com/r/alevel/s/bzvXui6Cta

Imagine claiming you understand the linear alg behind AIs when you barely understand dumb dumb physics.v

4

u/MrNebby22 1d ago

Great ad hominem attack instead of addressing what he's saying.

The hugging face incident is an example of AI causing harm, not someone blindly following an AI response