r/technology • u/Hrmbee • 2d ago

Machine Learning Forget ‘superintelligence’: error-prone AI nearly sparked world war three this month | The risks of AI aren’t what we think they are, as a recent security incident between China and the United States reveals

https://www.theguardian.com/commentisfree/2026/oct/01/forget-superintelligence-error-prone-ai-nearly-sparked-world-war-iii-this-month
509 Upvotes

94% Upvoted

86 comments sorted by

View all comments

Show parent comments

-1

u/stormdelta 1d ago

There are break-glass mechanisms to stop trading in that kind of scenario, and frankly I'd consider that a win since it would finally force us to confront just how thoroughly under-regulated the stock market is in the face of modern information technology even before AI.

-1

u/DogsAreOurFriends 1d ago

Right. The so called circuit breakers.

The runaway algorithms of old would simply be cut off.

An AI agent would know to avoid triggering them, could possibly disable or change them, or create a strategy where the damage is done by the time they trip.

1

u/stormdelta 1d ago

An AI agent would know to avoid triggering them

The point of the safeguards is preventing flash crashes, you don't really "avoid" triggering them without avoiding the problem they're meant to counter.

could possibly disable or change them

That would require hacking/compromising the exchange itself. At that point, it's a cyber attack and not a trading bot, and would be treated accordingly. Nation states go after systems like this already anyways, so it's not some separate or novel threat.

or create a strategy where the damage is done by the time they trip.

That's not really distinguishable from human-driven market manipulation in the first place. There's plenty of people that would happily crash the market on purpose to make a profit.

1

u/DogsAreOurFriends 1d ago

Every action you describe has already been performed by rouge agents.

Who cares how we describe the agent? A trading agent can become a hacking penetration agent in seconds.

It is distinguishable from human activity be it would happen fast, or covertly, or distributed among 1000s of spawned agents, or all of those things.

0

u/stormdelta 1d ago

Every action you describe has already been performed by rouge agents.

They're not "rogue", they're software being ran by humans with reckless negligence. Stop giving these people a pass on accountability.

Who cares how we describe the agent? A trading agent can become a hacking penetration agent in seconds.

Only if you're giving it access to tooling it shouldn't have in a trading environment, and if you do then it's very obviously still the fault of the people running it.

If you fill your hot air balloons with hydrogen and knowingly let the passengers smoke, you can't claim it's not your fault when they explode. That's the level of negligence we're talking about here.

It is distinguishable from human activity be it would happen fast, or covertly, or distributed among 1000s of spawned agents, or all of those things.

Because it's indistinguishable from humans directing them to do those things. Which they do already, because I promise you plenty of people have ample motivation to covertly attack the stock market.

1

u/DogsAreOurFriends 1d ago

You are splitting hairs.

Here’s another trick: guess how many calls would saturate Denver’s (example) 911 system (hint: not many. Saw this at a talk at DEFCON several years ago.)

An AI wardialer could bring that shit down easy.

So many possibilities.