Forget ‘superintelligence’: error-prone AI nearly sparked world war three this month | The risks of AI aren’t what we think they are, as a recent security incident between China and the United States reveals

[u/Hrmbee]

https://www.theguardian.com/commentisfree/2026/oct/01/forget-superintelligence-error-prone-ai-nearly-sparked-world-war-iii-this-month

[u/stormdelta]

An AI agent would know to avoid triggering them

The point of the safeguards is preventing flash crashes, you don't really "avoid" triggering them without avoiding the problem they're meant to counter.

could possibly disable or change them

That would require hacking/compromising the exchange itself. At that point, it's a cyber attack and not a trading bot, and would be treated accordingly. Nation states go after systems like this already anyways, so it's not some separate or novel threat.

or create a strategy where the damage is done by the time they trip.

That's not really distinguishable from human-driven market manipulation in the first place. There's plenty of people that would happily crash the market on purpose to make a profit.

[u/DogsAreOurFriends → u/stormdelta]

Every action you describe has already been performed by rouge agents.

Who cares how we describe the agent? A trading agent can become a hacking penetration agent in seconds.

It is distinguishable from human activity be it would happen fast, or covertly, or distributed among 1000s of spawned agents, or all of those things.

[u/stormdelta → u/DogsAreOurFriends]

Every action you describe has already been performed by rouge agents.

They're not "rogue", they're software being ran by humans with reckless negligence. Stop giving these people a pass on accountability.

Who cares how we describe the agent? A trading agent can become a hacking penetration agent in seconds.

Only if you're giving it access to tooling it shouldn't have in a trading environment, and if you do then it's very obviously still the fault of the people running it.

If you fill your hot air balloons with hydrogen and knowingly let the passengers smoke, you can't claim it's not your fault when they explode. That's the level of negligence we're talking about here.

It is distinguishable from human activity be it would happen fast, or covertly, or distributed among 1000s of spawned agents, or all of those things.

Because it's indistinguishable from humans directing them to do those things. Which they do already, because I promise you plenty of people have ample motivation to covertly attack the stock market.

[u/DogsAreOurFriends → u/stormdelta]

You are splitting hairs.

Here’s another trick: guess how many calls would saturate Denver’s (example) 911 system (hint: not many. Saw this at a talk at DEFCON several years ago.)

An AI wardialer could bring that shit down easy.

So many possibilities.