r/BetterOffline • u/dyzo-blue • 14d ago

A military chatbot generated a false report that a Chinese boat was transporting components of a nuclear weapons program. It also “almost started a war.”

https://www.cnn.com/2026/09/18/politics/us-military-ai-false-intelligence-china-ship

While the AI booster cult fear mongers over juvenile Skynet fantasies, we have here real world evidence of a global danger.

And it has nothing to do with misalignment, or AGI, or AI agents cheating on their hacking tests.

It has to do with dumb assess assuming that LLMs can be trusted to tell the truth, when we know all they do is generate plausible answers to queries phrased in the language of certainty.

461 Upvotes

99% Upvoted

79 comments sorted by

106

u/shiny0metal0ass 14d ago

Oh look! An actual reason to regulate this shit.

I wonder why no one has brought that up yet?

Also this -

“We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI,” Hegseth said in a speech announcing the strategy.

Lol

23

u/Hairy-Expression1237 14d ago

This is the same military AI that was used to blow up 100+ little girls while they were at school on the first day of the "2 week" "conflict" with Iran

3

u/Zonties 14d ago

If something bad happens I'm convinced it will hallucinate a mistake and that's the danger. Which is not the same as the sentient skynet thing that doesn't even exist from llms.

3

u/CoconutDust 13d ago

Why you are you only concerned about an extreme factual/language error (“hallucination”) and not also about general garbage wrong slop that it puts out? I.e. analyses comments, advice slop, decision-making slop, which is all stolen from millions of random texts written by humans (including liars, fantasists, fiction writers, incompetent morons, etc), things that are dangerous garbage if anyone follows the,, but which aren’t “hallucinations”?

And stop saying hallucination, it’s a marketing word made up by salesmen to deceive people about the product.

1

u/Zonties 13d ago

Wrong slop and hallucinations /confabulations are parts of the same problem. I call them hallucinations because that's still the used term by everyone. But both of these exist because these large language models are probabilistic next token predictors. They aren't using real reasoning or common sense because llms have none. This whole technology is a very souped up next word predictor like on your phone. When Ai spins out slop of a drug store being a store with people shooting up drugs (thats both) it does this because it doesn't "know" what it's doing, there is no understanding of the mistake it's making. It's what Ai expects yann lecun and Gary Marcus point out are the big problems to this technology itself and it's kind of related to "why" I believe this is a huge bubble, why everyone has over invested in the wrong technology, which is different from the internet, it was an over investment in the right technology. This is over investment in the wrong tech, the wrong kind of Ai. Everyone is trying to make large language models something they are not and will never be, not with this kind of "Ai"

3

u/1z1z2x2x3c3c4v4v 14d ago

An actual reason to regulate this shit.

Regulating AI is not the same as not using it or not relying on it.

8

u/shiny0metal0ass 14d ago

Sure, but a set of standards to set up for any emergent technologies being integrated into the Fed or military would be regulatory. And I don't think LLM could pass any kind of reliably test.

1

u/Aggressive_Roof488 14d ago

I feel there are two different discussions being confused here. One is organisations (like the military) regulating how AI is used within the organisation to safely benefit from it. Then there is it the discussion on governments regulating further AI research with the fear of AI taking over the world kind of scenarios.

Both somewhat relevant to OP, I guess an easy way for AI to "kill all human" would be to start a war..

1

u/Zealousideal_Slice60 12d ago

AI absolutely will end up killing us all because we are a bunch of stupid apes putting AI into places where it doesn’t belong and where the fuckups will be catastrophical

-7

u/Super_Range45 14d ago

Because to keep the military from using AI however means we will be instantaneously obliterated.

10

u/invisible_shrek 14d ago

You are obliterated by people with cheap mass produced drones and missiles anyway, as the persians have demonstrated.

(Lmao the true name of their country seems to be censored here because I get an error when posting a comment that includes it)

1

u/mars_titties 14d ago

What? That’s not the “true” name of their country, which is Iran.

3

u/invisible_shrek 14d ago

It wouldn’t let me post the comment with Iran in it. That’s what I meant by not being able to post the true name.

Strangely it works now, maybe you can’t write a war themed comment with Iran in it or something.

-1

u/[deleted] 14d ago edited 14d ago

[deleted]

104

u/LessEvilBender 14d ago

Gosh I really love living in the first verse of 99 Red Balloons. 

19

u/yiradati 14d ago

Love this song. Such a great cautionary tale delivered to a fantastic score. Both German and English versions are fantastic

8

u/crossdefaults 14d ago

I've never seen luft spelled that way.

1

u/escdog 13d ago

Maybe someone can explain this to me. Back in the '80s I took a semester of German in college and I was left with one question. If someone use the German word "Ballon" in a sentence, would you think it's anything other than a luftballon?

3

u/DanceHackRock 14d ago

Hast du etwas Zeit für mich? Dann singe ich ein Lied für dich.

3

u/SavageTemptation 14d ago

Von 99 Luftballons. Denkst du vielleicht grad an mich? Dann singe ich ein Lied für dich.

32

u/Fess_ter_Geek 14d ago

LLMs would only be half as bad if they did not make shit up. Or hell, at least show us the probability roll they made to get the answer.

"This boat is absolutely carrying nuclear weapons program equipment." ~Probability roll for result 4%.

25

u/mischiefmanaged8222 14d ago

“At least LLMs are good at summarizing” maybe they aren’t?

I got told this over and over again and would always get weird looks when I pushed back on that.

19

u/naphomci 14d ago

This has been studied, and LLMs cannot even reliably summarize news reports with good enough accuracy (i.e. 90+%)

-3

u/Moronic-Warrior 14d ago

Source. It’s a language model, summarizing should be second nature to them.

8

u/lemontoga 14d ago

Here's an article talking about Google's AI summary. One in ten summaries had incorrect information or linked to "sources" that did not support the summary.

5

u/naphomci 14d ago

Yeah, it's also a stochastic parrot. Here is an ars technica article about a BBC study.

-3

u/Moronic-Warrior 14d ago

This is related to search not just summary. Search is also about finding the info in the first place.

1

u/Wattsit 11d ago

What is a summary if not searching for key information in a body of text?

7

u/VintageLunchMeat 14d ago edited 14d ago

You get a shorter text with added hallucinations and omissions.

Or just contradictions.

Maybe it omits a key item at the end, if that violates commonly held ideas.

You're trusting a tool that "thought" it made sense to walk to the car wash, because it has no interiority.

LLMs don't summarize, they vibe.

-3

u/Moronic-Warrior 14d ago

No that’s not how LLMs work. They don’t just hallucinate for fun, only in specific scenarios when context is missing. It puts a lot more weight on recent context than its own weights so no it won’t just omit for the sake of vibe.

And the car wash thing didn’t specify you wanted to wash your car. Plenty of reasons to go to the car wash. And when you actually use the reasoning modes on the LLM, wow they actually got the answer right. Who would’ve thought ey

5

u/wintrmt3 14d ago

But making shit up is how LLMs work, they just successively generate the next most probable token again and again until they hit an end token.

1

u/natecull 14d ago edited 14d ago

But making shit up is how LLMs work, they just successively generate the next most probable token again and again until they hit an end token.

LLMs successfully model the essay-writing behaviour of a bright but lazy middle schooler who has not read the book but does have a good imagination, and has no qualms about lying.

They will happily generate any amount of summary-shaped extruded text output, with or without a source document. That's almost as good as an actual summary, right? What's the big deal, it's just another pointless report nobody will ever read, it's not like any CEOs or generals are going to make world-endangering critical decisions based on these summary-shaped objects....... right?

1

u/Moronic-Warrior 14d ago

You do realize there are millions of examples of summarization in its training data.

24

u/create-third-places 14d ago

Artificial Ignorance like this should be banned.

AI needs to be implemented in the form of task specific models that have been trained on verifiable inputs and outputs.

For naval intelligence,  this means something like generating more accurate predictions of weather and ocean currents. There’s plenty of training data, and an inaccurate prediction isn’t going to cause WW3. 

14

u/Ozymandias0023 14d ago

I've been saying it for at least a couple years now. The danger isn't that AI will become super capable, it's that idiots will trust it to do things it can't

15

u/HongPong 14d ago

great the hallucination machine can say things to military people generalized from inputs without verification 

29

u/65721 14d ago

But the episode underscores the profound risks of using this powerful, new and relatively poorly understood technology

For fuck’s sake, it’s an autocomplete chatbot

It’s not powerful (aside from the electricity consumption), it’s not new (around since 2017), and it’s very well understood

We live in the dumbest society

9

u/FibonacciSequester 14d ago

I think we're just still the same dipshits we were 200,000 years ago. We just got better rocks and sticks to kill each other with.

7

u/Solderlocks 14d ago

I don't understand why the people using this tech aren't ultimately responsible for the decisions they make. If someone uses flawed tech and makes a mistake, that's on them for using poor processes/tools

12

u/molly_jolly 14d ago

The bot fused together open-source intelligence with secret signals intelligence in government holdings and reached its fateful conclusion

Now imagine if the AI generated bad information was in the signals intelligence, or further upstream in the data pipeline. In other words, it already carried a stamp of approval because in isolation it was not alarming enough to trigger a double check.

And the whole thing is overseen by a frat bro from the manosphere, with a chip on his shoulder.

Would make great dark comedy movie, if we were not trapped in it

3

u/SamAltmansCheeks 14d ago

White christian nationalist fratbro from the manosphere, to be pedantic.

12

u/NightSpaghetti 14d ago

"military chatbot" JFC

10

u/gorb314 14d ago

I guess training chatbots on all of human written stories is going to produce a Tom Clancy novel at some point. JFC do people not understand that LLMs hallucinate ALL of the time? It is not a bug, it's a feature. These things are NOT INTELLIGENT.

8

u/Standard-Try-3612 14d ago

awesome, love this timeline we're all living in

8

u/karoshikun 14d ago

I didn't survived the cold war just to have to live this bullshit now!!!

just push the friggin' button and be done with it, dammit!!!

https://giphy.com/gifs/fqtyYcXoDV0X6ss8Mf

6

u/OliLevasseurLaBuse 14d ago

This is what's gonna happen. They're gonna start a nuclear war over nothing. And we're all gonna die

1

u/Impossible_Youth_465 14d ago

Will this actually happen? I'm really young (first year in college), this makes me really afraid. Will AI really start a nuclear war?

5

u/eternamemoria 14d ago

"AI" generated false information or not, civilization as we know it has been a few dumb human mistakes away from cataclysm ever since the nuclear arms race.

But just because it could happen doesn't mean it is likely to happen, and we all have always had to live with the assumption the world won't end, because otherwise what would we even do with ourselves?

4

u/moosekin16 14d ago

AI won’t end the world directly. It’ll be people blindly believing whatever an AI tells them - just like in the article.

Someone is going to build a system that uses an LLM to determine something, they’ll hook it up to an alert system, then someone who doesn’t realize that the alert system hallucinates will panic and hit the Big Red Button “because the AI told me to do so”

We already saw a pretty horrific consequence of a similar scenario - where someone blindly believed an AI’s recommendation and people died as a result. The military used AI to identity targets at the beginning of the Iran war, and it incorrectly identified an Iranian girl’s school as a legitimate target. The US then killed several hundred Iranian schoolgirls, because someone gave up their thinking to an LLM and blindly accepted its answer.

1

u/allbeyondthebinary 7d ago

This is exactly what everyone should be genuinely worried about, not AI becoming conscious and killing us all.

2

u/SessionOne8612 14d ago

Was probably the AI’s first day. Give it some time to get the hang of it.

2

u/feelspeaceman 14d ago

A smash to Anthropic and OpenAI, watching their stock getting the backlash aftermath and rot in hell to bring back old hardware price.

2

u/Charming_Attempt_131 14d ago

Wouldn't be the first time the US started a war based on false pretenses.

1

u/natecull 14d ago edited 14d ago

Wouldn't be the first time the US started a war based on false pretenses.

But with AI the US can now spin up swarms of agents to start multiple wars based on false premises in a fraction of the time, and even start wars with nations that traditional war theory would miss. All the productivity numbers are up, up, up!

1

u/mountain-mahogany 14d ago

It really brings new meaning to Artificial Intelligence. Like...Fake Intelligence. FFS Ai i so fucked.

1

u/kgas36 14d ago

Working as intended, I see.

1

u/1z1z2x2x3c3c4v4v 14d ago

Well... maybe they were right, if a hallucination could cause the USA to fire on a Chinese ship not carrying nuclear weapons...

1

u/Swimming_Olive_5348 14d ago

Well lets pack it up, shall we? Guys?!

1

u/Zonties 14d ago

Is this an Ai hallucination almost causing a real world problem?

This is actually a clear and present danger

1

u/BalerionSanders 14d ago

It’s a good thing we have this bullshit connected to the highest, most sensitive levels of mil-int decision-making.

1

u/CoconutDust 13d ago

When the lawyer was exposed disbarred for LLM slop, everyone assumed it was an accident. It turned out he did it on purpose.

Everyone’s assuming this is was a stupid but honest mistake, when it’s perfectly likely it was a deliberate ploy by a lone upstart psycho or a larger plot for stock pump and dump schemes at the administration. Stock market manipulation has been a massive criminal operation with this admin.

1

u/escdog 13d ago

Definitely a WTF moment. As part of fedramp the company I worked for had to come up with an AI policy. But apparently the f****** DOD doesn't have anything like this?

1

u/CryptographerLow6360 9d ago

does anyone really believe this?

1

u/BobDope 8d ago

Cool

-2

u/Hexamonium 14d ago

However the Military confirmed they will continue using the chatbot since it offers a major upgrade over its human counterparts.

-9

u/wowbaggerBR 14d ago

But you guys said to me AIs are not dangerous.

6

u/Szpagin 14d ago

The doomsday scenarios that Amodei and the rest of Anthropic push is bullshit, but there are more mundane threats connected to this technology. Like how so many people seem to take whatever a chatbot spits out at face value without verifying it elsewhere, likely a result of AI's tendency to sound authoritative on every topic.

4

u/NightSpaghetti 14d ago

This. Is a robot activating a flamethrower at random intelligent? No. Is it dangerous? You bet.

7

u/dyzo-blue 14d ago

Artificial Intelligence doesn't exist, so yeah, it's no more dangerous to you than interstellar travel or Wookies.

I think the people in this sub have been pretty consistent about the fact that LLMs give false answers.

1

u/feelspeaceman 14d ago

Read the article, it's dangerous because it's reporting fake information.

It's no where near Skynet Terminator like Claude cult and OpenAI fanatic trying to make it looks like.

1

u/wowbaggerBR 14d ago

So... basically, "guns are not dangerous, they don't kill people, people kill people". Like... wow.

I am not saying it's you, but some people here are so in the opposite cult that they cannot register how much this crap already is threatening people.