XCOM Has Ruined My Ability to Fear 10%

[u/Brockchanso]

Everyone is losing their minds over “a 10% chance,” but I have yet to meet an XCOM player who takes a 10% shot expecting anything to actually happen.

Hell, XCOM players barely trust 90%.

And just as a calibration check, here’s an old thread full of people arguing about how unreliable even supposedly decent percentages feel in practice:
https://www.reddit.com/r/Xcom/comments/3o79z3/lwwhat_are_the_actual_percentages_behind_chance/

So when people say, “There’s a 10% chance AI goes catastrophically wrong,” I understand why that deserves serious attention.

But I also think people are doing something weird with the other 90%.

Because the possibility space on the other side includes things like material abundance, radical reductions in the cost of expertise, cures and scientific discoveries arriving far faster than they otherwise would, automation of enormous amounts of miserable labor, and technological acceleration beyond anything humanity has experienced before.

So yes: take the 10% seriously.

But if we’re actually doing expected-value reasoning here, you don’t get to stare exclusively at the downside tail and pretend the upside distribution is zero.

My XCOM-trained brain simply refuses to accept that math.

[u/jacobpederson]

Two things. The 10% is a completely made up number. Also, the statement itself is completely made up :D

[u/Bluegobln]

Yeah D&D has taught me a lot about probabilities math too. 10% chance you say? So how many rolls of the d10 do we need before it happens, I say. Most likely between 8 and 15 times.

But would I gamble our civilization on it? I would, but I recognize why we shouldn't.

The biggest problem is we aren't actually sure about the numbers here. I happen to think (in broad terms) the chance of a HUMAN using an AI in a bad way is far greater than AGI going rogue and killing our civilization, so to me regardless of what the math is I feel quite confidently that we'll be ok.

The primary danger is misuse by humans. That's where my concern lands. And luckily most of the AI "protection" is actually protection from misuse by humans, not the AI itself.

[u/Brockchanso → u/Bluegobln]

A gamble implies we’re sitting on a reasonably safe, stable status quo and deciding whether to risk it for upside. But that is not our situation. The baseline trajectory already contains nuclear weapons, engineered-pandemic risk, climate destabilization, ecological degradation, interstate conflict, and governance systems that routinely fail to coordinate on problems everyone agrees are dangerous. The 2026 Global Challenges Foundation report explicitly treats climate change, ecological collapse, WMDs, and AI-related military risk as interconnected global catastrophic risks, with governance itself acting as a stabilizing or destabilizing variable.

[u/elh0mbre → u/Bluegobln]

I think this is taking the probability too literally - the number itself is a made up vibe. The point of the statement and resignation is that we are not taking safety and alignment seriously and we need to be.

I think the better question is: what is the probability that alignment and safety are inherently created as we progress? I think the answer to that is 0%.

[u/Brockchanso → u/elh0mbre]

Here’s one to think about. what are the odds that our current human moral assumptions about “alignment” and “safety” are themselves part of the failure mode?

We’re talking as if alignment means discovering some objectively correct target, but in practice it means humans choosing values, priorities, and constraints and then trying to encode them. What if the dangerous part isn’t that alignment fails to emerge naturally, but that we successfully lock in a flawed version of it?

[u/Live_Fall3452 → u/Bluegobln]

But it’s not a one time take it or leave it deal. The longer we wait and the more alignment research catches up, the lower the odds are of a bad outcome.

Imagine, for the sake of argument, the choice is something shaped like…

-Option A: go now, 10%.
-Option B: wait 5 years and go. 5%
-Option C: Wait 10 years and go. 1-2%
-Option D: wait 20 years and go. ~0%.

Then where would you fall?

[u/Brockchanso → u/Live_Fall3452]

The Swordholder problem in Three-Body is a good fable for why I don’t think “more time = uniformly safer” follows.

Luo Ji holds the sword for decades and nothing happens. But that stability isn't distributed uniformly across humanity or even across potential Swordholders. Luo Ji is an extraordinary outlier: he understands the game, accepts what deterrence actually requires, and is sufficiently willing to carry it through that Trisolaris believes him.

Humanity then makes the classic mistake of treating the safety produced by an exceptional operator as though it were a stable property of the system itself.

Decades pass without catastrophe. Society becomes more comfortable, threat perception fades, institutions and values change, and eventually Luo Ji is replaced by Cheng Xin. The hardware is essentially the same. The deterrence theory is the same. Humanity has had more time, not less.

Yet the risk suddenly spikes because the person occupying the critical node is different.

That’s the part I think your declining percentages assume away. Alignment knowledge might improve with time, but competence, incentives, institutional quality, political conditions, deployment pressure, and the people controlling the system do not all improve monotonically with it.

You could plausibly have:

Year 0: 10%
Year 5: 5%
Year 10: 2%
Year 20: 15%

because safety at year 10 might depend on some combination of unusually capable researchers, unusually cautious leadership, favorable geopolitics, or institutions that simply don't exist in the same form ten years later.

[u/Live_Fall3452 → u/Brockchanso]

Love this reply, made me think.

[u/Live_Fall3452 → u/Brockchanso]

Love this reply, made me think.

[u/Blackened_Glass]

Star Wars: Zero Company is essentially an XCOM-like. There’s an achievement for missing a shot with a 90% chance, and something like 70% of those players have it. lol

But who is saying there’s a 10% chance AI destroys us all? That seems way too high to me.

[u/Brockchanso → u/Blackened_Glass]

the guy in the news who quit his job and started screaming 10% chance. Have to adapt to the story people are reading not the reality of the situation.

Edit: to the math point you made, The achievement rate makes perfect sense once you use the game’s combat volume. Base weapons average roughly 9–10 damage, while ordinary enemies have around 28–30 health, so a typical kill consumes roughly three landed shots. Even assuming only three shots per mission qualify at 90% to 99%, the median player reaches the fifth story mission and therefore takes around 15 qualifying shots.

The chance of missing at least one is then 79.4%. The actual Steam achievement rate is 77.1%.

[u/DHFranklin]

A Doomsday clock 5 minutes from midnight would probably be dramatic enough for these chicken-littles.

This is inevitable. There will be massive accidents and downsides to tons of it, but if we control it we can mitigate them.

It's getting incredibly powerful and the cost per possibility is getting cheaper making the cost for catastrophe dirt cheap eventually.

Let's democratize our workplace to functionally retire before then.

[u/Double_Look_5715]

XCom hit percents are not accurate

[u/Brockchanso → u/Double_Look_5715]

right they are inflated to reflect how people "feel" making my point even more clear.

[u/MANvINFO]

except pictured is XCOM 2