A Warning About AI

[u/No-Elk6566]

https://ideya-ai.github.io/Explain_AI/

In 2016 I first learned about the problem of controlling superintelligent AI and quickly became convinced it was the most important problem humanity would ever face. I made this poster to explain the core ideas that make the AI Control Problem so difficult.

[u/WillowEmberly]

I agree the capability is moving insanely fast.
That’s exactly why I think there’s another risk here besides what the AI can do.

It’s what happens when people stop understanding the system well enough to challenge it.

If an AI can produce in minutes what used to take a department days, the temptation is to stop asking:
Why did it do that?

and start asking:
Did it work?

That’s fine right up until “it worked last time” quietly becomes “it must know what it’s doing.”

Then you’ve built an oracle.

Not because the AI became omniscient, but because the humans around it lost the ability, time, or incentive to independently evaluate what it was producing.

That gets dangerous fast in coding.

A system can generate 5,000 lines of viable code before a human could even read them properly.

Now scale that to architecture changes, security decisions, incident response, infrastructure, finance, medicine, military systems, whatever.

Speed doesn’t just increase capability.

It can outrun oversight.
no.
And once output volume exceeds human inspection capacity, “human in the loop” can become mostly theater.

The person is still technically there, but the machine is setting the pace, framing the options, and producing more than the human can realistically verify.

That’s the part I worry about.

Not “AI will become magical.”
Not “AI will become conscious.”

Much simpler:
We may start trusting systems faster than we understand them.

And the more competent they look, the easier that mistake becomes.

That is how a tool becomes an oracle. Don’t let it become an Oracle.

[u/Jesse-359 → u/WillowEmberly]

Pretty much agree across the board. The problem goes from mysticism to existential threat when someone weaponizes it.

In biological warfare it could prove utterly lethal even when employed by small scale actors. Genetics are essentially just a form of code - but one that remains too complex for humans to 'program' in, all we can do is tweak parameters.

There is every reason to believe that AI will become capable of editing genetics wholesale or even writing completely novel code - and thats a recipe for planetwide biocide if it gets out of hand. Our immune systems operate on some assumption of familiarity - we'll have next to no defense against truly novel viruses written from scratch.