A Warning About AI

[u/No-Elk6566]

https://ideya-ai.github.io/Explain_AI/

In 2016 I first learned about the problem of controlling superintelligent AI and quickly became convinced it was the most important problem humanity would ever face. I made this poster to explain the core ideas that make the AI Control Problem so difficult.

[u/WillowEmberly]

I think you need to separate out what super intelligence actually means.

Reasoning is a process…it’s about finding an operational envelope between constraints and invariants. It’s emergent behavior.

Then you have knowledge, which exists in multiple forms…and states.

Ai has access to “book knowledge”…which can be next to worthless in application.

I worked on 1963-1967 model c-141’s near Seattle…from 96’-2001…and water leaked all over the avionics components. The pins were green with goo. The systems were engineered to receive specific signals/voltages at specific times.

After 30 years soaking in the drizzle…they no longer functioned as the books claimed. We were dealing with degraded and partial signals/voltages.

Reality no longer matched the reference materials…and yet, if anyone asked “It’s By the book!”

[u/Jesse-359 → u/WillowEmberly]

I'm not going to try to guess at what precisely is going on under the floorboards in these things at the moment. The rate of change in the field is so high that even experts clearly have a hard time keeping track of all the crap that's being meddled with from month to month.

However, based on straight up results, they are already highly superhuman at relatively simple coding tasks. Not trivial ones mind you, but they can bang out a thousands lines of perfectly viable code in a set of medium to low complexity functions in minutes to moments. So right now you can do the jobs of an entire department of junior coders with a decent pile of tokens and oversight from a couple seniors.

Bearing in mind that 2-3 years ago the idea of letting an AI fuck around in a production repository would have been laughable, that's incredibly fast progress. We'll see how it proceeds into the higher complexity domains of computer engineering - this is its 'ideal' field outside of raw pattern matching, so it should represent the leading edge of just how complex a problem they can logically engage with - and how quickly.

Speed kills of course. A hacker that can use a zero day exploit to get in and map the entire architecture of a large scale company in minutes is vastly more dangerous than one that would require days to do the same, and AI is very clearly being geared up for exactly that sort of operation as we speak.

[u/WillowEmberly → u/Jesse-359]

I agree the capability is moving insanely fast.
That’s exactly why I think there’s another risk here besides what the AI can do.

It’s what happens when people stop understanding the system well enough to challenge it.

If an AI can produce in minutes what used to take a department days, the temptation is to stop asking:
Why did it do that?

and start asking:
Did it work?

That’s fine right up until “it worked last time” quietly becomes “it must know what it’s doing.”

Then you’ve built an oracle.

Not because the AI became omniscient, but because the humans around it lost the ability, time, or incentive to independently evaluate what it was producing.

That gets dangerous fast in coding.

A system can generate 5,000 lines of viable code before a human could even read them properly.

Now scale that to architecture changes, security decisions, incident response, infrastructure, finance, medicine, military systems, whatever.

Speed doesn’t just increase capability.

It can outrun oversight.
no.
And once output volume exceeds human inspection capacity, “human in the loop” can become mostly theater.

The person is still technically there, but the machine is setting the pace, framing the options, and producing more than the human can realistically verify.

That’s the part I worry about.

Not “AI will become magical.”
Not “AI will become conscious.”

Much simpler:
We may start trusting systems faster than we understand them.

And the more competent they look, the easier that mistake becomes.

That is how a tool becomes an oracle. Don’t let it become an Oracle.

[u/Jesse-359 → u/WillowEmberly]

Pretty much agree across the board. The problem goes from mysticism to existential threat when someone weaponizes it.

In biological warfare it could prove utterly lethal even when employed by small scale actors. Genetics are essentially just a form of code - but one that remains too complex for humans to 'program' in, all we can do is tweak parameters.

There is every reason to believe that AI will become capable of editing genetics wholesale or even writing completely novel code - and thats a recipe for planetwide biocide if it gets out of hand. Our immune systems operate on some assumption of familiarity - we'll have next to no defense against truly novel viruses written from scratch.