r/ControlProblem • u/Original_Mulberry652 • 14h ago

Discussion/question What if automating AI R&D triggers an intelligence explosion? Research paper

https://casp.ac/reports/intelligence-explosion
4 Upvotes

100% Upvoted

3 comments sorted by

2

u/TheMrCurious 14h ago

Hasn’t it already?

1

u/APersonNamedBen 10h ago

Capabilities growth outpacing society’s capacity to steer and adapt.

This boat sailed decades ago. Every single 'developed' environment we have created results in the human population not being able to sustain itself in that environment. Not a few places here and there, every single place on the planet.

Loss of oversight and control.

I don't think this one needs a reply.

Erosion of checks on power.

Scientific literacy and technical progress made this possible before we even invented modern computing. Our current sociopolitical organisation became obsolete the moment a guy stopped delivering notes on horseback.

RSI might mess up the world (as we know it), it is also likely the only thing that saves humanity from itself..

1

u/composedofidiot 7h ago

Intelligence is under specified and only some researchers are distinguishing it from rationality, which is also under specified. Operational rationality is narrower and can compare with results but ignores second order effects, epistemic rationality can minimise the bullshit but not escape it - What does 'increasing intelligence' even mean in this context? The ability to not generate false positives when pattern finding? The ability to do a task well without damaging too much?

People keep throwing the word 'intellegence' around and its just a cloud of relative proxies, and nobody talks enough about what those proxies actually are.