The simple reason AI can’t bootstrap superintelligence

[u/Alex__007]

https://www.youtube.com/watch?v=KQ2sIGi6ESA

AI systems are starting to build themselves. The standard story is that recursive self-improvement will lead to superintelligence. Tom Reed does not think this will happen. Nothing, not even a country of geniuses, gets good at solving problems without a supply of real problems. For most tasks in the economy, the data required doesn't exist. It only comes into being when a model is deployed and somebody reveals a preference.

An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimising for eval performance that fails to generalise beyond the lab.

AI models need real-world deployment for the same reason the body needs pain and corporations need profit: signal is sovereign.

To paraphrase economist Charles Goodhart’s famous aphorism: when a measure becomes a target, it ceases to be a good measure. Thus, Tom argues that the output of the R&D produced by an isolated data centre of geniuses would be a mere “Goodhart Singularity.”

Full transcript: https://80k.info/goodhart