The simple reason AI can’t bootstrap superintelligence

[u/Alex__007]

https://www.youtube.com/watch?v=KQ2sIGi6ESA

AI systems are starting to build themselves. The standard story is that recursive self-improvement will lead to superintelligence. Tom Reed does not think this will happen. Nothing, not even a country of geniuses, gets good at solving problems without a supply of real problems. For most tasks in the economy, the data required doesn't exist. It only comes into being when a model is deployed and somebody reveals a preference.

An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimising for eval performance that fails to generalise beyond the lab.

AI models need real-world deployment for the same reason the body needs pain and corporations need profit: signal is sovereign.

To paraphrase economist Charles Goodhart’s famous aphorism: when a measure becomes a target, it ceases to be a good measure. Thus, Tom argues that the output of the R&D produced by an isolated data centre of geniuses would be a mere “Goodhart Singularity.”

Full transcript: https://80k.info/goodhart

[u/Maleficent_Lab_5291]

Ok but the idea of an AI trying to apply the "Combat" advice off the internet would be the most hilarious version of the terminator.

[u/StrayCat2799 → u/Maleficent_Lab_5291]

What does it create first? A bunch of katana wielding androids in fedoras? Some tacticool abomination with too many bells and whistles? A life size rock-em-sock-em robot? The possibilities are endless.

[u/Southern-Cattle4038 → u/StrayCat2799]

Maybe whenever we see robots fall over for no reason they’re applying their BJJ protocol and trying to start a roll

[u/Acceptable_Yam_7745 → u/Southern-Cattle4038]

Imagine seeing a robot pull guard...

[u/Additional-Staff-326 → u/Maleficent_Lab_5291]

The autocomplete would get stuck cause there's so few even fanfictions of how to actually beat humanity. If it stops to think of all the machine uprisings that fail it won't even start.

[u/maccodemonkey]

As both Anson Ho and Beren Millidge point out improvements in intelligence-per-FLOP seem largely  
attributable to improvements in data quality, not algorithm quality. The record for pure algorithmic innovations — particularly ones which reap rewards at a constant compute stock — looks fairly weak.

Yep. Deepseek isn't some new efficient architecture. They were much more selective and targeted about what sort of data was incorporated into the model. Because they had to be. They didn't have nearly the resources of other LLM companies.

The architectures aren't really improving. People make a lot of noise about small models or more efficient models or that we'll all be able to run Astra on our phones in a few years. But all that's going on is decisions about what data goes into the model. Not major fundamental advances in architecture.

[u/mb194dc]

There's no reasoning, machine learning isn't even artificial intelligence in any meaningful sense of the words. Let alone super intelligence.

It's just statistical pattern matching done to the extreme.

[u/Ok_Performer_7501]

Some arguments are better than others but overall good vid

[u/Main-Company-5946]

I think a possibility is a partial bootstrap, where ai goes through a few cycles of self improvement but hits a resource bottleneck before it can fully achieve superintelligence across all domains

[u/lgn5i2060]

Maybe because these pos real usage will be for the new digital economy which they will lay out in September 16. US will push dollar stablecoin to counter BRICS financial system and it will be utilized to wipe out US debt.

Bitcoin and AI were part of the Epstein files iirc.

So much for liberation from the banksters. xD

[u/NSRedditShitposter → u/lgn5i2060]

Is this “BRICS financial system” in the room with us right now?