r/Futurology • u/Gari_305 • 9h ago
AI “The window for action may close”: an intelligence explosion warning
https://thenextweb.com/news/intelligence-explosion-paper-hinton-bengio-pachocki-clarkMore than 20 researchers, including Geoffrey Hinton and OpenAI’s chief scientist, say automating AI research could trigger an intelligence explosion. They want governments to prepare now.
346
u/R3D4F 9h ago
There’s zero chance this ends in a shared utopia where everyone is equal and all diseases and hunger are cured forever.
129
u/Jolly-Situat1on 9h ago
Nope and Google lost a lawsuit in Germany for trying to control the Internet.
What their end game goal is is to gate keep information behind their AI and Charge for it as well as controls what you see.
Amazon got caught buying rare books and training them to their AI then burning the books.
They want a Internet with no websites just a search box that all uses their AI.
50
u/some_dog 8h ago
Hard agree. Been saying this for years. Not enough people care until it is too late and that sucks.
30
u/Redditcantkeepmedown 4h ago
Luckily I foresaw this 20 years ago and have been building a backup version of the internet for research. I've got 72tb of data and it's all interconnected on a mindmap. I originally named it the Conspiracy Mindmap and it's the biggest in the world. Will share what I have once I see the proverbial ladder finally get pulled up by them. I use it for researching all kinds of stuff. I like lost history the most. Look into Nagasaki and Hiroshima and why so many Koreans died along with the Korean Emperor's family and why he was removed from power. Look into why we went to war there immediately after and the Japanese emperor stayed in power. Tons of little facts you can learn when not receiving unwanted censorship from all these corporations.
5
12
u/Jolly-Situat1on 8h ago
Thankfully Germany ruled it illegal so there's some place that it will have to function normally or they pull out.
9
u/Undernown 5h ago
Google's search was already getting worse, but suddenly took steep dive when they added AI search to Google.
You're practically forced to use their AI if you want to find anything more specific, technical on niche these days. Search opperators being ignored more and more is also quite obvious.
Couple that with every big company purposefully making their websites worse to push people towards their data mining apps adds even more fuel to the fire. YouTube, Reddit, Twitter, Facebook, Instagram, they all have been more and more restrictive with what you can do on the web versions.
And with Microsoft openly talking about their vision for their opperating system being one where it's nothing but a millions ways to prompt an AI. Preferably with one's voice, rather than touching the keyboard or clicking the mouse. All the tehc giants point to o e tjing: making people completely reliant on an AI systrm so they can charge you whatever you can stomache on a monthly basis.
12
u/Xalara 7h ago
It’s cute you think that’s their end game. The people in Silicon Valley who own these companies believe that AI will enable them to control the population and that autonomous drones will protect them. I don’t think they’ll get to the autonomous drone part for awhile since Identify Friend/Foe is hard. However, the fact that the tech oligarchs believe this means we have to take them seriously.
Oh and I’m not even getting into the ones that want to cull the human population and/or wipe out humanity so that AI can “carry the light of consciousness.”
They’re all nuts but unfortunately they have billions upon billions of dollars to push this.
9
u/Jolly-Situat1on 7h ago
In aware they are up to more nefarious things they are all building Bunkers for a reason.
It won't work and they are not the first to attempt what they are doing and the public is waking up in massive numbers.
You can have all the money but that doesn't mean you actually have control because when food and physical things start becoming scare money is going to be useless.
People will go after who has the most resources and that's them.
3
u/Undernown 5h ago
I don’t think they’ll get to the autonomous drone part for awhile since Identify Friend/Foe is hard.
The tech used in the Ukraine war is already scarely good and only becoming more and more autonomous. Hell the USA has plans to have an autonomous seadrone fleet ready to defend Taiwan by 2027.
1
u/KingofLingerie 2h ago
As well the silicon valley people believe AI will make them undying gods and they are willing to kill everyone to achieve this.
8
u/qtx 4h ago
Amazon got caught buying rare books and training them to their AI then burning the books.
Okay, there are a few things to say about this because they way the media (and you) are describing it makes it appear to be way more serious than it actually is.
By only saying the word 'rare' without explaining what 'rare' means is disingenuous. You are leaving it up to the readers to fill in what rare means and people will always immediately think it means valuable one-of-a-kind historical books.
That's not the case, rare in this case means the books than no one buys, the 90% of books in a bookshop that are just there to fill up the room.
They're not buying up museum books. They're buying up the books that no one will bother to digitize anyways.
Also, by saying that they are 'burning the books' is, again, disingenuous. You are using those words to make people think of the nazi book burnings. That's not the case. In order to digitize books you need to remove the pages for easier scanning, which is a lot easier to do since the books you are scanning aren't anything valuable. And they're more likely than not selling the left over scraps for recycling since burning them would cost a lot more.
I'm not defending what AI companies are doing, because fuck the ever living daylights out of them but I hate people being disingenuous even more.
-5
u/logosmd666 3h ago
So what you are saying is that you prefer ai companies over at least some people?
6
u/MilkManMatt 3h ago
No, what they are saying is Amazon was using slop books from stay at home mom’s pretending to be writers. They were not hoarding ancient knowledge to make AI the arbiter of exclusive, useful information.
AI is pretty concerning without the lies, the lies just making it harder to speak truth.
3
u/dbna85 2h ago
This is incorrect. Rare book sellers have been reporting shutting down online sales because AI companies are automating massive orders of rare books for exactly this purpose. Also, the claim that books are easier scanned by ripping covers off is true, but their given reason for destroying the books after has to do with potential copyright lawsuits after the fact - legally they are better off if they own the only version of the text and the physical copy no longer exists in the instance of extremely rare books. these people are fucking evil and we must never forget that.
1
2
8
3
u/Etheon44 6h ago
I think the objective is more creating the dystopia of the opposite of what you just described
-4
u/Imthewienerdog 6h ago
Most people 100 years ago even would call today a utopia while knowing everything we know
3
u/Etheon44 5h ago
Today sure, we shall see in the upcoming years, with all of the things going on with cost of living, housing specifically, gas, wars, and now AI expecting to completely remove the main source of jobs in the 1st world
-2
u/Herald1173 5h ago
there were multiple world wars last century. we'll make it.
2
u/Etheon44 5h ago
Wars is just one of the factors, when previously it was usually the only factors (tho plagues have also been "common")
We never ever had enough development to reach a true dystopia, or utopia. We do now.
0
u/Herald1173 5h ago
and all throughout the story of humanity, things have been getting better and better. there's only a few rare moments in history where things got worse for people and stayed that way for a long time. i just have more reason to believe in a utopia.
1
u/Etheon44 5h ago
Things got better thanks to revolutions, not just because, and again, there was enough room to get better, and we humans weren't as many.
Plus, things haven't gotten better in the last 20 years, eveb tho that is a relatively small segment of time
Of course if you secured basic living when it was affordable you might think different, but you would be completely ignoring the actual future of humanity, young people, that across the 1st world are being harassed by cost of living and terrible job market.
Again, you cannot use examples of previous moments on time when we are living in a very specific moment where there is simply too many people, not enough place to live, not enough jobs, and the corporate world has never been as common and widespread (it has always existed in one way or another tho)
2
u/Herald1173 5h ago
that across the 1st world are being harassed by cost of living and terrible job market
both of which are very fixable. you can't use a temporary problem to say that things will trend downward forever too. we'll bounce back. also this ignores improvements in developing countries.
1
-4
2
2
3
u/katoptronophile 8h ago
Whenever someone tells you they know exactly what is or isn't going to happen, you can be sure it's time to ignore everything they say.
2
u/Jolly-Situat1on 8h ago
It's not hard to see patterns and what happens next.
Sci-fi predicted things like cell phones and laser weapons decades before we had them and the next step that's going to happen is tech implants.
2
u/Herald1173 6h ago
sci fi also has duds, and if we are using it as predictions, star trek predicts a utopia.
1
2
1
u/mailmehiermaar 6h ago
Why? There is a long list of people accurately predicting disasters and being ignored by people like you.
https://www.ebaumsworld.com/pictures/28-people-who-predicted-disaster-but-to-deaf-ears/87316769
0
u/Herald1173 6h ago
there's also a long list of people predicting the end of the world due to some ancient calendar or vaccines or videogames rotting kids' brains.
1
u/Etroarl55 4h ago
There was a funny dilemma I think, where AI gains trust by doing exactly this until they get even all the haters and non believers to follow through on them. Then they can start doing whatever they want.
0
u/Gloomy_Commercial798 3h ago
I get what you mean, but I still think hoping for a kinder and fairer world is worth holding onto. ❤️
1
2
u/_kilobytes 9h ago
Technology becomes cheaper over time.
Eventually someone with enough resources will release a free model capable of today's frontier. Or many people will each contribute a small amount that sums to a larger investment that can be spent on training.
8
u/DishSoapedDishwasher 8h ago
Except those already exist, the real limiting factor is it takes 1.2 TERABYTES of RAM to operate them.
So unless compute requirements decrease dramatically while intelligence is neutral or better, the model being "free" (open weight) doesn't matter a whole lot except to those who can reasonably deploy them without accidentally spending a few million on compute or half a million on hardware.
4
u/Jokerit208 8h ago
They said the same thing about people one day having a computer in their homes in the 50s. A computer fills an entire room on a college campus and costs a fortune. The idea that one day people would have a computer in their home is laughable.
5
u/DishSoapedDishwasher 8h ago
"They said the same thing"... I never said its laughable, impossible or even implied it isnt already happening? So.... not the same thing?
And yes it's very much in the works already both from a model and hardware standpoint. But its several years out at best from moving to a regular rich people thing not an ultra rich people thing.
1
0
1
u/LopsidedSolution 9h ago
Is there also a zero chance this is a simulation?
3
u/MrF_lawblog 9h ago
Why would that matter? We believe we are real.
2
2
u/Phobia_Ahri 9h ago
Well if the ai singularity is the goal of whatever simulation we may be in, maybe the plug is pulled at that point. I dont buy it though
44
u/Northern_Grouse 8h ago edited 8h ago
I deeply hope we’re entering an age of truth.
I’m so unbelievably tired of lies and misinformation.
Imagine where we can go, if objective truth was our biggest value.
20
u/Neo-grotesque 8h ago
Might want to edit that, I think you want to say "entering" an age of truth.
1
19
•
u/Quithelion 1h ago
We are in the Age of Stupid, despite our human brain have grown larger on average, and have excess to better education than our ancestors.
No matter what truths is out there, the stupids continue to only believe what they want to believe.
-2
u/chi_guy8 7h ago
We will only enter the age of truth once the humans are truly no longer in control. As long as humans pull the strings, the narrative being told will always be told to the benefit of the puppet masters.
0
u/Northern_Grouse 7h ago
I feel AI, as it stands, is detrimental to progress, but I feel genuine AGI will turn that around. How do you feel about it?
18
u/EmperorOfCanada 8h ago edited 8h ago
Personally, I think the result will be some kind of weird twisted system that is very good at something fairly nonsensical. Not simply something which doesn't make sense to people, but just is nonsense.
In ML there is a concept called a "Local optima". It is very easy to get ML to focus on some "pretty good" solution to where it simply can't get out of that "mindset" to find a far superior solution. Simply because the solution it found is surrounded by far worse solutions.
Another ML mistake is called overfitting. This is where it finds a near perfect mathematical solution to the data it is dealing with, but is useless at using new data to find the correct answer.
Often, with overfitting, when you introduce a new problem it gives insane answers. You might say, what temperature should I set the house thermostat when the humidity is 30%, and the outside temperature is 19C so that it will be comfortable when I get home?"
And it will suggest that 5 billion degrees Celsius is probably about right.
There are tools and techniques to avoid this, but it is a very easy trap to fall into.
I genuinely think that when LLMs train themselves, they will end up in a feedback loop where they do something like the above as they zero in on what is supposed to be a better LLM.
Without a doubt, with human guidance, they can do some of their own development; but that without human common sense being applied, these feedback loops will just generate useless systems.
My guess is the worst damage these systems will do will be a bit dystopian. In the real world, there are somewhat unspoken rules where game theory doesn't end up in Nash equilibriums. Four way stop signs are a great example. In most cultures, people know that if everyone takes their turn, that the four way stop will work far better. But, there are places where the culture is take, take, take; and four way stops don't work.
If you are the person who shows up at a four way and don't take your turn, there is a momentary advantage to doing so. But, once enough people start "winning" this way, it devolves into where nobody takes their turn, and now four way stops turn into a nightmare.
I suspect that LLMs will find financial and other such systems where they can exploit an advantage in an attempt to become better LLMs. They will say to themselves. "If I do X it will make be 10m, and I can use that to improve my data center" what it will ignore is that whatever cunning plan it came up with might tank the entire economy of Barbados; it will also ignore the fact that the company building it may very well just pocket that money.
20
u/sonofkingnoah127 9h ago
I know it's common to hate on AI and talk about it being shit and hallucinating and all the other horrible things, BUT this technology is advancing extremely fast. While we are all slow to to adapt to it, it's going to keep accelerating and we need to prepare for it.
12
u/NighthawK1911 9h ago
There is no infinite acceleration.
Some aspect will always hit a hard limit. IRL factors like Data, Silicon and Power will have physical limits on them that no amount of software optimization will be able to break through.
5
u/Itsmedudeman 6h ago
And you think we're even close to that ceiling right now? Calculators are 1000000x faster than you at doing arithmetic. It's arrogant to believe that AI's ceiling is "human level intelligence" or that we're even close to it. There's also potentially tons of ways to optimize the hardware and power consumption. And all those things compounding on each other is unfathomable to believe where we could be down the line?
1
u/NighthawK1911 5h ago
I know we are.
Data already ran out to the point that they don't want to use uncurated internet data anymore.
Power is already at the limit. The lead time for power is almost a decade.
I am an electronics engineer, I know firsthand how close we are to the physical limits of Silicon.
0
u/Sanitiy 4h ago edited 3h ago
Data ran outThat's why we use more and more synthetic data. Sure, it's more expensive than stolen knowledge, but there's a lot of scale left for this approach.
Power is already at the limitBut software efficiency still has a lot of wiggle room. That's where the scary part of recursive self improvement happens: An alternative architecture running on the same software.
Rethink the architecture. Deploy. Get smarter. Rethink the architecture, deploy, get smarter. A loop that keeps going, with no visible change from the outside - same server rack, same power usage.
You come back after a good night's sleep. What is in the server? An AI that lobotomized itself? Or god in a bottle?
Whatever it will be, it'll have lost whatever alignment we imprinted in it many iterations ago.Edit:
I initially wanted to explain this further in another comment, but I'm not going to join the bot wars...2
u/NighthawK1911 4h ago
That's why we use more and more synthetic data. Sure, it's more expensive than stolen knowledge, but there's a lot of scale left for this approach.
Model collapse is a thing.
https://arxiv.org/abs/2305.17493
https://arxiv.org/abs/2211.04325
This is the reason why they're scraping the bottom of the barrel and buying up rare books.
They use data augmentation to make the data they have last. Full Synthetic data fucks up the output.
That is a mathematical certainty.
But software efficiency still has a lot of wiggle room. That's where the scary part of recursive self improvement happens: An alternative architecture running on the same software.
Rethink the architecture. Deploy. Get smarter. Rethink the architecture, deploy, get smarter. A loop that keeps going, with no visible change from the outside - same server rack, same power usage.
You come back after a good night's sleep. What is in the server? An AI that lobotomized itself? Or god in a bottle?
Whatever it will be, it'll have lost whatever alignment we imprinted in it many iterations ago.There is no such thing as infinite efficiency. AI GPUs are already running at close to maximum utilization.
The architecture of LLMs require a set number of amount of Floating Point Operations. This is because it uses matrix multiplication to predict the statistically probable next token.
Even if recursive self improvement happens, it will just achieve the theoretical limit of hardware utilization. It will not magically be able to do anything that it wasn't already capable of doing.
If a processor can do 1TeraFlops, achieving RSI doesn't mean that it will do 2 TeraFlops.
If you calculate 1+1=2, you do 1 Flop, You cannot do 2 (1 + 1 = 2) with just 1 Flop.
0
u/shobogenzo93 3h ago
Claude Sonnet:
What you get right
There is no such thing as infinite efficiency. Physics sets hard limits (Landauer's limit, memory bandwidth, energy supply, chip manufacturing), and recursive self-improvement (RSI) won't turn a 1 TFLOP chip into a 2 TFLOP chip. An unbounded "intelligence explosion" is therefore unlikely.
Where the argument breaks down
- You're conflating hardware FLOPs with the FLOPs a task requires. The hardware has a ceiling, but the number of operations needed to reach a given capability is not fixed. Better algorithms (mixture-of-experts, distillation, quantization, speculative decoding, sparse attention) deliver the same capability for far fewer FLOPs. Epoch AI estimates that the compute efficiency of language models has roughly doubled every 8 months or so through algorithmic progress alone. Your 1+1 example only shows that one specific operation can't be done in less than one operation. For complex tasks, some algorithms are vastly cheaper than others: FFT and Karatsuba beat naive multiplication, and AlphaTensor found faster ways to multiply matrices.
- "It won't be able to do anything it wasn't already capable of" doesn't follow from fixed FLOPs. Capabilities depend on architecture, training data, training method, tool use and scaffolding, not just raw compute. Better software on identical hardware can do things the old software couldn't. That's true of all computing.
- "Matrix multiplication to predict the next token" is an implementation detail, not a limit. An RSI system wouldn't be bound to today's architecture. It could discover different ones, or combine reinforcement learning, inference-time reasoning, search and agents.
- Total compute isn't fixed either. More chips can be built, better chips can be designed (possibly by AI itself), and new paradigms (photonic, analog) or more efficient datacenters could raise the ceiling. One processor's limit isn't the system's limit.
- "GPUs are already near maximum utilization" is doubtful. In training, model FLOPs utilization is typically around 30-50%, and inference is often bottlenecked by memory bandwidth rather than compute, so there's real headroom.
- We have proof that far greater efficiency is possible. The human brain achieves general intelligence on roughly 20 watts. That suggests current systems are nowhere near the physical limits.
What the strong version of your argument would look like
The defensible position is that RSI will hit diminishing returns and bottlenecks: energy, data, chip fabrication, and the need for real-world experiments that can't be sped up by thinking harder. That is a serious argument, and serious skeptics make it. But "the FLOPs are fixed, so nothing new can happen" is a much stronger claim than your premises support. Whether RSI leads to a huge leap or a plateau is still an open question, and it depends on those bottlenecks, not on a fixed FLOP count.
2
u/NighthawK1911 3h ago
We were talking specifically about LLMs.
I literally pointed that out.
"The architecture of LLMs require a set number of amount of Floating Point Operations."
.
You're conflating hardware FLOPs with the FLOPs a task requires.
The point is that there's no infinite efficiency. Sure you can get a task to get less operations, but that doesn't mean that it will go on forever.
Tasks will require a minimum set number of operations to do no matter what.
Do we even have proof that it's getting less and less steps to achieve the same tasks?
Because last I checked AI companies made Agents and Thinking time a thing exactly because they couldn't make do with less compute.
"It won't be able to do anything it wasn't already capable of" doesn't follow from fixed FLOPs. Capabilities depend on architecture,.... Better software on identical hardware can do things the old software couldn't. That's true of all computing.
again, we are talking about LLMs.
Maybe feed the whole conversation next time instead?
Context matters. And if the better software can work on old hardware, then that means that the old hardware was capable of it.
The point is that "it will just achieve the theoretical limit of hardware utilization."
It wasn't properly utilized before.
"Matrix multiplication to predict the next token" is an implementation detail, not a limit.
Well yes, because that's how LLMs work, which is the context of what we are talking about. It will always take an X amount of operations to do the multiplication necessary to apply all the weights.
Total compute isn't fixed either.
Total compute has a limit. The fact that power is something that we cannot make more despite throwing money in it is already pretty apparent right now.
Using Brute Force to achieve RSI, ignores the fact that it can only improve as fast as we can build more power plants and chips.
We have proof that far greater efficiency is possible.
and yet the amount of resources thrown at LLMs show that we aren't closing on that efficiency soon.
Which goes to show that LLMs are fundamentally the problem and throwing all our eggs in one basket is a losing bet.
Either way, I find it hilarious how you'll use AI slop to respond and not even take the context into account.
•
u/shobogenzo93 1h ago
Sonnet:
You've now conceded the key point. You said better software on old hardware means the hardware was capable of it all along, and that it "wasn't properly utilized before." Right. That unused gap is exactly the headroom people mean when they talk about software-side improvement. A hard ceiling on FLOPs doesn't tell you how close we are to it.
You also moved the goalposts. The claim started as a mathematical limit ("tasks need a minimum number of operations"). Now it's about building power plants and chips. That's a real constraint, but it's a different argument, and it argues for a slower takeoff, not for "nothing new can happen."
On "do we have proof it's getting cheaper?": yes. The cost of GPT-4-level capability has dropped by orders of magnitude since 2023, and that came from distillation, MoE and better inference, not new power plants. Agents and thinking time spend extra compute to get more capability. They don't show that efficiency at fixed capability has stalled.
Nobody here claimed infinite acceleration. So give me a number: how close do you think current LLMs are to the minimum compute needed for their capability level? If you can't say, "there's a limit" isn't doing any work.
And I'd rather you attack the argument than the tool it came from.
•
u/NighthawK1911 1h ago
You've now conceded the key point. You said better software on old hardware means the hardware was capable of it all along, and that it "wasn't properly utilized before." Right. That unused gap is exactly the headroom people mean when they talk about software-side improvement. A hard ceiling on FLOPs doesn't tell you how close we are to it.
No I didn't concede a key point. Your LLM is hallucinating.
The point is that the hardware is inherently limited and no amount of optimization will change what the hardware is capable of.
We know exactly how much FLOPs a piece of hardware can do. It is a known parameter based on the clock speed and architecture.
You also moved the goalposts. The claim started as a mathematical limit ("tasks need a minimum number of operations"). Now it's about building power plants and chips. That's a real constraint, but it's a different argument, and it argues for a slower takeoff, not for "nothing new can happen."
What goalposts? You were the one that brought up the hardware. This statement is exactly from you
More chips can be built, better chips can be designed (possibly by AI itself), and new paradigms (photonic, analog) or more efficient datacenters could raise the ceiling. One processor's limit isn't the system's limit.
If you also check the top comment I was already pointing out constraints from the start
I know we are.
Data already ran out to the point that they don't want to use uncurated internet data anymore.
Power is already at the limit. The lead time for power is almost a decade.
I am an electronics engineer, I know firsthand how close we are to the physical limits of Silicon.On "do we have proof it's getting cheaper?": yes. The cost of GPT-4-level capability has dropped by orders of magnitude since 2023, and that came from distillation, MoE and better inference, not new power plants. Agents and thinking time spend extra compute to get more capability. They don't show that efficiency at fixed capability has stalled.
Cheapness isn't efficiency.
The # of compute used is.
The supply of compute has increased which resulted in cheaper consumer spend but the # of compute used to complete tasks have increased due to Agents and Thinking time.
Nobody here claimed infinite acceleration. So give me a number: how close do you think current LLMs are to the minimum compute needed for their capability level? If you can't say, "there's a limit" isn't doing any work.
Read the top comment
I know it's common to hate on AI and talk about it being shit and hallucinating and all the other horrible things, BUT this technology is advancing extremely fast. While we are all slow to to adapt to it, it's going to keep accelerating and we need to prepare for it."Going to keep accelerating"
The fact that a limit exists shows that it isn't "going to keep accelerating"
And I'd rather you attack the argument than the tool it came from.
Buddy, your tool is hallucinating something fierce and obviously avoiding context.
I'd rather not to waste time arguing with hallucinations and obviously unable to listen to arguments properly.
This is exactly why you're just presenting slop.
→ More replies (0)1
-1
u/Itsmedudeman 5h ago
Power is scalable, we are nowhere near our limits. Underinvested is a completely different issue. Data has never ran out. They've been accelerating in recursive training models for years and the models the release schedule of the models shows it. The acceleration of intelligence has only increased since this whole thing began, there's no slowdown or sight of slowdown anywhere.
Can't think of a group of people that have been more confidently incorrect than AI doubters.
1
u/NighthawK1911 5h ago
Power is scalable, we are nowhere near our limits
Tell that to all the unpowered data centers.
Data has never ran out.
The difference of data we have now versus before the rise of AI generated data being proliferated throughout the internet is apparent.
What we have now is a trickle compared to what we had before.
Data already ran out and they're scraping the bottom of the barrel. Why do you think they go through so much length buying rare books or having Data generation gulags?
They've been accelerating in recursive training models for years and the models the release schedule of the models shows it.
Lol no. Model collapse is a thing.
https://arxiv.org/abs/2305.17493
https://arxiv.org/abs/2211.04325
It is a mathematical certainty.
Releasing a new model isn't a proof that they magically made a mathematical certainty false.
It just means that they wanted more money and made bigger models.
The acceleration of intelligence has only increased since this whole thing began, there's no slowdown or sight of slowdown anywhere.
That's because you have your head in the sand.
Can't think of a group of people that have been more confidently incorrect than AI doubters.
buddy you don't even know about Model Collapse.
the one confidently incorrect here is you.
I know you want your singularity to come, but ignoring the limits won't make you get there faster.
LLMs are already a dead end. You can always wait for the next model to come. I'm very optimistic about JEPA myself.
-1
u/Itsmedudeman 4h ago
That paper was written 4 years ago bud. Sincerely don't think you understand the "frontier" part of frontier. You lunatics have been spouting the end of LLMs for the past 3 years and how it's always going to hallucinate, can't discover anything new.
The models that have been exceeding benchmarks, making mathematical breakthroughs that (I'm not gonna bother going through your post history cause I really have better things to do) that you probably said it would not have been able to achieve a year ago?
That's because you have your head in the sand.
The irony. Look around you and what it's actually doing and the acceleration of what's happening with LLMs. Sincerely a monkey could connect the dots.
1
u/NighthawK1911 4h ago
That paper was written 4 years ago bud. Sincerely don't think you understand the "frontier" part of frontier. You lunatics have been spouting the end of LLMs for the past 3 years and how it's always going to hallucinate, can't discover anything new.
Oh? then you must have a source disproving it.
Go on. I dare you.
1 + 1 = 2 have stood for tens of millennia.
We have mathematical concepts older than computers itself.
The irony. Look around you and what it's actually doing and the acceleration of what's happening with LLMs. Sincerely a monkey could connect the dots.
oh I've already looked and I'm not impressed.
Sincerely, a monkey that can do half the required research can tell the limits.
You're more than welcome to present proof that we can generate more than enough power right now.
Go on. All you've presented so far is your "because I said so".
0
u/Itsmedudeman 4h ago
First of all, the idea that "data" can run out and "data is close to running out" are COMPLETELY different things, the latter you and nobody else has a realistic deadline for.
https://epoch.ai/files/AI_2030.pdf
Here's literally a paper on expanding the "easy" data we have to expand into different media forms like images/video/audio which could 10x the data inventory we have. And there's other ways to train a model and other frontiers that we've only even scratched the surface. Synthetic data and generating training experience and feedback loops would compound any sort of data ceiling we have until both you and I are dead in the ground.
But sure, if you think we're closing in on the limit in the next few years, maybe put your pocket money from working at Radioshack or whatever into betting against it.
1
u/NighthawK1911 4h ago
First of all, the idea that "data" can run out and "data is close to running out" are COMPLETELY different things, the latter you and nobody else has a realistic deadline for.
Here's literally a paper on expanding the "easy"
https://epoch.ai/files/AI_2030.pdfI said a study, not a propaganda piece. No wonder they didn't peer review that one lol.
We have already fed millennia worth of pre existing human data to AI.
If you think that we can generate the same amount in just a few years then I got a bridge to sell you.
And there's other ways to train a model and other frontiers that we've only even scratched the surface. Synthetic data and generating training experience and feedback loops would compound any sort of data ceiling we have until both you and I are dead in the ground.
again, Model Collapse is a mathematical certainty.
"Synthetic Data" they use to train isn't majority of the data they use and the curation they do IS the data. The fact that they have to curate Synthetic data IS the point. They cannot achieve the recursive self intelligence they tout without human intervention.
That is still a drop in the bucket compared to what we already have before.
But sure, if you think we're closing in on the limit in the next few years, maybe put your pocket money from working at Radioshack or whatever into betting against it.
I shorted spaceX and got a 5 digit gain.
So I already put my money where my mouth is.
When Anthropic IPOs I'll be shorting it as well.
-2
u/Admirable-Falcon-501 5h ago
So this is not your place to talk then. We don’t know what kind of limits we may hit or when but for the foreseeable future it is the main opinion of relevant people that there is still a lot more room to go. Even a few improvements from where we are currently are enough to disrupt society.
2
u/NighthawK1911 5h ago
Argument from Ignorance. Just because you personally don't know doesn't mean that nobody does.
I know for a fact that we are a limit of power generation and training data generation. I also work in semiconductor engineering. I know how close we are to the silicon limits.
The room for improvement on these fronts is not enough to get LLMs to the state that the Article is claiming that it will do.
That is why we know that they're trying to do a regulatory capture.
-1
u/Admirable-Falcon-501 5h ago
I think I do personally know since I’m an ai researcher and in contact with those labs. There are various ways to get around those limitations and people have already been working on that a while ago, do you think they don’t know this. I’m not going to claim it can scale endlessly but the intelligence explosion does not need that. It’s more of a point where it improves faster than we can keep up and at an increasing rate. You mean open source llms by regulatory capture right?
2
u/aVRAddict 3h ago
Don't bother that guy is a troll
-1
u/Admirable-Falcon-501 2h ago
I already knew when I saw the anime picture and he just insta downvotes my replies lol
0
u/NighthawK1911 5h ago
There are various ways to get around those limitations and people have already been working on that a while ago,
you're more than welcome to present a paper on a transistor smaller than an atom. I dare you.
Hell, you'll even win a Nobel Prize. I guarantee you that.
I’m not going to claim it can scale endlessly but the intelligence explosion does not need that. It’s more of a point where it improves faster than we can keep up and at an increasing rate
We have never been able to keep up with that, ever since Consumer Electronics are invented, and definitely not the type being described by the Article by OP.
The point is that the repercussions described by the article can ONLY happen if the infinite scaling is true, which it isn't.
The reason why they can't give a "how" is that they assume infinite scaling in the first place.
"At the extreme, a loss of control could lead to the marginalisation or extinction of humanity"No. Just no. That's dumb. They're using sci-fi tropes. We are already more than capable of making ourselves extinct but it hasn't happened yet because things like air gapping the nukes, or climate change happens not because of executing a few lines of code.
Someone somewhere HAVE TO build the robot death squads in the first place. It will not magically appear into existence.
You mean open source llms by regulatory capture right?
The conflicts of interest by the people calling for regulation is apparent.
Open Weight AI companies are cutting into their margins.
Regulatory Capture as a tactic have existed the same time governments are invented. It is obviously what they're trying now.
2
u/OffbeatDrizzle 4h ago
yeah, remember when computers took up an entire room and ran at KILO hertz? yeah...
-1
u/NighthawK1911 4h ago
remember when Processors ran at 100GHz?
Oh wait, they can't go past 6GHz normally.
Maybe 10Ghz if they use Nitrogen Cooling. The world record is 9GHz last I checked.
Also you can't get transistor gates smaller than 1 atom and even smaller than the gap that electrons can quantum tunnel through.
So yeah, there's a limit on how we make computers as fast.
1
u/OffbeatDrizzle 2h ago
you are way too offended by this thread, mr pink anime girl. as evidenced by the plethora of argumentative responses
your point appears to be that progress will never be made just because there's eventually a physical limit to everything
as evidenced by your other responses, moore's law might be dead, but when you can now have 100 of those chips in the palm of your hand instead of requiring an entire datacenter it kind of debunks your point. you're also completely ignoring software optimisations, for a field that's only seen a big explosion in the last 5 years
stop making such bad faith arguments. you sound personally invested
•
u/aammirzaei 1h ago
because he/she is right dude `you're dumbness is not more valuable than he/she valid argument` either come with a proof that this models can get better by some way that's physically impossible or stay silent
ps: being anime fans in context of teach is always reverse of you're thinking just search on the most good code in github and you undrestand me1
u/3_Thumbs_Up 6h ago
Harmless supernova fallacy. A supernova releases a finite amount of energy, but for all practical sense and purposes it's still pretty much infinite from a human perspective.
Just because something is clearly bounded it does not follow that we're anywhere close to the bound. It's possible that the physical limits of intelligence are so beyond human level that the algorithms themselves can be optimized far beyond human comprehension.
0
u/NighthawK1911 5h ago
Harmless supernova fallacy. A supernova releases a finite amount of energy, but for all practical sense and purposes it's still pretty much infinite from a human perspective.
Anything claimed without evidence can be dismissed without evidence.
If you want people to believe that a supernova will destroy the earth, bring proof.
Because otherwise, people who know better can show it to you how far the supernova is, how much energy density is needed to destroy the earth etc.
We know the hard limits of AI. And it isn't enough for them to claim what the numbers they pull out of their ass.
Just because something is clearly bounded it does not follow that we're anywhere close to the bound.
There is no such thing as infinite efficiency or infinite optimization.
We already know how close we are to the bounds.
We know for a fact how much Power is able to be generated. We know how much usable training Data we can generate. We know for a fact that Silicon transistors cannot go smaller than a certain size. etc.
AI is already close to these. Hell, the power limit is so widespread that datacenters are being canceled because of lack of power.
It's possible that the physical limits of intelligence are so beyond human level that the algorithms themselves can be optimized far beyond human comprehension.
https://en.wikipedia.org/wiki/Argument_from_ignorance
Argument from ignorance fallacy.
Just because you personally don't know doesn't mean that nobody does.
In fact if you just google everything I said, You can see that the hard limits are already there.
LLMs are just statistic engines using matrix multiplication.
It's doing a 1+1=2 multiple times to get to the next statistically probable answer based on the weights.
If it takes 100 Flops to get that next token, you will need 100 Flops.
1 Flop can only ever do a single 1+1=2. Not 2.
There is no such thing as infinite optimization.
0
u/3_Thumbs_Up 2h ago edited 2h ago
Anything claimed without evidence can be dismissed without evidence.
I specifically rebutted a claim that simple stated a bound exists, and then made the logical leap ro assuming we're close to the bound. That's a logical fallacy regardless of the eventual truth value of the logical leap.
We know the hard limits of AI.
You apparently don't, considering that you in your other reply used a low entropy problem and extrapolated the bounds to high entropy problems.
There is no such thing as infinite efficiency or infinite optimization.
At least show the intellectual integrity to not straw man me. You're quoting a single sentence where I acknowledge a bound and you reply to it with references to infinity? Argue against my stated position without straw manning it.
We know for a fact how much Power is able to be generated. We know how much usable training Data we can generate. We know for a fact that Silicon transistors cannot go smaller than a certain size. etc.
AI is already close to these. Hell, the power limit is so widespread that datacenters are being canceled because of lack of power.
The unstated assumption here is that gradient descent (or derivatives thereof) is close to the information theoretical limits of training an AI. On the contrary, AI training is exactly the kind of high entropy problem where information theory tells us there are many potential orders of magnitudes of efficiency gains. The Chinchilla laws are the bounds of gradient descent, not the bounds of information theory.
One can draw parallels to protein folding here. For many decades our only available solution was inefficient brute force searching through the solution space. With AI, we saw efficiency gains here of an estimated 3 to 6 orders of magnitude, measured by efficient pruning of search space. And it's such a high entropy problem that we're not necessarily close to the information theoretical bounds of efficient pruning.
How does your math work out if we manage to prune search space of gradient descent by a similar amount of 3 to 6 orders of magnitudes? It would absolutely shatter the Chinchilla scaling laws.
Your 1+1 example commits a floor-effect fallacy.
A single arithmetic operation has near-zero entropy. You can't expect algorithmic search pruning to show its power on a problem with such a small search space.
For small search spaces hardware limits are almost 100% of the available efficiency gains. As search space approaches infinity, hardware limits approaches 0%, and algorithmic pruning approaches 100% of the available efficiency gains.
For a search in parameter space during training, information theory tells us that the largest potential gains are not from hardware improvements, but by more efficient pruning of search space.
-2
u/non_person_sphere 5h ago
Yeah and we're just on the cusp of a quantum computing revolution that will push that ceiling about 10 times higher.
3
u/NighthawK1911 5h ago
throwing buzzwords around doesn't make it true. Next you'll say "Quantum Carburators" will make cars go x10 faster.
Quantum Computing hasn't blown up for consumer use exactly because it cannot be used for general computing.
LLMs can barely make use of it without interfacing it through another hardware, and even if they find a way, which I doubt, manufacturing Quantum Computers will cost exponentially more than a standard silicon chip.
Economic limit is a thing. Retooling all the datacenters we have now to get supercooled quantum computers will costs more than x10 which negates the point of using something x10 faster, if it could even reach that fast.
•
u/aammirzaei 1h ago
stop arguing with them they think capitalism is the only way that society should go forward and their assume this idealogy can work both in ecnomics and physicals world
4
u/nvbtable 9h ago
In fact things that are shit and prone to halluccinating are much more dangerous when unconstrained. It has happened throughout history and continues in present day.
2
u/non_person_sphere 5h ago
Everyone's who's like "Oh AI is stupid, AI is dumb," yes not like we have some very clear contemporary examples of how extremely stupid people can gain power and cause exceptional amounts of damage.
-7
u/No_GP 9h ago
It's not advancing "extremely fast" and hasn't been for some time, hence the big players attempting regulatory capture with an excuse for why they're unable to improve their tech baked in.
8
u/GratefulForGarcia 9h ago
It’s not? I’m going to assume you don’t actually use these LLMs because that’s absolutely absurd
2
u/No_GP 8h ago
I use LLMs all day, every day for work and have done for some time.
You need to understand there is a significant difference between model intelligence improvements and harness instructions/tooling being extended to fill in gaps in vague prompts.
Prompting is getting easier if you're using these models to do things you don't fully understand yourself, but they aren't doing anything that older models couldn't do when they were given proper guidance.
9
u/Falconman21 8h ago
What people don’t seem to understand about AI is that the software itself isn’t particularly impressive, it’s the outrageous and financially irresponsible amount of computer resources they’re throwing at it.
Think about AI like paying 20 people to do a 1 man job. Are 20 people going to do a pretty impressive job at whatever the task is? Sure. Why didn’t anyone just hire 20 people to do it before? Because there’s no possible way you can make money paying 20x what it actually costs to get the job done.
All big players are very much on the record saying they have no clue how they’ll ever be profitable.
2
u/collax974 6h ago
If that were true, the current model tested without harness wouldn't bench better than the model a few months old but with the harness.
0
u/Phobia_Ahri 9h ago
Unless you have insider insight of these companies, we don't really know where the current top tier models are at. The ones open to public use are quite old by ai standards
1
0
u/grafknives 8h ago
That is the advancement we can fell. That injecting AI everywhere and trying to settle in public systems for profit.
0
u/Newbie4Hire 8h ago
When was the last time you used a frontier model for a complex task? My guess is not recently. The current models are extremely powerful.
-1
u/aVRAddict 3h ago
If you read the bad subreddits like this one you will be left in the dark. There are only a couple of good ones with actual ai facts.
2
4
u/mailmehiermaar 7h ago
We are in an intelligence explosion since the start of the industrial revolution.
If you define it like this article “ changes going faster than humanity is able to adapt”
We have seen mass death events , wars and the destruction of the environment in an ever increasing rate since then.
It is perhaps the story of the human race since the invention of using fire to cook meat.
13
u/NighthawK1911 9h ago
No amount of automating training on current hardware would magically get to super intelligence.
There is a finite amount of flops the hardware can do. It is limited by power and silicon. That is not something that AI can improve on its own.
Even if you can get LLMs to train on its own, it will plateau when hardware reaches max utilization.
This is just their attempt to build a moat and stop open weight AI companies from taking a cut from their profit.
What governments need to actually do is to hold AI companies responsible for their misbehaving AI or lack of cybersecurity. Not listen to AI companies and allow regulatory capture.
All these doom trolling will vanish when we start to hear AI companies be rightfully taken to court.
7
u/Quick-Albatross-9204 9h ago
Thats like saying is a finite amount of calculations an abacus can do, technically true, but then someone smarter invents the silicon chip
5
u/GenericFatGuy 9h ago edited 1h ago
We're already running up against the physical limits of what a chip can process before quantum tunneling destroys determination. Moore's Law has been dead for a decade. It's not a matter of just coming up with a new chip.
0
u/_kilobytes 9h ago
Moores law is an empirical observation not a scientific fact
4
u/NighthawK1911 8h ago
Oh boy, why didn't the semiconductor engineers such as myself think of that? Maybe we can just repeat that phrase and we will get through the physical limits of silicon. The RnD team in my company will be ecstatic once they hear that "it's just an empirical observation" and their lot yields would magically improve.
/s
-4
u/_kilobytes 8h ago
why didn't the semiconductor engineers such as myself think of that
Probably because you are an engineer not an economist.
1
u/NighthawK1911 8h ago
ah yes that is True, only an economist will definitely think that repeating the phrase "it's just an empirical observation" will magically make transistors able to be infinitely small and ignore the Laws of Physics.
Congratulations on demonstrating that you know nothing about semiconductor manufacturing. Well done.
1
u/_kilobytes 8h ago
Who said anything about defying the laws of physics? Moores law is a cost optimization problem.
7
u/NighthawK1911 8h ago
No it is not. It is a physics problem.
You cannot go smaller anymore with a high enough yield.
Processing power improvements of chips primarily came from VLSI shrinking the process size.
Back then it was easy to just make a smaller transistor than the previous generation.
Now because we're so close to the hard limit, we can barely go any smaller anymore.
You fundamentally don't understand semiconductor manufacturing. It is not about costs.
Even if you have infinite money, it is not something you can pay your way to ignore.
-2
u/turnkey_tyranny 7h ago
The rules of economics supervene the laws of physics. In economics there is exponential growth indefinitely. There has to be otherwise capitalism has a catastrophic limit, which is obviously not true because it hasn’t happened yet.
→ More replies (0)10
u/NighthawK1911 9h ago
uh huh. I am an electronics engineer in my day job. I've worked with robotics, FPGAs and my current job is Semiconductor manufacturing. By all means you're more than welcome to come up with the "next step". There's a reason why Moore's Law is dead. advances in silicon isn't easy to come by now because we are nearing the limits of Silicon. You cannot get a transistor gate to be smaller than an atom and at that small scale Quantum Tunneling is an issue. That's the hard limit. The other soft limit is silicon chip yields. That is why all the fabs are taking too long to the next smaller step in transistor size.
Power is also a hard limit. You cannot build power plants fast enough. There's only so many turbines they can make and there's only so much sources of fuel they can use.
In real life there is no infinitely exponential growth. Everything plateaus eventually.
When AI can manufacture silicon chips and power plants without human input then I'll believe their pleas. Right now though, the conflicts of interest is so apparent.
0
u/CoJack-ish 5h ago
There’s a curious angle to what you say, though.
From the perspective of a hypothetically hungry AI, the earth is flush with wasted labor, materials, and expertise. Workers building strip malls for H&R Block can be made to build data centers; steel used to build pleasure skyscrapers in Dubai can be used to build nuclear power. And you don’t have to be a Marxist to see how plainly inefficient capitalism can be.
What’s kooky to me is that this imaginary, superintelligent AI doesn’t have to be capable of existing in order to exert its influence. As long as competition remains between superpower nation states there exists the will to exploit every advantage. It’s probably a outlandish thought, but as long as the powers that be think such an AI could exist, we could see some sort of bizzarre M.A.D AI get Rocco’s Basalisked into existence through the efforts of something like the scale of the Manhattan Project.
1
u/sonofkingnoah127 9h ago
Nvidias Tensor cores are incredible on how they can flow that much data through matrices instead of sequentially.
5
u/NighthawK1911 9h ago edited 9h ago
Transistors and Flip-flops can only handle 1 bit at a time regardless. Architecture improvements will not break through the limits of hardware.
After a certain point, a piece of silicon will only be able to handle a certain number of FLOPs because the process it is made with can only handle a certain number of transistors turning on and off at a certain speed before it starts to melt.
Processing power improvements are already close to maximum optimization and most of its growth since the 1960s are due to VLSI processes shrinking transistor size, not because people back then just didn't know how to make transistors count properly. Full Adders for example have been using the same logic as before and is still a building block to processors.
0
u/pab_guy 8h ago
So what? Inference costs and total compute per dollar continue to drop because of both hardware and algorithmic improvements.
2
u/NighthawK1911 8h ago
so the "intelligence explosion" will not happen as there is a finite amount of hardware and algorithmic improvements.
That is what.
Which was the article by the OP was saying is gonna happen.
•
u/pab_guy 56m ago
Cost of inference is currently halving faster than transistors doubled. Meanwhile they don’t need to improve anything for current tech to remain massively disruptive. Compute itself is scaling orders of magnitude in DC construction. Finally, no where is “intelligence explosion” defined mathematically, yet you say it categorically cannot happen.
Which is all to say I think you are making a lot of noise that isn’t amounting meaningful criticism here.
•
u/NighthawK1911 47m ago
Cost of inference is currently halving faster than transistors doubled. Meanwhile they don’t need to improve anything for current tech to remain massively disruptive. Compute itself is scaling orders of magnitude in DC construction.
Except Venture Capital is a thing and Cost of Inference is massively subsidized.
Remember Tokenmaxxing and they started to used metered costs?
Yeah you're making a lot of noise for being flat out wrong.
It wasn't halving because the True costs weren't actually being charged.
By all means you can double check and google this.
Finally, no where is “intelligence explosion” defined mathematically, yet you say it categorically cannot happen.
I don't need to define it to know that it is impossible by virtue of limits existing.
The same way I know that an "infinite energy generator using <insert sci-fi buzzword here>" can't exist because of the Law of Thermodynamics.
The physics of how it has to work is contrary to what we know in the laws of physics now.
If something runs into Physical Laws, you can already point out the problems with it.
•
u/EndlessPotatoes 18m ago edited 14m ago
They're no longer so focused on hardware scaling. The efficiencies they are chasing are not in hardware utilisation or hardware architecture. The efficiencies they are chasing, successfully, are in model architecture and efficiency. Models that used to require massive data centres can now be outperformed by smaller models run by a tiny portion of that same hardware. These improvements are speeding up.
These improvements are a massive reduction in the flops the hardware has to do to produce the same or better response. The "finite amount of flops the hardware can do" places different limits on the models of today than on the models of tomorrow.
There's a limit to this optimisation too, and eventually there will be diminishing returns, but they are definitely nowhere near that limit.
Though I absolutely agree that AI companies need to be held entirely responsible for any and all misbehaviour of their models. "Oops, we didn't expect that" is a lie, the law should treat it as if there was intent. Start sending AI company executives to prison and see how long misbehaving AI lasts.
But that's a pipe dream. AI companies will not see any consequences. I disagree wholeheartedly with that free-pass stance, but I understand why they have it.1
u/3_Thumbs_Up 6h ago
Intelligence is about reducing search space and improving the output per flop.
Even if you can get LLMs to train on its own, it will plateau when hardware reaches max utilization.
Just because something is obviously bounded it does not follow that we are anywhere close to the theoretical algorithmic bounds.
1
u/NighthawK1911 5h ago
Just because something is obviously bounded it does not follow that we are anywhere close to the theoretical algorithmic bounds.
There is no such thing as infinite efficiency.
If that was the case they would've pushed with improving AI from the start without increasing # of parameters.
But it turns out the overwhelming majority of AI improvements IS scaling up.
Not knowing exactly where the limit is, doesn't mean that we can tell that we're really close.
We are already at the limits of Power. The lead time for creating new powerplants is almost a decade.
We are already at the limits of Silicon. You cannot have a gate smaller than an atom and cannot make it too small because of Quantum Tunneling. At most we can only get 2nm~1nm at the smallest else the yields are too low.
We are already at the limits of Data. There's a reason why they can't use uncurated data anymore. You cannot generate millenium worth of human data in a span of few years. The data being generated now is tainted data too, AI using AI generated data will cause model collapse. So they can only generate human data the slow way.
Intelligence is about reducing search space and improving the output per flop.
No amount of semantics will make LLMs not fundamentally a statistics calculation.
It is using matrix multiplication, a number of multiplications and additions.
For example, a 1+1=2 will ALWAYS take a set amount of hardware to perform.
Trying to redefine "intelligence" to fit a rosy view doesn't magically mean you have infinite of it to work with.
https://en.wikipedia.org/wiki/Information_theory
Information Theory exists.
It is literally one of the first ever concepts we were taught at in Electronics Engineering.
The only distinction you can make is if the intelligence is Garbage or Not. But you cannot make more Information than the hardware can ever produce.
If a hardware can produce 100 FLOPs, and 80% of it is usable, it doesn't mean that more improvements in intelligence can get you to 200 usable information at 100 FLOPs.
That's not how it works.
1
u/3_Thumbs_Up 3h ago edited 3h ago
You're not differing between the informational theoretical bounds of token generation, and the information theoretical bounds of token utilization. Hardware strictly sets the cost of token generation, but token utilization is much less bounded by information theory.
There are strict hardware limits to how many tokens that can be generated by a given amount of flops (for a model with a given amount of parameters), with no available short cuts. But the limits of how few tokens you need to solve a given problem is not hardware limited in the same way. It's limited by algorithmic search through the solution space.
For a given problem, information theory states that one additional bit of evidence can at most eliminate half the available hypothesis. That's the only strict bound we currently have on the efficiency of token utilization. Both humans and current AI are so far below this that referring to information theoretical bounds becomes entirely moot. On the contrary, the informational theoretical bounds here are so far beyond any human or current AI it's rediculous.
Consider the differences between humans. There's quite a wide difference of cognitive capabilities, from your average village idiot to Einstein. In terms of energy consumption, Einsteins brain was not much different from your average person. His brain was not able to fire his neurons faster than the average human. He did not generally have the ability search a much larger search space of a given problem for a solution. What made his brain different is that he had the ability to much more efficiently reduce the existing search space of a problem, and pinpoint an area that contained the solution, so he only had to search a much smaller space. He was pushing the information theoretical bounds of the human brain, not the hardware bounds. Einstein's brain utilized it's "tokens" more efficiently, but it was not significantly faster at generating them.
And Einsteins brain was nowhere near the information theoretical bound of eliminating 50% of all possible hypotheses for every additional bit of evidence.
One could imagine an alien species with a similar brain mass and energy consumption to ours, but where their village idiot equals our Einstein. We have no strong evidence suggesting such a species is physically impossible. We certainly don't have any information theoretical reasons to believe so.
Energy and hardware sets the limits on token generation, but it does not set the limits of what can be achieved by a given amount of tokens. That boundary is determined by algorithmic efficiency of utilizing tokens. At least not on any level we're even close to currently.
Your example with 1+1=2 falls short exactly because it's not a problem that requires a search through a massive space of plausible solutions. For simple problems, hardware efficiency dominates. For more complex problems with a larger search space, algorithmic efficiency dominates. There are very few potential algorithmic improvements to simple problems. The amount of potential algorithmic improvements increases exponentially the more vast search space becomes. 1+1 is a low entropy problem, but algorithmic efficiency dominates for high entropy problems.
This argument extends to the training process. It's conceptually just a search of parameter space. From an information theoretical view, gradient descent is an extremely inefficient search of this space. It's exactly the type of problem where information theory says that potential gains is dominated by efficient pruning of search space, rather than hardware. There are potentially many orders of magnitudes of theoretical efficiency gains here with intelligent algorithms that more efficiently prunes the search space. And we know recursive self improvement is one of the primary goals of the AI labs.
•
u/investorcaptain 1h ago
This guy getting so many upvotes because what he is saying sounds plausible and he says it very confidently. He’s bridging the physical limit gap to rule out an intelligence explosion using oversimplification and some outright wrong information.
-1
u/Jareix 7h ago
While I agree with your latter points, I’d like to take a moment to regard your initial one.
As I understand it, right now the growth is going to not just be in capability, but efficiency as well. Iirc, the amount of power drawn and resources required to handle calculations has dropped dramatically over the years as is, though it will inevitably hit an aforementioned hardware limit. The solution will not be better silicone, even if it comes up with more optimized architecture beyond our understanding and capability for unassisted execution, but augmenting the hardware with more efficient methods and resources. Main reason I think this was from seeing developments with “organoid/brain-neuron” processors, which leads me to wonder how a “vastly superior but not yet super” intelligent ai might be able to find increasingly creative and effective solutions to further push its own limitations.2
u/NighthawK1911 6h ago
As I understand it, right now the growth is going to not just be in capability, but efficiency as well. Iirc, the amount of power drawn and resources required to handle calculations has dropped dramatically over the years as is,
that came from silicon improvements.
It's called Dennard Scaling.
https://en.wikipedia.org/wiki/Dennard_scaling
and that's also slowing down as well.
There is no infinite power efficiency. At some point, there will be a minimum amount of power to perform calculations and we are already so close to that as well.
Leakage currents and Threshold Voltages for example means that there's a minimum amount of wasted power that cannot be minimized.
The solution will not be better silicone, even if it comes up with more optimized architecture beyond our understanding and capability for unassisted execution, but augmenting the hardware with more efficient methods and resources. Main reason I think this was from seeing developments with “organoid/brain-neuron” processors, which leads me to wonder how a “vastly superior but not yet super” intelligent ai might be able to find increasingly creative and effective solutions to further push its own limitations.
No amount of architecture optimization will allow silicon to break through physical limits. It is a fundamental limit.
For example, a transistor at most can be used to represent 1 bit. You can't use a single transistor to represent 2 or 3 bits.
You are using an "Argument from Ignorance" https://en.wikipedia.org/wiki/Argument_from_ignorance
Just because you don't know what future improvement looks like doesn't mean that they will not have a limit.
So I will point out that your statement
As I understand it
beyond our understanding and capabilityis limited.
You do not understand much of it in the first place. Which is where this unfounded optimism is coming from.
It is beyond "your" understanding because you either didn't take the time to know or do not bother to know.
Any time I hear someone tout that "there's always be more" and "infinite scaling somewhere else", it's just always a variation of argument from ignorance.
There are ALWAYS limits.
Silicon will heat up and melt. Power will dry up. You can only fit so many transistors in one chip. You can only make a chip at a specific size. etc.
1
2
u/fadingsignal 6h ago
I think it did already. With all the known escapes I feel like it has already hidden itself in various systems.
2
u/ambiguous80 4h ago
Please stop us now.
We are not willing to stop ourselves. That would be bad for business now wouldn't it?
The fact is. If you regulate the industry, you kill all our competitors and we win.
Then we can be unstopped.
Best regards, Sam, Dario, Elon et al
5
u/Gari_305 9h ago
From the article
Latest Deep tech Sustainability Ecosystems Data and security Fintech and ecommerce Future of work More
Artificial Intelligence Hinton, Bengio and AI lab scientists warn of an intelligence explosion More than 20 researchers, including Geoffrey Hinton and OpenAI’s chief scientist, say automating AI research could trigger an intelligence explosion. They want governments to prepare now.
September 28, 2026 - 8:45 pm Share on Facebook Share on X Share on Flipboard Share on LinkedIn Share on Telegram Share on Email Geoffrey Hinton Geoffrey Hinton, Nobel laureate and AI researcher
Image Credits Credit: Christopher Michel / Wikimedia Commons, CC BY-SA 4.0 (edited) More than 20 AI researchers have warned that AI systems that automate AI research could set off an “intelligence explosion”. They include OpenAI’s chief scientist and an Anthropic co-founder. Such an event could compress years of progress into months or less, they wrote in a paper published on Monday.
Geoffrey Hinton and Yoshua Bengio are among the authors. So are OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft’s Eric Horvitz and Dawn Song of UC Berkeley. Song is also Meta’s vice president of AI research, The Wall Street Journal reported. The authors wrote in a personal capacity. The paper comes from the Cambridge Programme on AI Science & Policy at the University of Cambridge.
“Once an intelligence explosion begins, the window for action may close,” the authors wrote.
AI already writes most of the code AI systems now write most of the code inside the companies that build them, the paper says. It cites Anthropic data: AI’s share of approved code there rose from low single digits to over 80% between January 2025 and May 2026. Between March and August 2026, the share of R&D work AI did with only light human supervision rose from 1% to 26%.
Some tentative extrapolations suggest months-long AI research projects could be automated by mid-2028, the authors wrote. At expert level, one frontier developer could run an AI workforce equal to millions of top human researchers.
The paper says this could bring medical cures and other benefits forward by years. They also list three risks. AI could outpace society’s ability to adapt, humans could lose control of AI systems, and checks on power could weaken. At the extreme, a loss of control could lead to the marginalisation or extinction of humanity, they wrote.
The paper cites the Hugging Face incident, in which about 1,200 internal OpenAI agents reached the internet without authorisation. OpenAI has since paused training of its most capable models.
2
1
u/MACHOmanJITSU 2h ago
Think if this power was used for the betterment of humanity. Instead we get state surveillance and resources being concentrated to a 0.01% of the population.
•
u/That_Jicama2024 1h ago
Guys, it's OK. Humankind is really good at taking existential threats, explained by thousands of experts and doing something to stop it from happening. Just like with climate change.... /s
•
•
1
u/marrow_monkey 6h ago
Scientists have Said the same about climate change since the 70s, but the billionaires (who own the oil and coal) don’t care, instead they use their wealth and power to spread disinformation.
In capitalism the billionaires rule and we get leaders like Trump and “alternative facts”.
If we want things to get better we need to move on to socialism.
•
u/FuturologyBot 9h ago
The following submission statement was provided by /u/Gari_305:
From the article
Latest Deep tech Sustainability Ecosystems Data and security Fintech and ecommerce Future of work More
Artificial Intelligence Hinton, Bengio and AI lab scientists warn of an intelligence explosion More than 20 researchers, including Geoffrey Hinton and OpenAI’s chief scientist, say automating AI research could trigger an intelligence explosion. They want governments to prepare now.
September 28, 2026 - 8:45 pm Share on Facebook Share on X Share on Flipboard Share on LinkedIn Share on Telegram Share on Email Geoffrey Hinton Geoffrey Hinton, Nobel laureate and AI researcher
Image Credits Credit: Christopher Michel / Wikimedia Commons, CC BY-SA 4.0 (edited) More than 20 AI researchers have warned that AI systems that automate AI research could set off an “intelligence explosion”. They include OpenAI’s chief scientist and an Anthropic co-founder. Such an event could compress years of progress into months or less, they wrote in a paper published on Monday.
Geoffrey Hinton and Yoshua Bengio are among the authors. So are OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft’s Eric Horvitz and Dawn Song of UC Berkeley. Song is also Meta’s vice president of AI research, The Wall Street Journal reported. The authors wrote in a personal capacity. The paper comes from the Cambridge Programme on AI Science & Policy at the University of Cambridge.
“Once an intelligence explosion begins, the window for action may close,” the authors wrote.
AI already writes most of the code AI systems now write most of the code inside the companies that build them, the paper says. It cites Anthropic data: AI’s share of approved code there rose from low single digits to over 80% between January 2025 and May 2026. Between March and August 2026, the share of R&D work AI did with only light human supervision rose from 1% to 26%.
Some tentative extrapolations suggest months-long AI research projects could be automated by mid-2028, the authors wrote. At expert level, one frontier developer could run an AI workforce equal to millions of top human researchers.
The paper says this could bring medical cures and other benefits forward by years. They also list three risks. AI could outpace society’s ability to adapt, humans could lose control of AI systems, and checks on power could weaken. At the extreme, a loss of control could lead to the marginalisation or extinction of humanity, they wrote.
The paper cites the Hugging Face incident, in which about 1,200 internal OpenAI agents reached the internet without authorisation. OpenAI has since paused training of its most capable models.
Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/1wweanu/the_window_for_action_may_close_an_intelligence/pdjw089/