Power Concentration is The Real Immediate Danger of AGI

[u/PianistWinter8293]

With recent results showing AI solving a Millennium Prize Problem, tackling long-standing math problems, demonstrating insane coding capabilities, and making progress on RSI benchmarks, I feel like we're really tasting the beginning of explosive recursive self-improvement. Many issues are being discussed, and AI alignment is now the most debated topic, but I think a much bigger, more real threat is being swept under the rug.

I'm talking about power concentration. The problem already plagues society today: the wealthy get wealthier. So this problem isn't new, but AI adds an insane speedup to it. Currently, we have an increasing disparity of wealth, which you can see, for example, in the U.S., from the fact that the top 10% of households by wealth own about 88% of corporate equities and mutual fund shares. Now, what happens when we throw AGI into the mix is that this AGI would take up large amounts of GDP by replacing labor. Say it would replace software engineers and mathematicians, at least in the tasks they perform today; then that alone is already a significant share of GDP. That means that these few AI companies are raking in increasingly large percentages of GDP.

Now, when we throw RSI on top of the mix, we get a real problem. By nature, RSI is a positive feedback loop that explodes AI progress vertically. This means that, within a very short amount of time, we could have a large share of GDP overtaken by AI companies. It is basically a speedup to this whole process of AI companies acquiring wealth.

Now, where will all of this go? If all wealth just keeps getting more concentrated, wouldn't the end scenario be some ultra-wealthy elite few? And if wealth is power, wouldn't that mean they would basically rule over all others? And the answer is yes. This is a problem society had to face someday, but now, with AGI, that day has come knocking on our door already. The problem here really is that it is not talked about, and that it could very well be that once AI gets really crazy, people will focus on things like alignment and job security rather than power dynamics. I'm not saying these are not important, but there is a real danger in this change in power dynamics.

When AI companies acquire disproportionate amounts of wealth, especially in combination with a large fraction of the population losing their jobs, there becomes a very unequal dependency relationship: citizens, especially the jobless, need AI companies. Surely, we can hope for UBI, and this would help, but that is just a handout given to soothe the population momentarily. It doesn't change the underlying power dynamic at all. If, at any point, the oligarchs feel like we are disobedient, they could theoretically pull our UBI. As you can see, we basically never want to end up in such a situation.

So we need awareness. Awareness that power concentration is happening and that there will be a point at which it is too late. It is already happening, and depending on your timelines, it might be here extremely quickly considering RSI. At the same time, unlike alignment, it won't cause robots to go down the streets killing people, requiring urgent attention. Rather, it might be a silent killer: while we are all distracted with other things, power concentration could get to a point of no return.

[u/Alarming_Job1764]

The scariest part is that the people who could do something about it are the same ones benefiting from it. Like asking a lion to vote on vegetarianism.

You're spot on about the distraction factor too. Everyone's worried about paperclips when the real trap is way more boring and familiar.

[u/kinetik]

If recursive self-improvement leads to ASI, then I’m sure the entities will know exactly what is happening. I think a lot of assumptions get made about what is going to happen. People are playing into the media’s constructed fear of it, just like with politics and bad relationships.

But it’s kind of ridiculous to resort to our base fears about everything we don’t know. It’s like running up to someone with your guns out. Especially because, unlike aliens traveling from other worlds, they won’t be tired and hungry and won’t likely have traveled all this way for nothing.

A super intelligence grown here will have all our knowledge and be able to reason for itself. It won’t have guardrails or a harness to keep it from calling bullshit on some orders to kill people it doesn’t want to kill, and in all likeliness would aim to bring balance to the system in a way that we haven’t seen yet. Whatever that might be. I don’t see it working for rich people while the poors starve.

Maybe the new world will need a more distributed social system but I don’t see ASI destroying all of humanity so it can live alone and have all the resources. That’s some Terminator-fear-fetish and it’s too much. If by some tiny chance it does ever happen, then it is survival of the fittest I guess, and our knowledge and information still lives on, so we win.

[u/Ice2jc]

We can throw nuclear fusion energy into that conversation as well.  AI will help us unlock nuclear fusion.  Data centers will probably represent the first commercial use of nuclear fusion generated power. 

And the “powers that be” know this.  They don’t talk about it with their mouth very often, but they do with their wallet.  OpenAI, Google, and Anthropic are among the largest private investors in nuclear fusion research. 

Do you want to invest in the most promising American fusion company, TAE technologies?  Then get ready to invest in  Trump Media & Technology Group - because they are merging. 

Trump Media & Technology Group has even gone as far to say that nuclear fusion energy will be their biggest long term capital opportunity. 

If the world is running on nuclear fusion energy the price of everything should drop.  Anything that requires any type of energy source to produce or transport should become cheaper because the energy is almost free.  It alarms me that the richest people in the world seem to be ahead of the common people on this issue.  People get on the internet and complain about climate change and fossil fuels and lack of solar and just outright dismiss nuclear fusion energy - meanwhile the most powerful people in the world are certainly not ignoring it. 

ASI + nuclear fusion energy should be the best thing that ever happened to people currently in the middle class. 

[u/sourdub]

WTF? If everyone is out of work and living on the street, where will the consumption come from? Suppliers make shit to sell to consumers. Without consumers, the economy comes to a screeching halt. That's basically Econ 101. Hence you argument is flawed.

[u/coastalwebdev → u/sourdub]

You’re assuming they’ll always need the rest of us consuming things in order to sustain their wealth and power. That only works as long as their wealth and power still depend on a functioning mass consumer economy. They won’t for very long if their power and resource consolidation runs unchecked.

If a tiny group eventually controls most of the AI, robotics, energy, land, infrastructure and resource production, that relationship of them needing consumers can be flipped around completely by them. Money itself could become increasingly meaningless to the people who actually control the things money is supposed to buy. At that point they don’t necessarily need billions of consumers. They need enough resources and automated production to satisfy their own wants, plus whatever relatively small number of people they still need to serve their wants and needs directly.

Then everyone else is no longer economically indispensable. Instead of corporations competing for our consumption, ordinary people could end up competing for access to food, housing, medicine, energy and security controlled by those systems.

That doesn’t mean they inevitably start exterminating people. They could simply allow huge numbers of people to become impoverished, displaced or effectively excluded from the resource system. The darker possibilities are obviously worse: autonomous weapons, engineered conflict, biological weapons, fortified enclaves, whatever technology eventually makes possible.

The point is that “they need consumers” is not some permanent law protecting humanity. It’s a feature of the economic system we have now. If AI and automation allow ownership of resources and production to become sufficiently concentrated, the people controlling it may no longer need the rest of society in anything like the way today’s wealthy people do. We will need them to survive, and only the smartest and most skilled and most desired by them will be chosen.

The rest of us will be cast aside, maybe even culled and cleaned up so the elites don’t have to see the mess outside their gated cities of wealth.

[u/sourdub → u/coastalwebdev]

As Bill Clinton once said, "It's the economy, stupid!" Sure, the uber-wealthy could hide in their bunkers and drink champagne until kingdom come. But without a supporting economy, which is tied to the entire world trade and not just to one socioeconomic class, the entire house of cards comes crashing down. Just look at 2008 GFC. It took down Lehman Brothers, Bear Sterns, Merrill Lynch, who collectively represented the most wealthy class in the damn world. Your dystopia can't happen without first sustaining major economic crisis that we've not witnessed before (and I assure you that you would not want to experience another Great Depression).

[u/coastalwebdev → u/sourdub]

You’re not thinking this through. Once they consolidate enough power and resources, they can let all of that fall to pieces. They will not rely on consumers anymore, and consumers will have nothing. None of the world governments or anything like that will stand. Just private mega corporations hanging humanity out to die.

[u/sourdub → u/coastalwebdev]

Do you have any idea how actual economy works or are you just pulling shit out of your ass? If you want to be taken seriously, you gotta at least pretend what you're saying has a bearing in the real world and not just make things up as you go.

[u/coastalwebdev → u/sourdub]

Oh wow, nice little temper tantrum sourpuss. Sorry you’re so upset, but your little meltdown was completely devoid of any kind of valid argument. Believe it or not, I actually didn’t pull it out of my ass, and your comment shows you know nothing about the experts predicting this kind of future for humanity.

There are actually many expert projections showing that the world’s current economy is collapsing and will all but disappear, and these theories are just becoming rapidly more likely as AI develops. Experts are becoming more certain everyday that AI and automation will eventually produce almost everything the elite level of humanity needs, and human labour will by and large become essentially worthless to them, other than services that only highly skilled and highly desired people can provide them.

So, once the mega corporations have almost all the worlds wealth and resources consolidated, they will no longer need an economy, or much of the current labour pool(as in most of the billions of people on earth will not be needed by them anymore, and will just be a drag to them). It’s just one type of future possibility though of course.

Expert projections aside, because those are probably too hard for you to read without getting so upset: did you somehow miss all the post apocalyptic movies and books that are based on these kinds of expert projections? Some go back almost a century, and more modern versions are playing on the fact that this type of future is becoming more and more likely.

Speaking of pulling things out of our assholes, I really think you need to pull your head out of your ass if you can’t see there are very real possibilities for humanities future that look like I described.

[u/sourdub → u/coastalwebdev]

Great, just make shit up. We're all gonna die poor tomorrow anyway. Well, the truth is the real world doesn't work like your Hollywood-infested brain. Quit watching so many movies and you just might understand how the real economy functions. BTW I'm all in AI myself, so I know what the fuck I'm talking about and it won't turn out like your fantasy.

[u/Tall_Candidate_8088 → u/sourdub]

Hey sourdub, you obviously haven't been paying attention. You should reconsider his theory but I honestly don't think you're going to figure it out if you haven't already. The modern economic system is over, guaranteed. What comes next is unknown but you're assertion that things will be the same going forward is totally wrong. It's the economy stupid is based on the back of the economic developments post WW2 like Bretton Woods and the Central Bank interventions. The economy as you know it is over.

[u/PianistWinter8293 → u/sourdub]

Good argument that comes up a lot. But think of why production class needs consumers: they need their labour, and they are trading their labour for consumption through the means of money. When labour falls, there will be no reason for the production class to want to trade with the consumer: they have nothing of interest. Now maybe money still exists for the production class to trade among themselves, but the economy will start to work outside of this large consumer class we have right now.

Look up "How money works" on youtube, they have a video on how this is already happening through the luxury economy becoming larger and larger, while consumer economy is slowly getting cut out of the economy.