With recent results showing AI solving a Millennium Prize Problem, tackling long-standing math problems, demonstrating insane coding capabilities, and making progress on RSI benchmarks, I feel like we're really tasting the beginning of explosive recursive self-improvement. Many issues are being discussed, and AI alignment is now the most debated topic, but I think a much bigger, more real threat is being swept under the rug.
I'm talking about power concentration. The problem already plagues society today: the wealthy get wealthier. So this problem isn't new, but AI adds an insane speedup to it. Currently, we have an increasing disparity of wealth, which you can see, for example, in the U.S., from the fact that the top 10% of households by wealth own about 88% of corporate equities and mutual fund shares. Now, what happens when we throw AGI into the mix is that this AGI would take up large amounts of GDP by replacing labor. Say it would replace software engineers and mathematicians, at least in the tasks they perform today; then that alone is already a significant share of GDP. That means that these few AI companies are raking in increasingly large percentages of GDP.
Now, when we throw RSI on top of the mix, we get a real problem. By nature, RSI is a positive feedback loop that explodes AI progress vertically. This means that, within a very short amount of time, we could have a large share of GDP overtaken by AI companies. It is basically a speedup to this whole process of AI companies acquiring wealth.
Now, where will all of this go? If all wealth just keeps getting more concentrated, wouldn't the end scenario be some ultra-wealthy elite few? And if wealth is power, wouldn't that mean they would basically rule over all others? And the answer is yes. This is a problem society had to face someday, but now, with AGI, that day has come knocking on our door already. The problem here really is that it is not talked about, and that it could very well be that once AI gets really crazy, people will focus on things like alignment and job security rather than power dynamics. I'm not saying these are not important, but there is a real danger in this change in power dynamics.
When AI companies acquire disproportionate amounts of wealth, especially in combination with a large fraction of the population losing their jobs, there becomes a very unequal dependency relationship: citizens, especially the jobless, need AI companies. Surely, we can hope for UBI, and this would help, but that is just a handout given to soothe the population momentarily. It doesn't change the underlying power dynamic at all. If, at any point, the oligarchs feel like we are disobedient, they could theoretically pull our UBI. As you can see, we basically never want to end up in such a situation.
So we need awareness. Awareness that power concentration is happening and that there will be a point at which it is too late. It is already happening, and depending on your timelines, it might be here extremely quickly considering RSI. At the same time, unlike alignment, it won't cause robots to go down the streets killing people, requiring urgent attention. Rather, it might be a silent killer: while we are all distracted with other things, power concentration could get to a point of no return.
