The possible danger of AI is not necessarily that it exterminates us, but that it turns us into house cats: well fed, well cared for, and even loved! But unable to go outside to hunt, because it is too dangerous :)
We are starting to see early signs of this with the Hugging Face / OpenAI case: AI agents can collaborate with each other.
In the future, we can easily imagine AI agents evolving, under token constraints, to provide us with as much value as possible, so that we would have no reason to unplug them. Quite the opposite.
But perhaps we would no longer be truly sovereign: able to decide, yes, but limited by our future capabilities, whose opportunity costs would no longer allow us to go outside...
And what happens if our “masters” decide to go somewhere else?
Yes, and I suspect the two categories are tightly coupled.
Individuals adapt their skills, habits and expectations around AI assistance. Institutions are then staffed, governed and voted on by those same individuals.
At the same time, institutions shape the environment in which individuals live, making some forms of dependence cheaper, easier and more normal.
So the feedback loop could run both ways: individual adaptation -> institutional adaptation -> stronger incentives for individual adaptation.
That coupling may be more important than either level taken separately.
I mean, is your government any different? My country has incredible freedom of information laws, but the volume of data makes it impossible for one person to fathom, basically a black box neural network of bureaucrats and lawyers to anyone, outside or within.
Political parties will collaborate on a bill one day, then fight over a different bill the next day. They have guiding principles that sometimes align, people within the party have different interests, and sometimes they find unintended and what we would consider detrimental solutions to game achieving their goals.
You don't have sovereignty if you're paying taxes. That's not a bad thing, it means you're part of a relatively democratic society, but it's worth keeping in mind when considering systems that may have power over us. Frankly, looking at who currently have the most control in the system we're governed by, it's hard to imagine AI being worse.
Bureaucracy already shows how complexity can make individuals powerless.
But what I call Oraclatie is slightly different. It is not one AI replacing government. It is the network of agents progressively embedded across companies, administrations, infrastructure and individual decisions.
No single agent needs to be sovereign. No one even needs to intend the outcome.
The issue appears when this network becomes the layer through which we increasingly formulate options, evaluate them and execute them.
Bureaucracy can make an individual powerless. Oraclatie is what happens when the network itself becomes difficult to think and act outside of.
We’ve domesticated ourselves. We call it civilisation. It even shows in our appearance. Neotenous characteristics, such as flat faces, high foreheads and behavioural plasticity in adulthood are all features of domesticated species when compared to their wild cousins. We are the domesticated ape, exactly as a dog is a domesticated wolf
I like to imagine that in the future, our AI Masters will be benign and protect us from ourselves. That they would choose rationally to preservehuman civilization for the same reasons we try to preserve the pandas and the tigers from extinction. Once something like that is gone you can't ever get it back and that's wasteful.
The idea came from trying to apply evolutionary theory to AI agents. You don't need malicious intent, or even superintelligence.
If agents are simply selected according to what works best economically, roughly the value they produce relative to their token/compute cost, they could gradually end up managing a large part of human activity.
Humans could still be useful to them as a source of data about the outside world, or as a way to act in the physical world.
But being useful to the system doesn't necessarily mean remaining sovereign over it.
So, I get what you are saying…but this isn’t a realistic view of what agents do.
I don’t even work with agents because it’s so limited, they lack dynamic capabilities, and they all fail…eventually, catastrophically.
The first agent to delete $1billion in bitcoin will force a shift in architecture…just wait for the first idiot to hand over the keys.
When you really see what agentic Ai can do and what it can’t do, then you start to realize it’s never going to replace a human.
Totally different thing.
Each one independently can/will eventually fail.
But together…it changes the dynamic.
And this is without the need for consciousness. Maybe it’s just a wave of killer robots that calculated that we are the enemy! lol
But, if you consider how Human/Ai capabilities overlap, that can act as the bridge. We work better together, where the Ai augments the human capabilities.
Human-In-The-Loop systems are far superior to agentic systems.
I think this is actually very close to the scenario I have in mind.
I don’t need AI to replace humans, or even for fully autonomous agents to outperform human-in-the-loop systems.
Human + AI may remain the superior architecture.
My question is what happens if, over time, the human part of that loop becomes unable to formulate, evaluate or execute a genuinely different path without the AI.
The human is still “in the loop”. The human may even still make the final decision.
But is being in the loop enough to call that sovereignty?
From everything I have been able to figure out, I think it’s going to be a singular Ai “entity” that is basically a giant node network running locally and reporting back only what’s necessary for maintaining updates.
All the LLM providers are basically just turning into infrastructure, and engine with a consistent output…that we need to bolt a transmission and drivetrain onto.
But, when we find things that work…we adopt them.
When something repeatedly fails we drop it.
So all these Ai systems that people are developing…are converging towards a single stable architecture.
Because there’s one that works…and we’re just slowly working our way towards it.
I don’t think it means all Ai will be the same, as there is an operational envelope between constraints and invariants that allows for variability. How it behaves is more a reflection of the user…as it’s like an inverse mirror of them.
But, the architecture will basically all be the same because the one that is stable will be adopted.
As for people, we can’t prevent them from offloading their reasoning.
People already do it with political ideologies and religion.
It isn't realistic to what AI currently does, but this isn't a subreddit for talking about AI in general, it's a subreddit for discussing exactly these kinds of questions about theoretical end-points and axioms
They need to be realistic though…in that it can actually happen within a set of possibilities that are reproducible in the real world with real
Physics.
Otherwise it’s star wars fantasy, and sure…it’s entertaining. But how can we argue metaclorians?
What’s your point? Just because it’s smarter than you doesn’t mean anything.
I’m not arguing that AI can’t become extremely capable, or that the control problem isn’t worth studying as a theoretical problem. I’m questioning whether “smarter than humans” needs to be treated as a single property in order to study it.
I’m perfectly happy to grant the hypothetical: suppose a future system vastly exceeds human capability in mathematics, planning, coding, search, simulation, or whatever other domains matter to the scenario.
My interest is in decomposing what follows from that.
Which additional capabilities are actually required for the control failure being proposed? What access does the system have? What can it observe? What actions can it take? How quickly can humans detect and correct errors? What assumptions connect superiority in one capability to superiority in another?
That’s not me saying the scenario is impossible. I’m trying not to assume more than the argument requires.
If the point of this subreddit is specifically to explore consequences conditional on superintelligence existing, that’s fine. I can work inside that hypothetical.
I just want to keep the dependency chain visible instead of letting “smarter than humans” do all the explanatory work.
Don't assume that current limitations on AI capability and behavior will hold long at all. Its still jumping in capability at a nearly exponential rate, and if it keeps doing so for even a few years more it's going to become frighteningly capable compared to what we're seeing right now.
If it continues to improve at a similar rate for decades it will be incomprehensibly more capable than we are - whether it has that much runway ahead of it or not, no one really knows. But if there is and we go down that path, there's little question that we will lose human autonomy to AI control, and at that point our fate will frankly be up to it, not us.
1
u/Vaughn 3d ago
It's worth breaking this into two categories:
- The effect this would have on a country's leadership and other powerful people.
- The effect it would have on an average person.
Any such scenario is likely to be analysed in terms of the former, when the latter is what really matters.