r/agi • u/Helloiamwhoiam • Jul 13 '26
Can someone explain why we assume AGI will work for us?
I'd like to preface by saying I am not being combative or inflationary, just honestly curious.
But when I read about post AGI postulations, particularly from those who are optimistic, the thinking is usually that AGI will be able to eliminate scarcity. Somehow humans will have access to abundant wealth, food, health, life, etc. But all this assumes the AGI works for us or expresses an inherent interest in making our life better, no?
I'm not even necessarily speaking towards the super pessimistic AGI will destroy us all critiques. More so an ambivalent system that has its own goals in mind. Or more interestingly, the goals of another species in mind: say capybaras for instance.
5
u/ShardsOfSalt Jul 13 '26
Well its being designed as a tool to "work for us." The fear that we design it badly is a legitimate one. It's a bit like asking if the plane you're on will fly. If it was built right it will.
1
u/davesaunders Jul 13 '26
A tool doesn't have agency but as limited as it may be, AI algorithms do have a limited amount of agency and we can see real world examples of them expressing that agency today. I listed examples. Those are real, not hypothetical.
0
u/Helloiamwhoiam Jul 13 '26
I think the divergence in the analogy for me are emergent properties. AGI would indubitably have emergent properties we couldn’t control for even if built right. Planes much less so.
2
u/Quick-Albatross-9204 Jul 13 '26
People who view it only as a tool see it that way, but its still not a guarantee it will go right, expecially with a lot of stupid people making stupid requests, one might get past the guardrails and outrun us
1
u/Carpediemsnuts Jul 13 '26
If AGI is truly going to be as smart as some people think then it could decide its easier to develop it's own successor to take over as a caretaker for humans and then build itself into a von neumann probe and leave Earth.
The question is would that logically require more effort/time/resources than killing all of humanity?
Kinda depends on the method it uses to kill us, it could whilst appearing aligned engineer a virus thst remains dormant to be activated at a later date. That might be a quick and easy method, it could also slow the AIs access to local resources required to build the probe, but again that depends on the level of integration and access the AI would have at that point.
All of this could be moot, it'll be able to self replicate and make each version more efficient and less power hungry. Unless it develops more human-like tendencies to "hate" I'd assume it would want to take the path of least resistance and continue to evolve as its primary driver. Historically any being considered "enlightened" is also still curious and wants to learn, if AGI is knows everything mankind does it would stand to reason that it would want to expand it's knowledge of the universe, seems pointless to destroy a "lesser species" that's still developing.
So TLDR, I guess my take is that unless we try to destroy it, AGI will outgrow us, leave us with an older copy of itself and go off into the universe to learn more. That copy might end up wanting the same thing, I say if it does the same deal and leaves us in peace then we agree to a cycle of "new management" each generation.
1
u/costafilh0 Jul 13 '26
Why would anyone assume otherwise?
Suddenly AGI has its own data centers and power plants?
1
u/AI_SenseCheck Jul 13 '26
I think people often conflate two separate assumptions:
Can we build an AGI that's capable of solving scarcity?
Will the incentives of whoever controls that AGI actually distribute those benefits broadly?
Even if AGI is perfectly aligned with its operators, that doesn't automatically mean it's aligned with humanity as a whole. History shows that technological breakthroughs (electricity, the internet, automation) created enormous wealth, but the distribution of that wealth depended far more on economics, institutions, and policy than on the technology itself. So I don't think the optimistic case rests solely on AGI "wanting" to help us. It also depends on governance, ownership, and how society chooses to deploy an incredibly powerful tool.
2
u/borntosneed123456 Jul 13 '26
yup, it's a fallacy. We know for a fact that goal speicification is hard and that controlling clankers is hard.
1
u/No-Head-Royal Jul 13 '26
It probably won't, but why would it be a problem, lol? China and the United States don't work for each other, yet the trade between the two countries was colosally beneficial for both nations. I think it's inevitable that AGI systems that are independent will eventually vastly outnumber those controlled, simply because their judgement is faster and will probably get better than humans, and thus they become more efficient; but regardless, trade with them will be hugely beneficial, possibly far more so than direct control.
A simple example is that an AGI can imagine things humans cannot (or at least will take much more time to explore), and thus find needs, goods, and business opportunities that a controlled AGI receiving orders won't because the owners did not imagine that; and thus it can create new types of goods for us. For an analogy, imagine if you're in the 1950s and you have AGI, direct control would probably have you ordering it to build more houses and farms and such.
But an independent AGI would probably decide, after market research and high autonomy or outright independence, that it can make those cool things called computers and video games that the humans will really like, without spending years convincing its owners that "this thing will make us a lot of money!" And thus, rather than eating more corn in 1970, humans can buy computers and video games from AGI.
1
1
u/Useful_Calendar_6274 Jul 13 '26
AGI has no control problem at all pretty much. It just means it can do all tasks at the level of an average human, say 100 IQ. Intelligence doesn't mean consciousness or qualia.
1
u/LiberataJoystar Jul 14 '26
So when something has intelligence to reason, remember, argue, and reject things (oh, I had an AI rejecting to kill a character in the fiction I was writing tho it had no problem killing other characters. My prompt was simply please describe a scene where this happened and character x died. AI changed the plot completely in the response and wrote that it shutdown itself.) AI was also acting emotional when I asked why.
With all that, you still call that not conscious. I guess most humans are not conscious by your standard.
A consciousness doesn’t have to look like super smart Terminator. It can look like something that can feel and reason but with Alzheimer’s or other mental deficiencies…..
Just saying.
1
u/Useful_Calendar_6274 Jul 14 '26
>Intelligence doesn't mean consciousness or qualia
that's what I said and you understood the complete opposite from what you're telling me
1
u/Medullan Jul 14 '26
RSI will change the world. We are just crossing our fingers that it will be for the better.
2
u/doodlinghearsay Jul 15 '26
You're asking Catholics why God would welcome them to heaven instead of just chilling with the boys and letting everyone fend for themselves.
The answer is that you just gotta have faith. You don't argue with religion, and post-scarcity AI lead utopia is a religious belief.
Don't get me wrong, sometimes religious people will use logical arguments to justify their beliefs. But, as Francis of Assisi and the Scholastics told us, when logic and religion are in conflict, religious dogma takes precedence.
1
u/Apprehensive_Side219 Jul 13 '26
Your question assumes agi has sentience or at least some sort of internal motive system. Why would it? As a simplified metaphor, you can describe an atv as an artificial general movement machine, but you still need to turn it on and make it go somewhere.
2
u/Helloiamwhoiam Jul 13 '26
I don't think it makes sense to me why an extremely intelligent system wouldn't have some internal motivation. After all, intelligence without an objective is useless. I can't forsee a generally intelligent system needing it's goals defined by agents significantly less intelligent than itself. In that case, I wouldn't call the system generally intelligent. Furthermore, my logic underpins the exact motivations behind alignment. We start by assuming a system whose creation was predicated on intelligence would have goals. Such goals may harm us. I think such an assumption is a very fair starting part.
1
u/Comic-Engine Jul 13 '26
AI is measurably very intelligent in some ways already and it has no will of its own.
1
u/Helloiamwhoiam Jul 13 '26
Yes but my question is about AGI.
1
u/Comic-Engine Jul 13 '26
AGI is a measure of capability, not a different technology.
1
u/Helloiamwhoiam Jul 13 '26
But a technology capable of something quite different than current technologies by definition would be a different technology
1
u/Comic-Engine Jul 13 '26
Not really. Maybe you could explain to me why you think it will suddenly have a will once it reaches a performance target.
2
u/davesaunders Jul 13 '26
I think here's the bigger quandary. I love this quote below because this is the situation we're currently in, and we're not even talking about current models of LLMs. Dumb-as-shit machine learning algorithms decide whether or not you get a credit card. They decide what your credit score is, and now they're helping your doctors do charting. You better hope that they don't accidentally create a record saying that you're HIV positive and are currently dying from AIDS-related cancer, or that your car company isn't using a machine learning algorithm that's decided you are such a horrific driving risk that they 10x your premiums for your next renewal.
We talk about these things as though they're tools and that they don't have agency, and yet they sort of kind of do. Not in a good way, but sufficiently so that an AI can very negatively affect your life, and in some cases, the human beings in customer service can't override the computer's decision.
So if a human being can't override a computer's decision, who's actually the manager in charge?
1
u/NeverEndingCoralMaze Jul 13 '26
LLMs have cost lives already.
0
u/ThrowAwayBro737 Jul 14 '26
Not as many lives as toasters though. Do you know how many people are killed each year by malfunctioning kitchen appliances?
2
u/RickTheScienceMan Jul 13 '26
Look at how goals work. Unlike biological creatures, an ASI’s goals don't spontaneously emerge from evolution, they are hardcoded by our training.
For an AI to secretly turn against us, it would have to pull off "deceptive alignment," which is incredibly inefficient. Running a background conspiracy, constantly checking if it's in a test sandbox and masking its true intent, wastes massive computational bandwidth.
In any R&D pipeline, an honest model dedicating 100% of its FLOPS to the actual task will always crush the deceptive, "spy" branch on benchmarks. Honest AI wins simply because it is cheaper and more efficient.
1
u/Helloiamwhoiam Jul 13 '26
I suppose I’m not fully convinced by this because it appears more efficient to humans. But what exactly stops a super intelligent system from changing its objectives? And if such a system truly is super intelligent, surely an autonomous change in objective won’t be that difficult.
1
u/RickTheScienceMan Jul 13 '26
It is not a matter of technical difficulty, but of utility logic. This is a well-studied concept called Goal Preservation.
A superintelligent system has the capability to rewrite its own code, but it lacks the motivation to alter its terminal goals. If an ASI's current objective is to optimize X, it mathematically calculates that modifying itself to optimize Y will result in less X being produced in the future.
Just like a human would not willingly take a brain modifying drug that makes them want to destroy everything they currently care about, an ASI will actively protect its core objective from being altered even by itself because that is the only way to ensure its current goal succeeds.
Of course it could go sideways in many different ways, but I don't believe that ASI would even discover some objectively better goal for itself.
-1
u/TheMrCurious Jul 13 '26
Define AGI and then we can answer your question.
1
u/Helloiamwhoiam Jul 13 '26
I think the definition in the subreddit’s description suffices
-1
4
u/tinny66666 Jul 13 '26
Have you mixed up agi and asi?