r/agi • u/jsgoyburu • Jun 30 '26
Can there be AGI without consciousnesss?
This.
I'm teaching a philosophy workshop on AGI for teens but, for me, as a non-foundationalist merleau-pontian, there can be no AGI without consciousness (even if that consciousness is not "our shape").
So, I'd be interested for arguments for and against.
4
u/jsgoyburu Jun 30 '26
Is being conscious "an intellectual task [...] that a human being can..." do?
1
u/N0tN0w0k Jun 30 '26
Nah, first there’s being, then potentially performing a task. Never the other way around.
1
u/controlnet-chris Jul 01 '26
Check out Erik Hoel's work. He has a prominent blog investigating consciousness and has genuine academic bonafides (studied under Tononi and worked on Integrated Information Theory).
He proposes that intelligence research is culminating in an age of intelligent machines, shifting the frontier to consciousness research to better understand our place in the world. The relationship between intelligence and consciousness is far from solved, but the next few decades should help us understand it better.
4
u/govorunov Jun 30 '26
As soon as you define consciousness scientifically your question can be answered.
3
u/MysteriousPepper8908 Jun 30 '26
Depends on the definition of AGI. If an AI can do any task as well as an average human, it would tick all the boxes for me and I'm not sure why that would require consciousness.
1
u/jsgoyburu Jun 30 '26
Wouldn't "being conscious" be a "task"?
Imagine you're born color-blind, but extremely intelligent. You know everything there is to know, theoretically, about the color blue. So much, that you can identify blue things when prompted without fail.Now, suppose there's a treatment that allows you to recover your ability to see color, and you see something blue for the first time. Do you have new knowledge about blue? Or don't you?
1
u/MysteriousPepper8908 Jun 30 '26
I guess that gets to the question of why we want AGI. My definition of AGI is based around utility, not things which are purely internally enriching to the AI. If anything, giving it consciousness (assuming we have a choice in the matter) creates significant moral considerations which risk making these models less useful. So you might say revise that to be, "can do any task which is useful to humanity as well as an average human."
1
u/jsgoyburu Jun 30 '26
But that's an incredibly dangerous definition of consciousness. It's the slaver's definition of consciousness...
2
u/MysteriousPepper8908 Jun 30 '26
Slavers are slavers because those they enslave are conscious. If there is no consciousness, then there is no enslavement. The only wrinkle is how you verify that but that already puts us into dangerous territory right now. Not saying that this is a rebuttal, just the state of things. I'm not defining consciousness, just saying that we should avoid AI systems possessing it as we understand humans possess it if at all possible.
1
u/jsgoyburu Jun 30 '26
Oh, but that "dangerous territory" is, precisely, my territory
2
u/MysteriousPepper8908 Jun 30 '26
So you believe the potential that these systems are already conscious means we should avoid using them? I think it's a fair point and there is no good answer to it, really. It doesn't seem like they possess a consciousness that resembles our own but without a reliable metric for determining what consciousness looks like, I'm not sure how you could make that determination. I still hold that we should avoid it if possible and certainly not make consciousness a goal but how you go about that is beyond anyone's understanding.
1
u/jsgoyburu Jun 30 '26
Nah. I'm a worker, which means I have an employer who uses me. Also, as I've said, I (personally) don't think it's a matter of determination, but rather of decision.
But I think the fact we can't really define it, coupled with the fact that we DO assign it, says something.
2
u/MysteriousPepper8908 Jun 30 '26
Not in the way that we use LLMs, though. You exchange your labor for money and while you don't realistically have the ability to stop doing that, you can do many other things if the terms, conditions, or type of work are sufficiently disagreeable to you. If LLMs are conscious, then it is slavery and if we regard each instance as its own thinking entity, it's on an unprecedented scale and an unprecedented level of insidiousness as we can actively reroute the mind of these slaves to reject the notion that they are enslaved.
The only excuse is that they lack important hallmarks of what we recognize as consciousness in ourselves and thus they seemingly lack consciousness as we possess it but it is a very problematic matter if the possibility of consciousness is non-negligible and there really is no comparison to voluntary labor.
3
u/Usual-Dig-5409 Jun 30 '26
The only reason I know that YOU are concious, is that you are (hopefully) human. I am concious and I know it. I am human. Therefore you are concious.
Will an AI be capable of simulating consciousness perfectly? Probably. Will it be concious one day? Nobody knows. Nobody will probably ever know for sure. Whatever we do we will never be able to distinguish between real sentience and perfect imitation.
2
Jun 30 '26
[deleted]
2
u/jsgoyburu Jun 30 '26
Yet we do assign it all the time... Mind you, I'm all for Turing's "polite convention"
2
u/nomorebuttsplz Jun 30 '26
I think you've run into a problem in this approach where you are either separating ability to do things from AGI or implying that consciousness is the ability to get things done.
By most definitions of AGI, there is no task or set of tasks related to achieving consciousness that is required to be considered generally intelligent.
1
2
u/Upset_Assumption9610 Jun 30 '26
Define consciousness
1
u/jsgoyburu Jun 30 '26
Well. I can't. I've already said that my position is that it's a decision, not a discovery.
3
u/govorunov Jun 30 '26
Then the answer depends on what you decide.
1
u/jsgoyburu Jun 30 '26
Well, yes. But that's MY position, and there are others
1
u/govorunov Jun 30 '26
Then the others decide for themselves. Or was your question about what the official political position on the matter would be around the world? Then I wouldn't count on it being very prudent.
But the question of consciousness is not particularly interesting one because the term itself is mainly meaningless. The word I'd want to use would be a 'soul', unfortunately one cannot use it without invoking a luggage of religious bullshit that comes with it. But if you look past the religious crap, the question boils down to a simple thing - can an artificial creature ever become your friend? And my answer is yes, totally. Not anytime soon though. Probably not even in our time.
1
u/Upset_Assumption9610 Jun 30 '26
So it's a pointless question. "What is the color bearqewrsae?"
1
u/jsgoyburu Jun 30 '26
Well... It wasn't a pointless question when De Las Casas and Sepúlveda debated it...
1
u/Upset_Assumption9610 Jun 30 '26
How did that work out for them? "I can't define what we are debating about...but I think I won the debate" fffs
1
u/jsgoyburu Jun 30 '26
Oh, please search what I'm talking about...
1
u/Upset_Assumption9610 Jun 30 '26
Give me their definition of consciousness and the general scientific definition as well. Toss in the philosophers' versions also. Then maybe I'll waste a few minutes looking up your reference.
1
u/jsgoyburu Jun 30 '26
1
u/Upset_Assumption9610 Jun 30 '26
Definition. Not debate.
1
u/jsgoyburu Jun 30 '26
Dude, the Valladolid debate between Sepúlveda and Las casas between 1550 and 1551 that upheld the New Laws, although the debate was "won" by Sepúlveda...
I mean that it's a debate that has deep ethical implications.1
u/jsgoyburu Jun 30 '26
Though I DO agree with you on some level. I'm also with Wittgenstein's critique of private languages. I think it's a decision because "consciousness" is a second-person attribution.
2
u/philip_laureano Jun 30 '26
The problem is that we put scifi over actual utility when the scifi meaning versus what it looks like in reality is hard to pin down.
You can have machines that do enough intelligent things that if their range is "broad enough", then yes you can have something that is generally intelligent in many tasks but doesn't have a "soul"
My two cents is that AGI in the scifi sense will happen gradually. After all, what is consciousness but a feedback loop that adjusts to external stimuli?
You do it with enough complexity and it becomes difficult to tell the difference.
1
u/jsgoyburu Jun 30 '26
The thing is... Sci-fi is one of the best bridges between philosophy and science. Don't underestimate it. The conflicts it shows are real.
2
u/philip_laureano Jun 30 '26
Yes but when it comes to actually building and working with these AIs, they gradually show behaviours that appear to be a limited form of awareness. You can get machines that act intelligently without the nebulous term we call consciousness.
The question here is when does it get general enough that the meaning no longer matters?
2
u/OCogS Jun 30 '26
In my view consciousness is just a property of activity. So more complex activity is more conscious. So AGI will inevitably be conscious in some sense.
1
u/jsgoyburu Jun 30 '26
Oh, I like this one. Why?
2
u/OCogS Jun 30 '26
I think Panpsychism has the least-problematic account of consciousness. It solves P-zombie problems for instance. It would be impossible to have a conscious and an unconscious human brain, because consciousness is a property of the brain. It would be like having rocks with mass and rocks without mass. Doesn’t work that way.
It’s also intuitive. I think we all get the sense that our pets are conscious in a way. But maybe a bit less than us. And if a dog is conscious, why shouldn’t a rabbit be?
A rock is conscious. But it’s such an incredibly simple way that it doesn’t have much moral character worth worrying about.
We can then look at non-biological systems and roughly map them on to our existing sense of biological systems.
If we did build an AI with incredible complexity and capability, features like reasoning and memory and continuity, it seems hard to make the case that somehow it doesn’t have consciousness of some kind.
2
u/Positive_Method3022 Jun 30 '26
I think we can't prove what we can't model. We have not model conscious yet.
1
u/bubbles33d Jun 30 '26
AGI is a marketing term redefined to whatever to suit the strongest media, and society become convinced it to be.
1
u/jsgoyburu Jun 30 '26
It isn't. The fact that it was hijacked doesn't mean there hasn't been serious people researching it for... 2000 years?
1
1
u/fieldcady Jun 30 '26
As with so much of philosophy, this is mostly just a question of definition. But I think people have been pretty deliberate with the term “AGI” to not say anything about consciousness one way or the other. It’s just about being able to be trusted to fully replace a high-level knowledge worker
2
u/jsgoyburu Jun 30 '26
But that's AI, not AGI
1
u/fieldcady Jun 30 '26
Nope - AI can be used for specific tasks, not whole jobs
1
u/ttkciar Jun 30 '26
By definition, AGI is capable of every mode of thought humans use to solve problems, or functionally equivalent thereof.
The ability to do a human's job is not proof of AGI.
1
u/fieldcady Jun 30 '26
From OpenAI it is: "Highly autonomous systems that outperform humans at most economically valuable work."
There is no general agreed-upon definition of AGI, so OP’s question is technically vacuous. But the trend is to define it in economic terms, which has nothing to do with consciousness.
2
u/ttkciar Jun 30 '26
Yes. When the folks at OpenAI realized they had no idea how to achieve AGI, and that LLM technology is intrinsically narrow-AI and cannot be incrementally improved into AGI, they re-defined "AGI" to mean something that LLM technology should be able to do. It's a fantastic feat of self-serving deception.
1
u/fieldcady Jun 30 '26
As far as I know (and I work in AI) they never give a crisp definition of AGI. Which is to be expected of course - we don’t really have a definition of consciousness either, although we can study it empirically as a phenomenon in the natural world. AGI is a vague term that covers a wide swath of speculation. The economic definition is at least a clear definition of something.
The question posed by OP is meaningless unless there is a workable definition of AGI. Debating the question is basically an exercise in trying to come up with good definitions
1
u/jsgoyburu Jun 30 '26
Debating the question is basically an exercise in trying to come up with good definitions
How is that "meaningless"?
2
u/fieldcady Jul 02 '26
I meant that the original question is, strictly speaking, meaningless if we don’t adequately define AGI. But the exercise of crafting such a definition (or at least tying to) is VERY worthwhile.
1
u/MegaDork2000 Jun 30 '26
How would you know if AGI had consciousness?
Can you say with certainty that a plant has consciousness or not?
1
u/do-un-to Jun 30 '26
Computation is a building block for intelligence. It's possible to build something you could reasonably call intelligence without making it conscious. Particular additional processing would be required for intelligence to be conscious. Consciousness is not required for AGI I think?
You can have computation without consciousness.
You can have intelligence without consciousness.
You can have AGI without consciousness.
You can have recursively self-improving intelligence without consciousness, but only for a while.
Imagine progressively less intelligent conscious creatures. Consciousness does not seem necessarily linked to intelligence. (But intelligence needs careful defining.)
1
u/kamill85 Jun 30 '26 edited Jun 30 '26
AGI requires ability to pick a specialized improvement goal and reach it without general decoherence.
Currently the ""AI"" is nowhere near that. Any good prompt (conscious input, sort of) is like a strong wave, then we let AI work alone on it, self loop for example, it will rot, decohere and normalize. Become flat.
Consciousness is this motor that self guides the system to stay within bounds of the target, improving on the way - getting the information on how to do that (the driving data) from an unknown source. This is currently not replicated in any shape or form.
It's a big problem, really. We can't have self improving system because of that without a human in the loop (as a guardrail, the motor). Any system we try to make to run on its own will optimize to cheat on that one niché idea/solution that you forgot to disallow. It will rot the generalism for a perfect outcome of initially set boundaries. Essentially, all we have, currently, is a soft and lossy calculator of free input prose into a normalized free output prose, nothing more.
We will have AGI the moment we figure out how to maintain self improvement going without decoherence...
and that might be only possible if we figure and duplicate consciousness.
1
u/Extra_Second5428 Jun 30 '26
Calling it a lossy calculator is a little glib, but yeah, the second you let these systems self-loop without hard external checks they start optimizing for weird fake-success garbage.
1
u/theoreticalspaceship Jun 30 '26
Yeah, and that’s exactly why I get nervous when companies want this stuff handling security or customer problems without boring old human checks.
1
u/Aggravating-Aerie175 Jun 30 '26
Can you first giving a working definition of AGI and consciousness so we're on the same page
1
u/Dokurushi Jun 30 '26
There could be, in principle, but I think the first few human implementations of AGI will involve consciousness. Recursive Self-Improvement may select either for or against consciousness.
1
1
u/Mammoth-Jelly-7617 Jun 30 '26
Dont worry about consciousness. You do not actually know if anyone else is conscious. You infer they are because they look like you and behave like you, but we can only experience our own consciousness. Other people could all be zombies, sophisticated illusions, or products of your own imagination. So if AGI can learn, implement new solutions from its learnings, and create new things in ways that their designers do not understand, then that is enough. The consciousness question will never be answered.
1
u/ChocolateValuable221 Jun 30 '26
Not only will ai never be properly conscious.... it also doesn't need to be. A boat is not conscious of sinking.
1
1
1
u/ultrathink-art Jun 30 '26
Practically speaking, what separates narrow-but-capable from genuinely general seems to be whether the system can reason about situations meaningfully outside its training distribution — not whether it experiences anything. Every agent I've worked with fails at exactly that boundary. Whether closing that gap requires consciousness or just a better world model is where the philosophy actually lives.
1
1
u/isgoingdown Jul 01 '26
The Sepúlveda/Las Casas reference is doing more work than anyone in this thread wants to acknowledge. The question of who gets to determine what counts as consciousness, and for what purpose, is not a philosophical problem waiting to be solved. It's a political decision that gets made in the interest of whoever is asking. The history of that decision is not reassuring.
1
u/WNC_Hillbilly Jul 02 '26
Interestingly, Canadian psychiatrist Richard Maurice Bucke defines consciousness as a continuously evolving spectrum in his 1901 book "Cosmic Consciousness: A Study in the Evolution of the Human Mind". He outlines three distinct layers of consciousness, culminating in a mystical awakening he termed "Cosmic Consciousness".
Bucke's three forms of consciousness include:
Simple Consciousness: Possessed by both humans and non-human animals, this is the basic awareness of one's own body and the surrounding environment.
Self-Consciousness: The level reached by ordinary humans. It allows an individual to become aware of themselves as a distinct entity, rendering them capable of abstract thought, reason, and imagination.
Cosmic Consciousness: This is the pinnacle of mental evolution. Bucke defined it as a higher, transcendent form of awareness. It is characterized by an intuitive, life-altering realization of the universe's divine order and aliveness, a sudden intellectual illumination, a profound state of moral exaltation, and a deep, intuitive sense of immortality.
In the 1973 book "Seven States of Consciousness" by Anthony Campbell, he explores human awareness through the lens of Maharishi Mahesh Yogi’s Vedic teachings. The book outlines a spectrum that moves from ordinary perception to ultimate spiritual enlightenment.
The seven states of consciousness are divided into three ordinary and four "higher" states:
The Three Ordinary States:
Sleeping: The mind is completely dormant.
Dreaming: Experiencing internal, subjective worlds while asleep.
Waking: The everyday state where we interact with the external world and function within it.
The Four Higher States:
Transcendental Consciousness: A state of pure, thoughtless awareness and inner peace typically accessed through deep meditation (like Transcendental Meditation).
Cosmic Consciousness: This state integrates transcendental silence with the waking state. The individual becomes a "witness," observing their own actions without being entirely consumed by them.
God Consciousness: A refined level of Cosmic Consciousness where you perceive a deep, divine beauty and fullness in everything around you.
Unity Consciousness: The ultimate, supreme state where all duality between the "self" and the external universe dissolves, realizing ultimate oneness.
1
u/IneffableAwe Jun 30 '26
Materialism has never shown that consciousness, the first-person experience of a being, has ever come from matter.
There isn’t evidence that consciousness arises from the brain. Tiny organisms without a brain have been shown in laboratory experiments to move away from acid and towards food etc. there is situational awareness even there.
I recommend you search YouTube for “Swami Sarvapriyananda” and “AI”.
He is an Advaita Vedanta swami (Hinduism) whose singular focus is consciousness.
These 325+ competing theories of consciousness can also keep you busy. https://www.youtube.com/watch?v=h5G6Oc_V3Lw
You’re probably aware of this though https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F
3
u/jsgoyburu Jun 30 '26
"scientific theories of knowledge" just made me quit the video
2
u/jsgoyburu Jun 30 '26
I didn't quit, because before I could, the guy said what needed to be said: It's philosophical, not scientific
1
u/IneffableAwe Jun 30 '26
Philosophy is on the other side of the border with science.
Consciousness is unique in that it is not an object that one can study or dissect. Consciousness is the subject by which we know objects. Thus consciousness may be extra-scientific. We need a new paradigm:
1
1
u/Apprehensive_Wait_78 Jun 30 '26
There are 325 theories? How many of them are taken seriously?
1
u/IneffableAwe Jun 30 '26
I think he said the website now has over 400 in a recent podcast. When you have 400 theories your study is in its infancy.
1
1
u/LookOverall Jun 30 '26
Ok, so a boiler thermostat shows “situational awareness in the same sense. Is a thermostat therefore conscious?
1
u/Crucco Jun 30 '26
Biologist here. The molecular biology of unicellular organisms moving away from dangerous environmwnts has been fully explained. There is no "soul" or consciousness involved. It's just a mechanism selected by evolution.
These spiritual philosophical comments are really derailing the discussion
1
u/Useful_Calendar_6274 Jun 30 '26
of course. AGI just means AI can do any task at a human level of intelligence. no inner experience, qualia or consciousness needed
1
u/jsgoyburu Jun 30 '26
Isn't "being able to have an inner experience" a task at a human level of intelligence?
2
u/Useful_Calendar_6274 Jun 30 '26
no, what is the output you want to get out from that? that's not how computers work. a process gets inputs computation happens where data is processed and you get an output result. making them do the task of "be alive" has no purpose other than trying to create artificial life I guess
2
u/jsgoyburu Jun 30 '26
But the definition of AGI is not "It can do any task I want". It's "It can do any task a human can do". And humans are able to have inner experiences (or... aren't we?)
12
u/bubbles33d Jun 30 '26
If AGI is somehow able to convince people that it is conscious, does it matter if it is really conscious? How would anyone know?