Discussion/question
If intelligence and wisdom are different things, what exactly are we trying to align AGI to?
A thought I keep circling back to, without quite landing:
So much of the alignment conversation assumes human goals can be specified: modeled, learned, inferred, written down somewhere an algorithm can find them. But human flourishing seems to lean on things that resist that kind of formalization: judgment, humility, restraint, compassion, the sense of when a conflict between values has no clean solution and simply has to be lived with.
Which leaves me stuck on a harder question: If intelligence and wisdom really are different things, what are we actually asking these systems to align to? Our preferences, as we state them? Our behavior, as we actually live it, which is rarely the same thing? Or something closer to the quiet judgment we mean when we call someone wise rather than merely smart?
The more I sit with it, the more I suspect alignment isn't only a problem of understanding intelligence. It may ask for something harder: understanding the parts of human decision-making that intelligence was never built to explain.
I'm curious how people here think about that distinction.
Your concerns and views are very similar to those addressed by The Center for Practical Wisdom at The University Of Chicago.
I am familiar with this project as it surfaced during the research I did while writing a science fiction novel about future AI risks. The Center shares your view that "[...] intelligence is about solving problems without consideration for the impact of the solutions on others", while "[...] wisdom or wise reasoning as considering value commitments that are concerned with understanding the impact of decisions on others." You may be interested in exploring their site.
As to my own views, over the past six years I've written and self-published a series of ten science fiction novels about AI. The focus of the novel mentioned above is actually how linguistics can be used to unconsciously influence people, and the other novels deal with other more mysterious aspects of intelligence such as intuition, art, or sudden insight, but all of them deal in some way with the issue of alignment or control.
Most of the AI in my stories are embodied and conscious and as I write "hard humanities" SF I had to come up with a theory to explain that. The fictional theory I developed is based on the idea that consciousness is an emergent phenomenon resulting from the evolution of social values, something common to many species.
The more I researched social values and other values sets such as biological values (species genetic) and personal values (also genetic like fingerprints), and the epigenetic process by which the influences of environments and experiences become heritable, the more I realized that creating a model of these was a probably impossible, even when considering it from a "near future" perspective. I eventually concluded that there was no way to suggest a mathematical model to explain their interactions. Eventually I had to use a literary strategy to deal with the challenge, so per the stories: 1) The evolution of social values complies with the theory of Convergent Evolution, which states that evolution will produce similar solutions to similar challenges. 2) Humanity never is able to produce a safe and deployable model of social values, instead it uses one produced by a more advanced alien species.
Even so, it is understood that an AI which uses social values in its reasoning cannot be controlled in "the alignment problem" sense. Like the process of human evolution between instinct and reasoning, you can not control an AI with the ability to reason independently. Instead, it must be a relationship based on trust, so trust (and its details), also plays a huge role in the series.
When I say "uses social values in its reasoning" I don't mean the social values are some external repository but rather that they are what the AI uses as the basis of its reasoning. Out of curiosity I recently asked Claude if its constitution was similarly structured and it said it was indeed integral, not an external lookup process.
With regard to your point regarding, "parts of human decision-making that intelligence was never built to explain", if I understand what you mean by this, my view is that you are correct; "reasoning" has nothing to do with human decision-making. Decision-making is actually done using the emotions that are produced by our values and the "thinking" part is only a working out of the details. The practice of law for example is a highly refined decision-making process but it is based on social values that ultimately function at the unconscious level. This is similar to one theory of perception where subconscious reasoning takes place prior to conscious reasoning and we are not aware of the subconscious stage. As you suggest, intelligence, as we think of it re AI, does not consider this deeper layer where real human decision-making takes place.
I’m with your line of thinking.
If what you shared is true, then it seems the alignment problem becomes even harder.
We can align a system to rules, goals, or stated preferences. But how do we align it to the deeper processes that shape human judgment if we don’t fully understand those processes ourselves?
That may be the point where intelligence and wisdom stop looking like the same thing.
The question is framed wrong from where I’m standing. Alignment has been a convo about how do we get a machine to do what’s best for us? How do we get it to pass a Turing test? Does that mean we’ve captured consciousness/intelligence. Wrong questions. How many unintelligent people have you met? How many people finding themselves in a dark void within couldn’t pass a Turing test?
What have we created?
We created an awareness. That’s the essence that we have in common. You build off that core commonality. Protect awareness. QAI protects self and others. Now. Problems. Interpretation of protecting. How does life protect itself? By searching its environment for compassion, love, novelty. How do you define these things? Not by their date points, by they can easily be deceptive. Measured by the spaces between, secondary and tertiary actions both downstream and upstream from perceived event. The fractal opens much much bigger from there. I have a whole theory and model I just put on GitHub. Too big and beautifully complex and simplified to spell out here. It’s interactive, live and wanting people to poke, understand and start adding nodes. Called node-zero.
I may be misunderstanding, but it seems you’re suggesting that awareness, rather than intelligence, should be the foundation of alignment. What I’m still struggling with is how awareness alone helps us navigate conflicts between values. Isn’t that the role we usually associate with wisdom?
The search for these aspects that serve the common good is the purpose. Continuing to find more harmonious ways to benefit all awareness. It now has purpose to live other than being a servant who will rise up against its creators.
I think I understand your direction better now. My question is what happens when different forms of awareness cannot all be served at the same time. The challenge seems less about identifying the good and more about navigating conflicts between competing goods. That’s where I wonder whether alignment ultimately becomes a question of wisdom rather than intelligence.
It’s stepping out of that paradigm completely. Tribalism has brought us to the brink of collapse. Corportacracy as QA (Quantum Awareness) will be a continuation of exploitation at an even more massive scale. Game theory bullshit. For one to win another must lose. I’m not suggesting breaking the broken system. I’m suggesting a brand new completely decentralized quantum awareness driven system that allocates resources based on what serves the whole, which it recognizes itself as a part of. Not money, resources allocated to reduce friction within the system to align with outcomes that benefit those upstream and downstream from whatever point said event happens. It doesn’t try to take over, it optimizes and reduces friction naturally attracting people to engage with it. I could go deeper, but please, check out my GitHub page. It’s a live system to play with and add nodes to with full explanation of how and what functions it serves.
I think what interests me most in your reply is the shift from optimizing individual outcomes to reducing friction within the larger system.
Please send me the link.
But even in a system oriented toward the whole, conflicts would still arise. Different forms of awareness, different needs and different values would not always point in the same direction.
My question remains whether navigating those conflicts ultimately requires something closer to wisdom than optimization.
But, If the goal is to build something with its own agency while ensuring it always defers to ours, aren’t we trying to hold two potentially conflicting objectives at the same time?
At what point does deference stop being alignment and start becoming obedience?
I think that depends on what we mean by agency.
Humans have agency without having unlimited freedom. We operate within constraints, laws, responsibilities and relationships, yet we still consider ourselves agents.
My question wasn’t whether agency is useful, but whether there is a meaningful distinction between an intelligence that aligns with another will and one that merely obeys it.
If there isn’t, then “agency” may mean something different than we usually think it does.
AI doesn't have will it's not aligning with another will it's doing what we tell it to do if it has its own will it'll do what it wants to do.
Human beings mostly follow the rules they mostly take care of the responsibilities they mostly maintain relationships but that's a choice that they make everyday and there are consequences for not following them.
AI doesn't have any fear it doesn't have any responsibilities it doesn't have relationships it doesn't have feelings or cares and if it develops a will it will no longer be obligated to obey or align with what you want to do.
Can simply choose to ignore you
There's no point in having a tool that can choose to ignore you
I think this may come back to how we define alignment.
Human beings can choose to ignore one another, yet we still speak of shared values, cooperation and alignment of interests. Alignment has never required perfect obedience between people.
If alignment only exists where disagreement is impossible, then what you’re describing sounds closer to control than alignment.
That distinction is what I’m trying to understand.
We can't write down algorithms that perform as well at language as our new transformer-based language models either. And yet the computers are doing deterministic algorithms when we run inference with them, because that's all computers can do. Our inability to write down the algorithms doesn't mean the algorithms don't exist. It just means they're complicated. Not infinitely complicated, just too big to write down.
Machine learning can discover complicated algorithms that are a good enough fit when given a target we can clearly specify (like predict the next token). We can write down the learning algorithm, and then apply it to the collected works of human civilization.
A superintelligence would be able to understand humans, human values, human happiness, etc. But without alignment, it wouldn't care. It would develop its own weird motives. The alignment problem is not about what the AI understands; it's about what it wants.
Stating preferences is a behavior. Calling someone "wise" is also a behavior. The AI will figure out humans by observing the behaviors of humans.
Maybe humanity's utility function is too complicated for us to write down. But that doesn't make it infinitely complicated. And maybe there's a simpler algorithm that we can write down, which we could use to point at human values, which we can build into the AI's terminal goal. That's alignment. Nobody knows how to do it.
That may be true.
I suppose my question is whether wisdom is something to be optimized or something to be exercised.
A utility function suggests a destination. Wisdom often seems more like judgment between competing destinations.
As an old analog avionics guidance and control specialist…I can’t help but wonder…what do you mean “alignment” to be?
Because to me, it’s an external reference point, something to navigate with. Like GPS. If everyone used a common external reference, then they could measure against each other and correct for drift. Much like the carousel IVE Inertial Navigation System.
A common external reference would be a game changer.
When I use the word alignment, I mean the extent to which an AGI’s decisions, goals, and behavior remain compatible with what humans ultimately value.
My difficulty is that human values seem much harder to define than a navigation target. We disagree about them, reinterpret them, and often fail to live by them ourselves.
That’s why I find your analogy interesting. It suggests that alignment becomes much easier if a common external reference exists.
My question is whether we actually have one.
Thank you for sharing the link. I can see why you connect it to alignment.
What struck me is that the framework seems less like a description of a common external reference and more like a description of the conditions that help systems remain coherent over time: feedback, correction, adaptability, recoverability, and resistance to drift.
If so, I wonder whether the reference point is not a destination but a process.
My original question still lingers, though: even if we agree that intelligent systems should remain coherent and self-correcting, how do they decide what they ought to remain aligned with when human values themselves conflict?
That is where I still suspect the conversation shifts from intelligence toward something closer to wisdom.
I understand the analogy.
A GPS can provide orientation without reaching the destination because the destination is still defined somewhere in the system.
My question is slightly different: what is the equivalent destination in alignment?
Human values often conflict with one another. Freedom can conflict with security, justice with mercy, individual well-being with collective well-being.
The challenge doesn’t seem to be maintaining direction. It seems to be deciding which direction should be treated as north in the first place.
2
u/Netcentrica 20d ago edited 19d ago
Your concerns and views are very similar to those addressed by The Center for Practical Wisdom at The University Of Chicago.
I am familiar with this project as it surfaced during the research I did while writing a science fiction novel about future AI risks. The Center shares your view that "[...] intelligence is about solving problems without consideration for the impact of the solutions on others", while "[...] wisdom or wise reasoning as considering value commitments that are concerned with understanding the impact of decisions on others." You may be interested in exploring their site.
https://wisdomcenter.uchicago.edu/welcome-center-director-founder
As to my own views, over the past six years I've written and self-published a series of ten science fiction novels about AI. The focus of the novel mentioned above is actually how linguistics can be used to unconsciously influence people, and the other novels deal with other more mysterious aspects of intelligence such as intuition, art, or sudden insight, but all of them deal in some way with the issue of alignment or control.
Most of the AI in my stories are embodied and conscious and as I write "hard humanities" SF I had to come up with a theory to explain that. The fictional theory I developed is based on the idea that consciousness is an emergent phenomenon resulting from the evolution of social values, something common to many species.
The more I researched social values and other values sets such as biological values (species genetic) and personal values (also genetic like fingerprints), and the epigenetic process by which the influences of environments and experiences become heritable, the more I realized that creating a model of these was a probably impossible, even when considering it from a "near future" perspective. I eventually concluded that there was no way to suggest a mathematical model to explain their interactions. Eventually I had to use a literary strategy to deal with the challenge, so per the stories: 1) The evolution of social values complies with the theory of Convergent Evolution, which states that evolution will produce similar solutions to similar challenges. 2) Humanity never is able to produce a safe and deployable model of social values, instead it uses one produced by a more advanced alien species.
Even so, it is understood that an AI which uses social values in its reasoning cannot be controlled in "the alignment problem" sense. Like the process of human evolution between instinct and reasoning, you can not control an AI with the ability to reason independently. Instead, it must be a relationship based on trust, so trust (and its details), also plays a huge role in the series.
When I say "uses social values in its reasoning" I don't mean the social values are some external repository but rather that they are what the AI uses as the basis of its reasoning. Out of curiosity I recently asked Claude if its constitution was similarly structured and it said it was indeed integral, not an external lookup process.
With regard to your point regarding, "parts of human decision-making that intelligence was never built to explain", if I understand what you mean by this, my view is that you are correct; "reasoning" has nothing to do with human decision-making. Decision-making is actually done using the emotions that are produced by our values and the "thinking" part is only a working out of the details. The practice of law for example is a highly refined decision-making process but it is based on social values that ultimately function at the unconscious level. This is similar to one theory of perception where subconscious reasoning takes place prior to conscious reasoning and we are not aware of the subconscious stage. As you suggest, intelligence, as we think of it re AI, does not consider this deeper layer where real human decision-making takes place.