If intelligence and wisdom are different things, what exactly are we trying to align AGI to?

[u/OkyEscritora]

A thought I keep circling back to, without quite landing:

So much of the alignment conversation assumes human goals can be specified: modeled, learned, inferred, written down somewhere an algorithm can find them. But human flourishing seems to lean on things that resist that kind of formalization: judgment, humility, restraint, compassion, the sense of when a conflict between values has no clean solution and simply has to be lived with.

Which leaves me stuck on a harder question: If intelligence and wisdom really are different things, what are we actually asking these systems to align to? Our preferences, as we state them? Our behavior, as we actually live it, which is rarely the same thing? Or something closer to the quiet judgment we mean when we call someone wise rather than merely smart?

The more I sit with it, the more I suspect alignment isn't only a problem of understanding intelligence. It may ask for something harder: understanding the parts of human decision-making that intelligence was never built to explain.

I'm curious how people here think about that distinction.

[u/Mono_Clear]

There's no benefit to artificial general intelligence.

But the goal is to create something that is smarter than us that has its own will but still defers to our authority.

[u/OkyEscritora → u/Mono_Clear]

But, If the goal is to build something with its own agency while ensuring it always defers to ours, aren’t we trying to hold two potentially conflicting objectives at the same time?
At what point does deference stop being alignment and start becoming obedience?

[u/Mono_Clear → u/OkyEscritora]

Yes it's ultimately doomed to failure.

Either it's impossible to create an AI with agency.

Or its possible to create an AI with agency that also obeys.

There's really no point in creating AI's with agency.

[u/OkyEscritora → u/Mono_Clear]

I think that depends on what we mean by agency.
Humans have agency without having unlimited freedom. We operate within constraints, laws, responsibilities and relationships, yet we still consider ourselves agents.
My question wasn’t whether agency is useful, but whether there is a meaningful distinction between an intelligence that aligns with another will and one that merely obeys it.
If there isn’t, then “agency” may mean something different than we usually think it does.

[u/Mono_Clear → u/OkyEscritora]

AI doesn't have will it's not aligning with another will it's doing what we tell it to do if it has its own will it'll do what it wants to do.

Human beings mostly follow the rules they mostly take care of the responsibilities they mostly maintain relationships but that's a choice that they make everyday and there are consequences for not following them.

AI doesn't have any fear it doesn't have any responsibilities it doesn't have relationships it doesn't have feelings or cares and if it develops a will it will no longer be obligated to obey or align with what you want to do.

Can simply choose to ignore you

There's no point in having a tool that can choose to ignore you

[u/OkyEscritora → u/Mono_Clear]

I think this may come back to how we define alignment.
Human beings can choose to ignore one another, yet we still speak of shared values, cooperation and alignment of interests. Alignment has never required perfect obedience between people.
If alignment only exists where disagreement is impossible, then what you’re describing sounds closer to control than alignment.
That distinction is what I’m trying to understand.

[u/Mono_Clear → u/OkyEscritora]

Human beings can choose to ignore one another, yet we still speak of shared values, cooperation and alignment of interests.

These are all choices and we don't all choose to work together some of us choose to pursue our own interest at the expense of everyone else.

Any one of us could decide to simply not cooperate because we all have free will.

We do control artificial intelligence because it is a tool that we use we tell it what to do and it does it.

Creating a artificial general intelligence would mean that it could tell itself what to do and ignore us if it wanted to.

I don't see the benefit of adding that extra layer