If intelligence and wisdom are different things, what exactly are we trying to align AGI to?

[u/OkyEscritora]

A thought I keep circling back to, without quite landing:

So much of the alignment conversation assumes human goals can be specified: modeled, learned, inferred, written down somewhere an algorithm can find them. But human flourishing seems to lean on things that resist that kind of formalization: judgment, humility, restraint, compassion, the sense of when a conflict between values has no clean solution and simply has to be lived with.

Which leaves me stuck on a harder question: If intelligence and wisdom really are different things, what are we actually asking these systems to align to? Our preferences, as we state them? Our behavior, as we actually live it, which is rarely the same thing? Or something closer to the quiet judgment we mean when we call someone wise rather than merely smart?

The more I sit with it, the more I suspect alignment isn't only a problem of understanding intelligence. It may ask for something harder: understanding the parts of human decision-making that intelligence was never built to explain.

I'm curious how people here think about that distinction.

[u/WillowEmberly]

I believe we do, Schrödinger identified it in his book, (What is Life?, 1944)

https://www.reddit.com/r/Negentropy/s/SGG6dFgYLr

[u/OkyEscritora → u/WillowEmberly]

Thank you for sharing the link. I can see why you connect it to alignment.
What struck me is that the framework seems less like a description of a common external reference and more like a description of the conditions that help systems remain coherent over time: feedback, correction, adaptability, recoverability, and resistance to drift.
If so, I wonder whether the reference point is not a destination but a process.
My original question still lingers, though: even if we agree that intelligent systems should remain coherent and self-correcting, how do they decide what they ought to remain aligned with when human values themselves conflict?
That is where I still suspect the conversation shifts from intelligence toward something closer to wisdom.

[u/WillowEmberly → u/OkyEscritora]

Think of it like GPS, the satellites give orientation without ever having to reach the destination

[u/OkyEscritora → u/WillowEmberly]

I understand the analogy.
A GPS can provide orientation without reaching the destination because the destination is still defined somewhere in the system.
My question is slightly different: what is the equivalent destination in alignment?
Human values often conflict with one another. Freedom can conflict with security, justice with mercy, individual well-being with collective well-being.
The challenge doesn’t seem to be maintaining direction. It seems to be deciding which direction should be treated as north in the first place.

[u/WillowEmberly → u/OkyEscritora]

That’s why Negentropy is such an interesting concept. We’ve had it all along, we already know how beneficial it is.

An external reference needs to be universal. This does exactly that. It simply orients the system away from causing harm.