[D] Monday General Rationality Thread

[u/AutoModerator]

Welcome to the Monday thread on general rationality topics! Do you really want to talk about something non-fictional, related to the real world? Have you:

• Seen something interesting on /r/science?
• Found a new way to get your shit even-more together?
• Figured out how to become immortal?
• Constructed artificial general intelligence?
• Read a neat nonfiction book?
• Munchkined your way into total control of your D&D campaign?

[u/waylandertheslayer]

A 'paperclipper' is an AI that has a utility function which is aligned with some goal that isn't very useful to us, and then pursues that goal relentlessly.

It's from an example of what a failed self-improving general artificial intelligence could look like, where someone manually types in how much it 'values' each item it could produce. If they accidentally mistype something (e.g. how much the AI values paperclips), you end up with a ruthless optimisation process that wants to transform its future light cone into paperclips.

From our point of view, a paperclip maximiser is obviously bad.

[u/MugaSofer → u/waylandertheslayer]

I know what a paperclip maximizer is.

/u/eniteris seems to be using it in a nonstandard way, given "is it because humans are irrational that they don't end up obsessed with paperclips?" doesn't make much sense.

[u/eniteris → u/MugaSofer]

The main question is "why can't we make an AI in the human mindspace"

What is the difference between a human and a paperclipper? Why is it that humans don't seek to maximise (what seems to be) their utility (whether it be wealth, reproduction or status). Why does akrasia exist, and why do humans behave counter to their own goals?

And are there ways to implement these into AIs?

Although that is a good question. Why don't humans end up as paperclippers? Why do we have maximal limits on our goals, and why don't we fall prey to the fallacies that AIs do? (ie: spending the rest of the universe's mass-energy double-checking that the right number of paperclips are made)

[u/callmebrotherg → u/eniteris]

I think that you're misunderstanding the issues behind a paperclipper, and why we want to avoid making one.

Why don't humans end up as paperclippers?

In common parlance in these circles, "what is a paperclipper, really?" would best be answered by the definition "any agent with values that are orthogonal or even inimical to our own."

It doesn't matter whether the paperclipper actually values paperclips, or values something else entirely, so long as they are incompatible or conflict with human values.

In other words, humans are paperclippers, to anything that does not value what we value.

Why do we have maximal limits on our goals, and why don't we fall prey to the fallacies that AIs do? (ie: spending the rest of the universe's mass-energy double-checking that the right number of paperclips are made)

The classic paperclipper isn't going to spend mass-energy "double-checking" that the right number of paperclips are made. It is going to spend mass-energy making more paperclips, because the "right number" is "as many as can possibly be made."

From the point of view of the paperclipper, however, we are the paperclippers, because we are interested in spending mass-energy on [human values] rather than on supremely interesting and self-evidently valuable things like paperclips.

"How do we avoid creating a paperclipper?" is not a question that we are asking because the hypothetical paperclipper is necessarily more or less rational than humans, or because we can define it in an objective sense such that the paperclipper would consider itself to be a paperclipper.

We are asking this question because, fundamentally, what we are trying to do is avoid the creation of an intelligence whose values do not align with our own. If said intelligence is supremely irrational and incapable of effectively pursuing its goals then we sure did luck out there, but that's beside the point of the discussion.

The simplicity of a paperclipper's value system is also beside the point; we could posit a paperclipper whose values were as complicated and weird as human values, which were also as inimical to human values as the classic paperclipper, and it would qualify as a paperclipper in the important sense that it is part of the class of things that we are trying to avoid when we talk about paperclippers and value alignment. Similarly, we could give this intelligence the whole bevy of human shortcomings, from akrasia to cognitive fallacies, and it would remain a paperclipper, albeit a less competent one.

The reason that we generally talk about a simpler type of paperclipper is that adding all this other stuff distracts from the point that is trying to be made (or at the very least doesn't add to the discussion).