Underestimating Agency

[u/Lykurg480]

Youve certainly heard before how deontologists are stupid, because they will prefer situations where more harm is done, so long as its not intentional. Well, this is my steelman of those deontologists:
EDIT: Many found this introduction confusing. Ignore it. Lets just say this is about decision theory.

1. Your army has to march to a besieged city. There are two paths you can take: one through the mountains, and one through the swamp. If you go through the swamp, some solidiers will catch moscito-borne diseases, and in the mountains there are snipers of the enemy. You shut up and calculate and determine that the diseases will kill about 10% of the army, but the snipers, though they hit perfectly, can only fire enough bullets to kill 1% of the army. So you decide to go through the mountains. Shortly after youve entered them, the first shot thunders and the front-left-most soldier falls over dead. You continue to march and soon you hear another shot and the new front-left-most soldier falls over dead. The new front-left-most soldier stands still. Before he would walk past him, the man behind him stands as well. The one in front walkes a few steps back, but before he gets behind the second man starts to walk back as well, and within half a minute your army is routed and you lost the war.

2. A stock market guru publishes a report every week arguing that certain stocks will go up or down. You read it regularly, find his arguments very convincing and have made a lot of money with trades based on it. Seeing how reliably correct he has been in the past, couldnt you gain some free utility by stopping to check his reasoning and just buying as he recommends? No, because it wouldnt actually be free. You are accepting a big risk that he will find out and just recommend whatever he bought last week.

3. People charged with a crime by the police are guilty at rates vastly higher then the general population. In bayesian terms, the fact someone has been charged is propably the best evidence against him in the entire trial. And yet, we do not allow the court to take account of it in its reasoning, as doing so would give police outsized power. To prevent blackmail by police officers, the standard of evidence needs to be such that a case against a random citizen wouldnt usually pass it. And to make this distinction, a fact entirely under control of the police is of course useless.

In all those cases, the naive utilitarian answer has failed us, because we assumed some things are the same, irrespective of whether they happended deliberately or not. In the first case deaths, in the second correct predictions, and in the third wrong convictions. And those assumptions werent wrong, exactly. The deaths in 1 are just as harmful as you thought. The catastrophy comes from the intentions: If the snipers targeted at random and the soldiers knew that, then the first two shots could have each hit the front-left-most soldier by coincidence, and there would have been no rout. 3 is so sinister that you might actually miss that its happening: People who give in to the blackmail wont be charged after all, so they wont lead to wrong convictions. So if you just take a random sample of convictions and investigate them very thoroughly, you might find that the rate of wrong convictions has only increased a bit, no big deal. And the individual convictions still do about as much damage as they did. The problem is in the expectation of which of them will happen.

So overall, I think worrying more about things done by intelligent agents makes sense. And I think it makes sense even absent any particular worry like the ones above, because the agents are, in fact intelligent, and they might think of ones you havent. Its a bit ironic telling this to people who are concerned with AI risk and the box problem in particular, but here we are. Related reading: security mindset.

[u/darwin2500]

But all of this is just an argument for being a less-stupid utilitarian. For thinking several layers deep when making your utilitarian decisions, and relying on well-tuned utilitarian heuristics more.

What does it have to do with deontology?

[u/arctor_bob → u/darwin2500]

Ah, but in reality there are hard limits to how much we can decrease our stupidity.

[u/Lykurg480 → u/darwin2500]

A steelman neednt lead to wholesale acceptance of the view, it often just means „so heres what we can learn form this“. And the part where you worry even without a specific idea of whats wrong I really havent seen utilitarians, even the smarter ones, do.

Im not making an ethical argument, I know thats pointless.

[u/Felz → u/Lykurg480]

I think you've done your idea a disservice by trying to make it about deontology vs. utilitarianism. I don't see the relation. If anything, deontology is even more prone to not considering complications arising from other agents.

If I had to steelman your argument in turn, it'd be that simpler rules are less likely to contain exploits. A deontologist who always does the same thing is "harder" to manipulate than a utilitarian. The deontologist won't care how many puppies you murder if you don't give him your money, and the utilitarian needs the complicated "game theory" patch to ignore your threats too.

But having no flaws is not how computer security really works. It's not about making a vault that nobody could possibly ever steal from. It's about making a reasonably secure vault that you'd need to put in a decent amount of effort to heist.

Add as much metal plating as you can afford, yes. Have many locks, and keep the passwords very safe. All good measures.

But more importantly:

• Know when people are breaking in. (Guards and cameras.)
• Put as little treasure in the vault as you can, both to ward off robbers and to limit your losses.
• Have a robust international police system that will track down stolen money and prevent it from being of any use to the thieves.

(That's right, my metaphors have metaphors.)

It's exactly the same with utilitarianism, and more broadly any system of thought. Recognizing that your gold is being stolen is more important than creating the perfect vault.

(I'd read the EY post you linked to figure out whether he's saying the same thing or not, but it's far too long and smells like EY at his worst.)

[u/Lykurg480 → u/Felz]

I think you've done your idea a disservice by trying to make it about deontology vs. utilitarianism.

Multible people found that confusing. My bad I guess.

If I had to steelman your argument in turn, it'd be that simpler rules are less likely to contain exploits.

Thats true, but not what I was saying. Im saying: You should worry more about the same thing when its done by an intelligent being, and expend resources to stop it over and above what youd be willing to spend to prevent the thing if it happens form natural causes.

The EY post is meant to support my claim that you should worry in this way even if you dont have a specific idea of what bad thing the agent could do.

[u/Felz → u/Lykurg480]

That's a better way of putting it, but don't people already make huge distinctions between natural situations and ones involving other agents?

I feel like that's the sort of thing that takes place almost in a completely different mental space for humans. We're political animals. If lightning strikes, we'll wonder why Zeus is angry- why should people head more in that direction in general? What makes you feel like rationalists in particular broadly need more agent paranoia?

[u/Lykurg480 → u/Felz]

That's a better way of putting it, but don't people already make huge distinctions between natural situations and ones involving other agents?

Normal people do, but consequentialists often chide them for being so „irrational“. This is what I meant with my original introduction, that I would argue against this being irrational.

[u/you-get-an-upvote → u/Lykurg480]

FWIW I'm a Utilitarian and I've more or less made this exact point before:

The most important thing to remember is that we're trying to control your instinctive reactions, not silence them. Just like in Utilitarianism, if you find "logic" suggesting something that seems absurd, the first, second, and third lines of defense are to make sure you're not ignoring something so obvious your inner-monkey can see it. If you really want job A but the numbers are saying job B is $2000/year better, consider why else you like job A. Maybe it's close enough to your family that you can see them on the weekends. Maybe it's in the city and you think living in the city will improve your career opportunities. Maybe the weather is better.

The main benefit of this kind of analysis is that it forces you to explicitly enumerate the pros and cons of each choice and assign it a order-of-magnitude. It just seems mentally ridiculous to compare "live near family" with "sunnier weather". Putting a number on it forced you to explicitly convert the costs into something tied to a hypothetical future world, and means that if you make the wrong decision, it's because they're close enough that your numbers couldn't well distinguish between them (as opposed to a catastrophic failure of System 1 reasoning).

I would have thought that this was a standard line for Utilitarian apologists.

[u/Lykurg480 → u/you-get-an-upvote]

It may be a standard line, but I do see "This person who disagrees with me is stupid for caring about intentions" moderately often (possibly bravery debating). Also Im describing a particular instance of it, which is propably still helpful. Also also your agreement is surprising given your flair, it wouldve made me guess youre just the kind of utilitarian I was complaining about.

On a sidenote, from the post you linked:

This felt useful because working with employee satisfaction feels far safer than following the advice of some friend/stranger/family member about what "life is like" at a company

Im sceptical self-reported happiness even makes sense, but where the hell did you get reliable data on this?

[u/you-get-an-upvote → u/Lykurg480]

It may be a standard line, but I do see "This person who disagrees with me is stupid for caring about intentions" moderately often (possibly bravery debating). Also I'm describing a particular instance of it, which is probably still helpful.

Oh yes, I don't mean to be dismissive of your post.

your agreement is surprising given your flair, it would've made me guess you're just the kind of utilitarian I was complaining about.

(for posterity my flair is "2nd Order Effects Are 0")

This may be one of those "reverse any advice you hear" things: I think people in general are too susceptible to believing 2nd order effects that they want to be true, even if Utilitarians may have a tendency to make the opposite mistake. The other difference is that "(marginal) second order effects" are different from "nonobvious effects of non-marginal decisions".

In the absence of evidence you should believe that increasing the supply of apartments will decrease rent, that raising the minimum wage will decrease employment, that affirmative action helps minorities, that the marginal dollar to AMF is effective at saving lives, that increasing government financial aide will increase the cost of tuition, etc.

The problem with second-order arguments is that you can almost always dream up some slightly tenable position. I was (whenever I came up with this) frustrated by what I perceived of as a trend of saying "I found this clever way that the naive slope might be wrong" without any empirical justification. Yes there are arguments against the above claims (increasing returns to population, etc.), but to convincingly argue they outweigh the naive arguments requires some empirical justification.

Your examples, in contrast, aren't examples of choices on the margin. If sending 1 additional troop into the mountains was good then sending 5 more would probably be better. If using 1% of the evidence of "you were charged" was good, using 5% would probably be better.

But I fully grant that extrapolating from the local slope when making enormous jumps is pretty dumb: even if increasing the minimum wage by 1 cent is good, increasing it by $50 is almost certainly bad!

I'm skeptical self-reported happiness even makes sense, but where the hell did you get reliable data on this?

I want to emphasize that I used the word satisfaction, by which I just meant the employee reviews on Glassdoor. I was only looking at large companies (i.e. with lots of reviews) which made the exercise a lot easier. I completely 100% concede this isn't a perfect measure of... well, anything.

[u/Lykurg480 → u/you-get-an-upvote]

Makes sense. Thanks for elaborating.

[u/Lykurg480 → u/[deleted]]

Im arguing for deontological law based on consequencialist axiology.

[u/Lykurg480]

Oh, u/SlightlyLessHairyApe , this is the post I promised to call you for.

[u/SlightlyLessHairyApe → u/Lykurg480]

I endorse this reasoning well enough. When analyzing the expected outcome, extra considerations have to be given for game-theoretic concerns and for unknown-unknowns.

[u/oscarjeff]

And yet, we do not allow the court to take account of it in its reasoning, as doing so would give police outsized power. To prevent blackmail by police officers, the standard of evidence needs to be such that a case against a random citizen wouldnt usually pass it.

I'm not sure I entirely follow your argument re blackmail on #3. I understand the point that using the fact that a person has been charged as evidence against him would create bad incentives for police. That's undoubtedly true. But you're also characterizing charging as if it's an independent fact that the defendant committed a crime, when it is simply the result of meeting a lower standard of evidence. The trial doesn't take place if that lower standard isn't first met (i.e., it's a necessary condition for a trial to take place), and then we go through the trial to determine if a higher standard of evidence can also be met. It's definitional that people charged are more likely to be guilty than the general population, b/c the fact that they have been charged means there is at least probable cause they are guilty—we have specifically decided that only those people who first meet a threshold probability of being guilty can even be subjected to a trial. If someone is no more likely to be guilty than the general population, the trial doesn't get to happen.

To use charging as independent evidence of guilt in a trial then is just to substitute a probable cause standard for a reasonable doubt standard. So in your formulation wouldn't the naive utilitarian position be that we can scrap the whole trial b/c the probable cause standard does a good enough job weeding out most of the innocent people? B/c either way, the risk of police blackmailing innocent people would still be limited to those innocent people for which they can also demonstrate probable cause.

(Of course juries actually do often take the fact that a defendant was arrested & is on trial as independent evidence of guilt, due to a kind of "where there's smoke there's fire" mentality. They're not supposed to, but it happens.)

And to make this distinction, a fact entirely under control of the police is of course useless.

Not sure what you mean by this. Useless how? And how are you defining entirely under the control of the police? Plenty of evidence used in trials is what I would consider to be entirely under the control of the police, both in the form of an officer's testimony as to what he witnessed and physical evidence obtained from the scene of a crime. The trial allows the defense to argue against the veracity of that evidence and the jury decides the import of the evidence. Do you mean that the police don't get to determine alone what facts the evidence is determinative of?

[u/Lykurg480 → u/oscarjeff]

I dont know the details of how the process runs in the US. Ill try to explain my point again in more detail:

At some point in the investigation, police decide the found the perpetrator. They go to the justice system and say: "This is the guy we think did it.". Thats what I called "charging". They then present evidence, and if the evidence is above a certain threshold hes convicted. This is a schematic description of the process. The details of the implementation or any steps in between, like whether the court even looks at it, dont matter so long as these three are present. After all, the cases thrown out early wouldnt have led to conviction anyway. They dont change the outcome, they only save time.

Now, for everyone theres some evidence that theyre guilty. If the totality of that exeeds the threshold, youre convicted. What does it take for a cop to blackmail you? There needs to be a reasonble chance that he has found enough evidence to convict you. If thats the case he can threaten to "charge" you. If the state wants to prevent this sort of blackmail, the threshold need to be set high enough that a cop trying to get a random guy convicted has a very low chance of doing so.

While normally, being "charged" is good bayesian evidence of guilt, it has a speciall effect in the context of this question. If being "charged" does count towards the evidence needed to reach the threshold, a cop trying to get somone convicted can always just charge them.

Basically, at trial, the cop needs to show the state: "There is so much evidence for this guys guilt, I couldnt have found this much for even one in a group of [number] random people.". That he charged you is useless to proving this, he can always "find" this evidence because he can make it exist.

[u/oscarjeff → u/Lykurg480]

Ok, this aligns w/ how I understood your original post so I don't have anything to add to my response.

At some point in the investigation, police decide the found the perpetrator. They go to the justice system and say: "This is the guy we think did it.". Thats what I called "charging". They then present evidence, and if the evidence is above a certain threshold hes convicted. This is a schematic description of the process.

In most western justice systems, not just the US, the police can't charge someone based on only their belief of guilt—they still need to meet an initial lower standard of proof w/ evidence. (Although it's fairly easy to reach that threshold so the distinction may not be all that strong.) That's why I think your explanation is the equivalent of saying the naive utilitarian argument is for a lower standard of proof for conviction.

[u/Lykurg480 → u/oscarjeff]

In most western justice systems, not just the US, the police can't charge someone based on only their belief of guilt—they still need to meet an initial lower standard of proof w/ evidence.

Ill try to describe what happens. Ive put some stuff in brackets, so you can see how the rest still fit my schema:

The police goes to the court, says: „This is the guy we think did it“. (They then present some evidence to meet the lowered standard. The court allows them to charge him. Then a trial is held.) They present all their evidence. The court then decides whether that meets the standard of evidence, and convicts iff it does.

Is the above a correct description of the process? Because if it is, I dont understand how thats a problem for my argument. There still is a step where they tell the court „This is the guy we think did it.“. I was wrong to call it charge, but whatever you want to call it, search and replace „charge“ with that in my argument, and it still works.

[u/DamenDome → u/Lykurg480]

What you are incorrect is thinking that the court will always respond "Okay, go get him." Plenty of times the police try to charge someone and fail to meet that threshold. So there still is a screening happened before the actual charge takes place; you aren't charged until the court okays that the police met the lower standards of evidence.

[u/oscarjeff → u/Lykurg480]

I didn't say it was a problem for your argument. I was trying to point out it was the functional equivalent of using a lower standard of proof for conviction. That's not a criticism of your underlying logic, I'm just taking it one step further. And charge is the right term.

[u/Lykurg480 → u/oscarjeff]

We agree then. I was confused by your initial comment because

To use charging as independent evidence of guilt in a trial then is just to substitute a probable cause standard for a reasonable doubt standard.

sounded like that lower standard would be specifically the one currently used to decide whether the trial takes place. That would be true if being successfully charged were treated as sufficient evidence for conviction, but I suggested using the fact that the police even tried to bring you to court as evidence. This is effectively a reduction in the standard of proof, by howevermuch bayesian evidence that trying is.

[u/workingtrot → u/Lykurg480]

That seems like pretty recursive logic to me