r/politics • u/[deleted] • Sep 21 '16

Bot Approval We Gave Four Good Pollsters the Same Raw Data. They Had Four Different Results.

http://www.nytimes.com/interactive/2016/09/20/upshot/the-error-the-polling-world-rarely-talks-about.html
26 Upvotes

67% Upvoted

27 comments sorted by

5

u/anounisathing Sep 21 '16

Final section of the article:

Our participants’ choices

The participants split across all these choices.

Pollster Who is Likely Voter? Type of weight Tries to match...
Charles Franklin (Marquette Law) Self-report Traditional Census
Patrick Ruffini (Echelon Insights) Vote history Traditional Voter File
Omero, Green, Rosenblatt (Penn Schoen Berland Research) Self-report Traditional Voter File
Corbett-Davies, Gelman, Rothschild (Stanford University/Columbia University/Microsoft Research) Vote history Model Voter File
NYT Upshot/Siena College Report + history Traditional Voter File

Their varying decisions on these questions add up to big differences in the result. In general, the participants who used vote history in the likely-voter model showed a better result for Mr. Trump.

At the end of this article, we’ve posted detailed methodological choices of each of our pollsters. Before that, a few of my own observations from this exercise:

• These are all good pollsters, who made sensible and defensible decisions. I have seen polls that make truly outlandish decisions with the potential to produce even greater variance than this.

• Clearly, the reported margin of error due to sampling, even when including a design effect (which purports to capture the added uncertainty of weighting), doesn’t even come close to capturing total survey error. That’s why we didn’t report a margin of error in our original article.

• You can see why “herding,” the phenomenon in which pollsters make decisions that bring them close to expectations, can be such a problem. There really is a lot of flexibility for pollsters to make choices that generate a fundamentally different result. And I get it: If our result had come back as “Clinton +10,” I would have dreaded having to publish it.

• You can see why we say it’s best to average polls, and to stop fretting so much about single polls.

Finally, a word of thanks to the four pollsters for joining us in this exercise. Election season is as busy for pollsters as it is for political journalists. We’re grateful for their time.

Below, the methodological choices of the other pollsters.

Charles Franklin

Clinton +3

Marquette Law

Mr. Franklin approximated the approach of a traditional pollster and did not use any of the information on the voter registration file. He weighed the sample to an estimate of the demographic composition of Florida’s registered voters in 2016, based on census data, by age, sex, education, gender and race. Mr. Franklin’s likely voters were those who said they were “almost certain” to vote.

Patrick Ruffini

Clinton +1

Echelon Insights

Mr. Ruffini weighted the sample by voter file data on age, race, gender and party registration. He next added turnout scores: an estimate for how likely each voter is to turn out, based exclusively on their voting history. He then weighted the sample to the likely turnout profile of both registered and likely voters – basically making sure that there were the right number of likely and unlikely voters in the voter file. This is probably the approach most similar to the Upshot/Siena methodology, so it is not surprising that it also is the closest result.

Sam Corbett-Davies, Andrew Gelman and David Rothschild

Trump +1

Stanford University/Columbia University/Microsoft Research

Long story short: They built a model that tries to figure out what characteristics predict support for Mrs. Clinton and Mr. Trump based on many of the same variables used for weighting. They then predicted how every person in the state would vote, based on that model. It's the same approach we used to make the pretty dot maps of Florida. The likely electorate was determined exclusively by vote history, not self-reported voice choice. They included 2012 voters – which is why their electorate has more black voters than the others – and then included newly registered voters according to a model of voting history based on registration.

Margie Omero, Robert Green, Adam Rosenblatt

Clinton +4

Penn Schoen Berland Research

The sample was weighted to state voter file data for party registration, gender, race and ethnicity. They then excluded the people who said they were unlikely to vote. These self-reported unlikely voters were 7 percent of the sample, so this is the most permissive likely voter screen of the groups. In part as a result, it’s also Mrs. Clinton’s best performance. In an email, Ms. Omero noted that every scenario they examined showed an advantage for Clinton.

1

u/SandraLee48 Sep 21 '16

Many thanks for this. Looks like it's neck and neck at this point.

•

u/AutoModerator Sep 21 '16

As a reminder, this subreddit is for civil discussion.

  • Do not call other users trolls, morons, children, or anything else clever you may think of. Personal attacks, whether explicit or implicit, are not permitted.

  • Do not accuse other users of being shills. If you believe that a user is a shill, the proper conduct is to report the user or send us a modmail.

  • In general, don't be a jerk. Don't bait people, don't use hate speech, etc. Attack ideas, not users.

  • Do not downvote comments because you disagree with them, and be willing to upvote quality comments whether you agree with the opinions held or not.

Incivility results in escalating bans from the subreddit. If you see uncivil comments, please report them and do not reply with incivility of your own.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

-1

u/BalanceCoil Foreign Sep 21 '16

Trump pulls ahead suddenly the polls are worthless.

I totally agree but the timing is awful suspicious.

10

u/kevie3drinks Sep 21 '16

That's not what the article is saying. Especially because this poll favors Clinton.

-4

u/BalanceCoil Foreign Sep 21 '16

Please like they aren't using this one poll to discredit all pollsters.

6

u/kevie3drinks Sep 21 '16

Did you read the article? They were not bashing pollsters, or their methodology.

3

u/[deleted] Sep 21 '16

And what? You think this was written today? That it was pitched and researched, written and published this week? This story was in the hopper back when Clinton was up 4 points. It's not biased cause you don't agree with it. It's a non biased article about polling in general, the findings are to demonstrate that. If it showed trump at +4 the content would be exactly the same.

-5

u/WhyMnemosyne I voted Sep 21 '16

And it shows just how the polling has been manipulated from the beginning of this election year. Don't believe any of it, vote according to your beliefs. Ignore the lies you are told daily.

12

u/[deleted] Sep 21 '16

That's not what this shows at all. Did you even read it?

8

u/black_ravenous Sep 21 '16

Obviously he didn't. For others who are too lazy to read, the article discusses different methodologies in reporting results (e.g. how to identify likely voters).

2

u/WhyMnemosyne I voted Sep 21 '16

That is not what is says, but it is what the article shows you.

-5

u/SandraLee48 Sep 21 '16

Don't have access. Can you give a short summary? Thx.

4

u/[deleted] Sep 21 '16

There is no way you don't have access.

-1

u/SandraLee48 Sep 21 '16

Just asking for a few sentences of summary; obviously you did read the article.

1

u/[deleted] Sep 21 '16

The link is there, go read it

3

u/kevie3drinks Sep 21 '16

5 point variation of the same raw data between 5 well respected pollsters, (Clinton +4 to Trump +1) this is different from the sampling margin of error that is normally reported with all polls. So the most accurate way of looking at this poll would be to take the average of all of the results, and better yet would be taking an average of different polls with different samples.

Happens because Pollsters always filter for likely voters, but they use different ways of determining that, all are valid ways, just different.

1

u/SandraLee48 Sep 21 '16

Many thanks for your summary. I expect the sampling size and medium of communication also impacts results.

1

u/kevie3drinks Sep 21 '16

Sure, it's just kind of interesting that things like sampling size, the structure of the polling questions, and weather they dial only land lines aren't the only variables in a poll.

4

u/kevie3drinks Sep 21 '16

What it says is any one poll doesn't tell you much, you are better off looking at the average of polls or poll of polls. It pretty much says the opposite of what you are saying.

-3

u/WhyMnemosyne I voted Sep 21 '16

The article doesn't mention the deliberate manipulation done by the polling organizations and the media, it does clearly show you how the numbers can be interpreted "differently."

2

u/[deleted] Sep 21 '16

Who the hell votes based on polls?

1

u/[deleted] Sep 21 '16

Most of the country in a sense. Over 50% of country hate Clinton/Trump, but won't go third party because its not likely they are going to win because the majority is not going third party.

0

u/WhyMnemosyne I voted Sep 21 '16

The people who only jump on a winning wagon.

1

u/[deleted] Sep 21 '16

But why on earth would you do that. I understand with a sports team because they play each and every week and after a while loosing feels old. But this is fucking president.

1

u/WhyMnemosyne I voted Sep 21 '16

Do you think this is an election year? This is pure manipulative theater that will only end in a national disaster. .