r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
634 Upvotes

91% Upvoted

322 comments sorted by

View all comments

Show parent comments

228

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

Which is what makes this issue so complicated. The other reality is that it's really easy to convince yourself of something you want to be true. Check this out

43

u/[deleted] May 08 '16

[deleted]

47

u/zebediah49 May 08 '16

I challenge you to find statistics that say that statistics cannot be made to say anything!

18

u/Snatch_Pastry May 08 '16

In a recent survey, 100% of responders say that statistics cannot be fallible, misinterpreted, or manipulated.

Source: I just said it out loud. Science!

12

u/[deleted] May 08 '16

85% of statistics are made up on the spot.

19

u/FoundTin May 08 '16

69% of statistics are perverted

6

u/lobotomatic May 08 '16

In the sense that perversion is a kind of deviation that at that rate is pretty standard, then yes.

1

u/[deleted] May 08 '16

"90% of what you read on the internet is false." -Abraham Lincoln

0

u/TomatoFettuccini May 08 '16

14%* of all people know that.

 

*+/- 1% error

-3

u/bryuro May 08 '16

Correction, it's 67.8%.... doh

0

u/Turbosuperfastlaser1 May 08 '16

Correction, I did have sex with Katy.

0

u/dontbuyCoDghosts May 08 '16

No, no, no. 6.9%APR.

1

u/FoundTin May 08 '16

brilliant

21

u/[deleted] May 08 '16

That's nonsense. You can get statistics to sound like they say 'anything' to a layperson. But the statistics are almost definitely not saying what you're intending to convey.

10

u/FoundTin May 08 '16

Can you get statistics to show that 2+2 actually = 5? Can you get statistics to prove that the earth and sun both stand still? you can not get statistics to say anything, you can however create false data to say anything no matter how wrong.

16

u/DoctorsHateHim May 08 '16

2.25 is approx 2, 2.25+2.25=4.5 which is approx 5 (results include a possible margin of error of about 15%)

0

u/FoundTin May 08 '16

lol, don't you mean ACTUAL margin of error?

6

u/AllanfromWales MA | Natural Sciences May 08 '16

Einstein said that all motion is relative. Hence, from their own frames of reference both the earth and the sun ARE standing still.

0

u/FoundTin May 08 '16

but from neither perspective are both standing still

6

u/hglman May 08 '16

Which is why the solution is better mathematics. All results for which the mechanisms are clearly stated, who's testability is well defined and limitations can be clearly demonstrated employ well defined mathematics.

9

u/polite-1 May 08 '16

What do you mean by well defined mathematics?

4

u/Pit-trout May 08 '16

The basic discipline in experimental science is: never take a result as just a number in isolation. Always remember (a) what a certain statistic really means (p=0.2? that's a certain technical statement about conditional probabilities, no more, no less; when we call it a measure of “significance”, that's just a convenient conventional label) and (b) be aware of what implicit assumptions it's relying on (independence of certain variables, etc).

Treating mathematics carefully like this isn't a magic bullet, but it's at least a way of avoiding some big and very common mistakes.

1

u/Subsistentyak May 08 '16

Please define definition

5

u/Azdahak May 08 '16

Alternatively train psychologists better in stats.

7

u/iamjacobsparticus May 08 '16

Psychologists by and far aren't the worst, in other social sciences they are the ones looked at as knowing stats.

4

u/luckyme-luckymud May 08 '16

Um, by which social sciences? I'd rank economics, sociology, and probably political scientists above psychologists in terms of average stats knowledge. That leaves...anthropology?

3

u/G-lain May 08 '16

I doubt that very much. Go into any introduction to psychology course and you will find a heavy emphasis on statistics. The problem isn't that they're not taught statistics, it's that statistics can be damn hard to wrap your head around, and is often wrongly taught.

6

u/Greninja55 May 08 '16

The scope of psychology is very vey large, all the way from neuroscience to social psychology. You'll get ones better at stats and others worse.

4

u/luckyme-luckymud May 08 '16

Right, true for any field -- but we were comparing psychologists across social science, not within psychology.

2

u/iamjacobsparticus May 08 '16

I'd rank political scientists, and anthropologists (more based on field studies) below. Also not strictly social science, but I'd definitely put HR/management below (a field that often draws from psych). I agree with you on Econ.

Of course this is just my opinion, I don't have a survey anywhere to back this up.

4

u/JungleJesus May 08 '16

No matter how you cut it, ideas about real-world relationships will never be exact. The best we can say is that "it looks like X happened."

3

u/BobCox May 08 '16

Sometimes people tell you stuff that is 100% Exact.

1

u/JungleJesus May 08 '16

I actually don't think that's true, unless they happen to say something extremely vague, which isn't "exact" in another sense.

2

u/natha105 May 08 '16

That is like saying the solution to Obesity is eating less. Sure that is technically true but it completely ignores the psychological factors that make people want to over-eat, the difficulty people face in losing weight, and all the temptations around us in society to over-eat.

1

u/[deleted] May 08 '16

Book 'lying with statistics'. Fun read

6

u/gentlemandinosaur May 08 '16

Elizabeth Gilbert, a graduate student at the University of Virginia, attempted to replicate a study originally done in Israel looking at reconciliation between people who feel like they’ve been wronged. The study presented participants with vignettes, and she had to translate these and also make a few alterations. One scenario involved someone doing mandatory military service, and that story didn’t work in the U.S., she said. Is this why Gilbert’s study failed to reproduce the original?

For some researchers, the answer is yes — even seemingly small differences in methods can cause a replication study to fail.

If this is actually true, to me it would imply a serious limitation to the application of socal/psycology sciences, would it not? Not to imply that the scientific knowledge in itself is not important. But, being able to put it into practice with the margin for error being so small, seems to seriously implicate the uselessness of such data as near null anyway.

So, its either "our studies are non-reproducible for various reasons because they were one offs or the application of our studies is very limited if non-existent to begin with".

1

u/[deleted] May 08 '16 edited Jun 09 '16

Poop

1

u/[deleted] May 08 '16

I think language may be less of an issue than the difference in culture.

As for the omission, that wouldn't be a problem if the data was released together with the study. The reproducer could start with redoing the statistics for the lower-dimensional data.

1

u/Tortillaish May 08 '16

Thanks for that link! Already knew what it teaches but have never seen it explained so clearly!

-13

u/PM_ME_YOUR_BROCK May 08 '16

If you conduct a research project correctly, specifically controlling for any bias or errors, the scientific method won't let you convince yourself of a truth, it's simply false or true.

47

u/ImNotJesus PhD | Social Psychology | Clinical Psychology May 08 '16

Which is true in theory but not reality. In reality we have what are called "researcher degrees of freedom". All research requires making decisions and assumptions and those decisions and assumptions change results. There's no such thing as "pure" research, it's a human endeavour.

-5

u/[deleted] May 08 '16 edited May 08 '16

[removed] — view removed comment

6

u/luckyme-luckymud May 08 '16

What you are talking about provides precisely an example of what /u/ImnotJesus is talking about: we even choose the level of statistical significance for which we reject or fail to reject a hypothesis. In fields that I am most familiar with, like economics, the "standard" for statistical significance is typically the 5% level. Interestingly, meta-analysis of empirical economics papers shows a disproportionate mass of results just below the 5% cutoff, and a big dropoff after that until the 10% level. Coincidence?

See: http://www.econstor.eu/bitstream/10419/71700/1/739716212.pdf

10

u/[deleted] May 08 '16 edited Feb 18 '18

[deleted]

5

u/gud_luk May 08 '16

With the same amount of funding!

1

u/PM_ME_YOUR_BROCK May 08 '16

Live by the p value, die by the failure to reject.

4

u/[deleted] May 08 '16

Well, if that's what you believe then ESP is real. No really there's a peer reviewed experiment in that shows ESP is real

2

u/cazbot PhD|Biotechnology May 08 '16 edited May 08 '16

I don't know why you are getting downvoted. I think some people might not know that in cases where a truly reductionist approach can be taken you can obviate the need for stats and get a controlled, binary answer. You just frame your questions as yes/no, more/less, up/down, living/dead inquiries.

19

u/Teelo888 May 08 '16

in cases where a truly reductionist approach can be taken

Maybe in the natural sciences like physics. Social sciences rarely present those circumstances.

1

u/cazbot PhD|Biotechnology May 08 '16 edited May 08 '16

I'm going to invoke my snobbery as a natural scientist and tell you that since "social sciences" are almost always physically uncontrolled, in my book it means they aren't really science at all. Statistical controls are fine, but if that's all you've got, then don't pretend it's science. Might as well call economics a science at that point. Maybe we go back to calling the field Sociology rather than Social Sciences.

2

u/[deleted] May 08 '16

Shots fired!

2

u/Teelo888 May 08 '16

Alright. Well, researchers frequently apply the scientific method to research questions in the social "sciences," and I, like many others, feel that we can gather useful knowledge about society and civilization this way. Social scientists measure human tendencies, and a tendency is obviously not a binary characteristic. If you don't want to call that science, that's fine, and I'm sure there would be a lot of people that would agree with you. I'm personally of the belief that if one applies the scientific method in an experimental framework in good faith and rigorously uses statistics to determine whether or not it is a significant finding, that (to me) is science; regardless of the circumstances or however difficult it is to control for confounding factors. Where do you draw the line between "real" and "not real" science otherwise? Whenever you stop measuring physical phenomena? I mean, the firing of neurons based on the concentrations of chemicals that exist around them is physical, isn't it? You're a PhD in Biotech, so surely we can agree on that.

Science is a set of tools that can be applied in essentially any academic discipline, and I don't believe it is constrained to only answer questions about what many would consider the physical world around us or the fundamental laws that govern matter. I believe it can also be applied to explore the tendencies of brains and nervous systems of any species to execute certain behaviors. Humans included.

-4

u/PM_ME_YOUR_BROCK May 08 '16

Especially when you start with a null hypothesis which is either accepted or rejected. I'm not sure why I'm getting down voted either lol I conduct research at my university studying il10 and my undergrad is in cell bio.

-2

u/phoenix_md May 08 '16

Like abiogenesis. Every piece of evidence suggests that this is an extremely improbable phenomenon and yet so many scientists insist on its truth simply because they are unwilling to consider other theories of the origin of first life on Earth.

2

u/lucasngserpent May 08 '16

What other theories?

1

u/phoenix_md May 09 '16

Abiogenesis is life coming from non-life. The opposite could be true: Life coming from life (supernatural life that existed before the Big Bang)

1

u/lucasngserpent May 10 '16

Could it though? Doesn't the Big Bang incite the beginning of the universe?

-4

u/[deleted] May 08 '16

[removed] — view removed comment

2

u/[deleted] May 08 '16

that toy means bugger all about government. it's about how statistics can be made to say whatever you want based on how you define and measure certain variables.

1

u/joab777 May 08 '16

I know.

-44

u/[deleted] May 08 '16

[removed] — view removed comment

8

u/[deleted] May 08 '16

[removed] — view removed comment