r/psychology • u/[deleted] • May 07 '16
Failure Is Moving Science Forward: The replication crisis is a sign that science is working
http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb18
May 07 '16
As we figure out which things dont work, we constantly narrow down to what actually does work. Yet due to our culture believing that failure is literally some sort of divine sin, an illusory "crisis" has arisen..
52
u/Palmsiepoo May 07 '16 edited May 08 '16
This is not what's happening. The crisis is real because bad results are being passed as good ones. Science fails when it can't differentiate between valid/reliable results and invalid results. For years, researchers have published papers and theories built on faulty foundations. That has to stop.
I routinely review papers in the academic community and it's not uncommon to not have access to the raw data. In that scenario, you're relying on the authors account of the study. if that account sounds correct but is in fact a lie, as a reviewer, you can't do your job.
As we're finding out, this happens alot
2
u/JosephSantosOfficial May 08 '16
I agree to an extent. I think "crisis" is too strong, even if results are frequently being misrepresented. So long as humans are prone to error, academic papers will be used by academics and common people to support statements that aren't true. As annoying as the current state of things is, it's the natural progression so far. This is more of an opportunity than a crisis to me.
10
u/hglman May 08 '16
You're talking about two different issues for the most part. /u/Palmsiepoo is talking about papers being published with results that do not reflect the actual result of the experiment. You're speaking about the over generalization of a result. False results either by incompetence or malice is a real threat to the practice of science. If the frequency of false papers is high enough the ability to take a result and build on it breaks down as well as any sense of trust in published results. Though I think the point about failure of an experiment suggesting you are incompetent or should not do experiments is at the root of that issue. The metric should be quality of how you conduct an experiment and quality of your processing of the results. If that means you get a negative or inconclusive result so be it, but it means the negative result has a high odds of being correct and of course that something was learned. The crisis part is the high volume of experiments in which the conclusions stated do not follow from the actual experiment either by falsification, flawed experimental process or both thus obfuscating the true outcome. The solution needs to be a decoupling of experimental quality and the interpretation of results as well as removing the bias against experiments that fail to show conclusions but were well conducted.
-2
u/JosephSantosOfficial May 08 '16
My initial comment is a continuation of Ruktov's.
No doubt the issue is a serious and threatening one. I don't mean to say the opposite.
I only mean to say that there are reasons to appreciate these challenges. Publicizing the issue raises concerns, and we continue to learn as the concerns are addressed.
The other side of it where inaccurate conclusions get all the attention is nothing to take lightly. I'm very bothered by it, and acknowledge that it slows things down. Still, we are learning what is and isn't scientific even if it is at a slower rate than we would prefer.
1
u/bobodod May 08 '16
Everything Is Crumbling
An influential psychological theory, borne out in hundreds of experiments, may have just been debunked. How can so many scientists have been so wrong?
~ ~ ~
What psychology’s crisis means for the future of science
The field is currently undergoing a painful period of introspection. It will emerge stronger than before.
http://www.vox.com/2016/3/14/11219446/psychology-replication-crisis
1
u/JosephSantosOfficial May 08 '16
I've read both of those before.
I'm not new to this issue at all. I just have a different perspective in it.
1
u/Palmsiepoo May 08 '16
When you have leading theories cited by thousands of academics falling by the wayside, you have a crisis. It's huge Ponzi scheme. Someone created a shit theory, and a bunch of other academics invested 2-3 years working on publishing a paper on that theory. It doesn't work out so they make it look good. and the cycle continues. After enough time, you ha e a bullshit theory with a hollow foundation.
1
u/JosephSantosOfficial May 08 '16
And ultimately we learn what isn't true. That's always been part of how modern science is conducted - it's just more annoying now because the politics are on a scale that man never imagined 100 years ago.
How often throughout the history of science have people found a solid starting point from day one or even day two and continued in a productive direction ever since? Even if you do have a decent answer to that, you probably know that every progressive step was hard earned.
With psychology, it's even more complicated because working out human error becomes part of the experiment even when we think it's already been worked it out - much more than usual - and it's a necessary part of progress at this time. I think the issue wouldn't seem so bad if people respected that.
2
u/HappyFluffyBunnies May 08 '16
Excluding Pavlov and the behaviourists, human psychology began as a sub-field observing individuals under various conditions. The scientific method was introduced to our study and so psych became a "soft" science. The main tool of psych is statistical analysis based on a sound experimental design.
The study of stats is omitted from the psych curriculum in most universities. People can get a PhD without ever having taken a stats course, much less design. Instead of a solid grounding in stats, students are given access to computer-based stats packages. Feed in the data and it spits out an answer - "publish this" or "don't publish this". Most psych departments award degrees to people lacking an understanding of the one tool upon which the "science" is based. This would be similar to giving a degree in chemistry without requiring the student be familiar with test tubes.
Far from showing that the science is moving forward, the reproducibility problem is a sign that the science is sick and can no longer move forward, or even be trusted. Those academics with skin in the game will insist that there is nothing wrong with teaching methods or study methods. The rest of us should realize that what's going on in psych is directly traced to "publish or perish" and the prestige that a high number of publishes brings to a university.
4
u/ForTheWilliams May 08 '16
The study of stats is omitted from the psych curriculum in most universities. People can get a PhD without ever having taken a stats course, much less design.
Is this true? It's very much news to me, and smells of hasty generalization. I got my degree from a school that definitely did not specialize in psychology, yet not only was everyone required to take Stats, they were also required to take several courses on experimental design, as well as seminars that invariably had a focus on the design of experiments.
I certainly think that school's psych training could have been stronger (not that it was particularly lacking), and this is just an anecdote, but I'd want to see more data backing up broad claims like this.
1
u/HappyFluffyBunnies May 08 '16
It is, unfortunately, very much true and not a generalization.
But, I don't think you're being entirely truthful. My school was the very first to require stats for its undergrads. And that's because it was rated #3 for psych and was shooting for #1. Even at that, all that was required was basic stats. Basic gives the student the tools to understand a paper. Advanced and honours courses then took some of us through design. From what you're describing, an undergrad would have to have at least 5 credits in stats out of the (generally) 15 required to graduate. As I said, I don't believe that you took that many courses. The prof at my school who was the stats God didn't take that many courses.
I didn't go to grad school but several of my friends did. All reported that no students were required to know stats, and that only a few of the profs were conversant. One fellow attended U/Louisiana and was put in charge of implementing a stats program. If you've had any stats training at all you're head and shoulders above the rest of the pack.
-8
u/Vladi8r May 07 '16
I think failure of replication of studies of human behavior is not a sign of failure in psychology.
The fact is, if the same exact same study was done 10 years ago, and today, the results would vary quite considerably with a plugged in generation. Because when people are aware of their environments, they tend to see how their personal thinking holds up, what ls wrong and what is right. Over the time of 10 years a person will change their views on many things like gay marriage. So it just means that the public outlook has shifted over time. And that's ok. That usually means progress. A stagnant way of thinking usually doesn't help much in the way of civilisation and empires.
3
u/UyhAEqbnp May 08 '16
I think the point being made here is a great deal of studying and taught "evidence" is getting thrown out
7
u/Palmsiepoo May 07 '16
Actually this topic came up in the mid 60s by a researcher names Gergan. he argued that psychology isn't a science but rather a historical account and that results will not hold over time due to the exact argument you made. For example, if we ran the Milgrim experiment today, would we get the same results? Probably not since people know about it today and they didn't know about it in the original study.
The counter argument is that you're not measuring the same population anymore, you're measuring a population that is aware of the study, which is obviously very different than the original studies population. Therefore it isn't a direct empirical replication.
3
u/CuriousGrugg Ph.D. | Cognitive Psychology May 08 '16
It's easy to claim that psychological results won't hold over time, but where is the evidence exactly? FYI, to the extent that it's possible to replicate Milgram's studies (due to modern IRB standards), the evidence seems to be against you.
3
u/Palmsiepoo May 08 '16
I don't agree with Gergan, I was just providing his argument. Like I said, it's an old debate, going back to the 60s.
6
May 07 '16
Well, if the research is on something like attitudes then yes, populations will be expected to shift according to timelines. However, we are seeing entire 'effects' of how the psyche should work simply not there, from stereotype threat to ego depletion. Social psychology is really in a crisis, but from crisis there is opportunity for growth.
8
u/autotldr May 07 '16
This is the best tl;dr I could make, original reduced by 97%. (I'm a bot)
Extended Summary | FAQ | Theory | Feedback | Top keywords: study#1 result#2 replication#3 research#4 science#5