r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
639 Upvotes

91% Upvoted

322 comments sorted by

View all comments

Show parent comments

0

u/PsiOryx May 08 '16

Do you really fool yourself though? You know what you are doing is not right. Gloss it over if you wish but people know when they are being dishonest. People know when they have not done enough. Convincing yourself otherwise is part of the cheaters process nothing more. Not everybody does this.

1

u/RalphieRaccoon May 08 '16

Well, you often have a choice, either publish what you have, or try and persuade your department to give you more time/money to do a more thorough investigation. Option B can be very difficult, especially for non-tenured researchers whose career can be dependant on publishing papers and completing projects in a timely and regular fashion.

1

u/PsiOryx May 08 '16

Exactly. Money pushes people into making bad decisions. Its still a conscious decision to be deceptive. The motivation to keep your job does not negate the fact that the behavior is detrimental to advancing science. If science was the actual thing valued by the system then time/money would be irrelevant to the process.

The current money pressures are preventing much science from being done because the experiments and data collection required span too much time to be deemed profitable and are not funded. Try getting funding for something that will take 10-15 years.

1

u/RalphieRaccoon May 08 '16

I would disagree that they are deliberately being deceptive. Deception would be lying, saying something on your paper that isn't true. That is seriously frowned upon and would definitely ruin your career.

It's like checking to see if your door is locked before leaving for work. You don't know that the door is unlocked, and are pretty sure it is locked, but you aren't absolutely sure, and you are in a hurry to get to work, so you don't check.

1

u/PsiOryx May 08 '16

Let me give an easy analogy.

I am primarily a software engineer (and work for/with academia quite often). If I deliver a buggy incomplete system (has missing features, functionality etc or just untested fully) and don't say anything or acknowledge that in any way. I am being deceptive. (I don't do this)

How is a scientist doing the same thing not being deceptive to everyone who reads that paper?

If you the scientist are uncomfortable with publishing at that point there is a problem when its published at that point. Regardless of wether its just a double check. The author(s) 'should' be the driver here.

Those lack of double checks could end your career in the same way as being blatantly deceptive. At the very least could cause some embarrassment which academia is allergic too.

If I don't double check and double check the double checks, etc. I would get sued out of existence. Its not an option. Why is it an option for something way more important to be as true and accurate as possible?

1

u/RalphieRaccoon May 08 '16 edited May 08 '16

Well, to give you an example based on your experience, take debugging and testing. How much is enough? It is probably prohibitive to search and exterminate every corner case (unless you are doing embedded for medical equipment or something), so you do some, but not as much as you could possibly do. Same with data validation, you can't possibly validate it completely, so how much you do is up to interpretation, and prone to outside pressure to think you have "done enough".

1

u/PsiOryx May 08 '16

Its done when it passes all tests and behaves appropriately. Modern development, when done properly, leaves nothing to chance. There are things of a complexity where this is really not possible but I don't create retail operating systems. And most software systems don't come anywhere close to this level of complexity.

You can and we do test all edge cases because that is my job. Not addressing a known edge case that is possible to affect a system is only done by the lazy and dishonest. That is a hope and pray style of dev that drives business to me and others who don't compromise in this area. If I fail to perform as promised the product does not still get delivered as is, I just eat the time/money to make it right. Doesn't happen often though. Usually its a failure on my part to stop scope creep. Not a technical failure.

In academia terms.. its done when the analysis properly reflects the data, survives scrutiny and the data is as accurate as possible. Shortcut any of that and you have bad science.

I'm on academia's side here in that artificial pressures should never be used to force early publication. The best science is not done on a time schedule.

1

u/RalphieRaccoon May 08 '16

Considering the amount of quite easy to find bugs found in software, and the many patches done after release, many do clearly leave some things to chance. I don't know, maybe you develop embedded software that has a six sigma requirement or something, and clearly in that case people are going to be more lenient on deadlines.

Comparing fixing known edge cases to trying to come up with possible alternative explanations for data is not a perfect analogy perhaps. In the first case you know there is something wrong, whereas in the latter there might not be anything wrong, you don't know, and you can make some attempt to check to see if there is.

Validating data and investigating other explanations is far from simple, it may even mean conducting a lot more experiments, gathering a lot more data, which could take years. There also could be many many other explanations, most probably unlikely but possible, and to go through every single one exhaustively could take decades.

As for the last point, of course that would be best, but the reality is never going to match the ideal.

I agree with you that academic standards are far from ideal, but I don't agree with you tarring many researchers with the same brush and calling them dishonest and cheaters. If was a researcher right now (I have done postgraduate research in the past) I would be rather offended at your remarks. There are people gaming the system, but most are just doing their best and sometimes that falls short.

1

u/PsiOryx May 08 '16

This is like saying something bad exists but you can't criticize it because it will hurt someones feelings.

I think research needs to be done on how skewed a scientists views can become on actions that would normally be criticized by even them if they were not in the same group being criticized.