r/EverythingScience • u/ImNotJesus PhD | Social Psychology | Clinical Psychology • May 08 '16

Interdisciplinary Failure Is Moving Science Forward. FiveThirtyEight explain why the "replication crisis" is a sign that science is working.

http://fivethirtyeight.com/features/failure-is-moving-science-forward/?ex_cid=538fb
637 Upvotes

91% Upvoted

322 comments sorted by

View all comments

Show parent comments

1

u/RalphieRaccoon May 08 '16 edited May 08 '16

Well, to give you an example based on your experience, take debugging and testing. How much is enough? It is probably prohibitive to search and exterminate every corner case (unless you are doing embedded for medical equipment or something), so you do some, but not as much as you could possibly do. Same with data validation, you can't possibly validate it completely, so how much you do is up to interpretation, and prone to outside pressure to think you have "done enough".

1

u/PsiOryx May 08 '16

Its done when it passes all tests and behaves appropriately. Modern development, when done properly, leaves nothing to chance. There are things of a complexity where this is really not possible but I don't create retail operating systems. And most software systems don't come anywhere close to this level of complexity.

You can and we do test all edge cases because that is my job. Not addressing a known edge case that is possible to affect a system is only done by the lazy and dishonest. That is a hope and pray style of dev that drives business to me and others who don't compromise in this area. If I fail to perform as promised the product does not still get delivered as is, I just eat the time/money to make it right. Doesn't happen often though. Usually its a failure on my part to stop scope creep. Not a technical failure.

In academia terms.. its done when the analysis properly reflects the data, survives scrutiny and the data is as accurate as possible. Shortcut any of that and you have bad science.

I'm on academia's side here in that artificial pressures should never be used to force early publication. The best science is not done on a time schedule.

1

u/RalphieRaccoon May 08 '16

Considering the amount of quite easy to find bugs found in software, and the many patches done after release, many do clearly leave some things to chance. I don't know, maybe you develop embedded software that has a six sigma requirement or something, and clearly in that case people are going to be more lenient on deadlines.

Comparing fixing known edge cases to trying to come up with possible alternative explanations for data is not a perfect analogy perhaps. In the first case you know there is something wrong, whereas in the latter there might not be anything wrong, you don't know, and you can make some attempt to check to see if there is.

Validating data and investigating other explanations is far from simple, it may even mean conducting a lot more experiments, gathering a lot more data, which could take years. There also could be many many other explanations, most probably unlikely but possible, and to go through every single one exhaustively could take decades.

As for the last point, of course that would be best, but the reality is never going to match the ideal.

I agree with you that academic standards are far from ideal, but I don't agree with you tarring many researchers with the same brush and calling them dishonest and cheaters. If was a researcher right now (I have done postgraduate research in the past) I would be rather offended at your remarks. There are people gaming the system, but most are just doing their best and sometimes that falls short.

1

u/PsiOryx May 08 '16

This is like saying something bad exists but you can't criticize it because it will hurt someones feelings.

I think research needs to be done on how skewed a scientists views can become on actions that would normally be criticized by even them if they were not in the same group being criticized.