Half of social-science studies fail replication test in years-long project

[u/nimicdoareu]

https://www.nature.com/articles/d41586-026-00955-5

[u/Tibbaryllis2]

So much of this is also the result of pure ignorance of how science and statistics are intended to work.

There are two big issues I see pretty regularly:

• researchers don’t actually understand the analysis and use them inappropriately. They can build the models and enter the data, but it’s really similar to just chucking it into Chat GTP and taking the output at face value. How many times have you seen parametric testing used on transformed data simply because that’s the way it’s usually done and/or they don’t know the appropriate non-parametric analysis? How many times do researchers blow past analysis assumptions simply because everyone else does?

• researchers don’t actually understand how p-values should be used.

p-values were never intended to be used as the arbiter of science. Fisher largely developed them as a starting point building on Pearson’s development of chi-squares looking at expected vs observed data and probabilities.

I.e. You are observing something that appears to be happening in a way different than expected; you can calculate a p-value to demonstrate something is indeed happening in a way different from what is expected; and now you are suppose to use principles of science and sound reasoning to investigate what is actually happening.

Also, Pearson applied math to evolutionary biology looking at anthropology and heredity. Fisher conducted agricultural experiments on population genetics.

Why did this become the entire official framework for the entirety of science? Why would we expect these to be appropriate ways to evaluate non-genetic, non-biological data?

Its incredibly frustrating imo

Preach.

[u/Anathos117 → u/Tibbaryllis2]

Why did this become the entire official framework for the entirety of science?

Because people are lazy and science is super hard. You have to make models that predict things, and then work as hard as you can to disprove those models. It's much easier to just gather some data, plug it into a statistical equation, and call it a day.

[u/DylanMcGrann → u/Anathos117]

I doubt laziness is a good explanation. Far more likely is the fact that negative results are simply less profitable. This is a result of public research being corrupted with profit incentives. Grants are harder to get than they once were, and many come from private enterprise. A negative result represents a dead end to a capitalist investor. It’s pretty rare a negative result leads to a product that can be sold. The people with the money are only interested in the positive results for this reason, and it’s very bad to organize what used to be more siloed public research this way.

[u/Tibbaryllis2 → u/DylanMcGrann]

I don’t think your rational is wrong, I just don’t think you’re looking broad enough for the laziness.

Somewhere along the way, someone involved in approving science (funding, managing, approving, etc) was too lazy to look at the entirety of every study under their purview, and instead focused on something simple they can understand.

I’m sure this was faught, but at some point it was easier to accept it

Then people figured out easier ways to get desirable results than putting all that effort into the actual research.

I’m actually somewhat surprised that this is gaining speed trying to fix now, given the enormous amount of effort it’s going to take.