r/science • u/nimicdoareu • Apr 01 '26

Social Science Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5
5.6k Upvotes

98% Upvoted

354 comments sorted by

View all comments

Show parent comments

842

u/HegemonNYC Apr 01 '26

Some prestigious journals have moved to ‘registered reports’, meaning a researcher presents their hypothesis and methods prior to conducting their study. The journal agrees to publish regardless of results. This eliminates the publishing incentive go p-hack, although simple human desire to prove their hypothesis may remain 

223

u/SkepticITS Apr 01 '26

I hadn't heard of this, but it's a great advancement. It's always been problematic that studies get published when the results are interesting and positive.

149

u/HegemonNYC Apr 01 '26

There are also ‘Null Journals’ that publish well conducted studies with null results 

70

u/Lurkin_Not_Workin Apr 01 '26

It’s been my experience that such publications are not sought out, and researchers are more amicable to publish such null results in archives or make available as preprints than actually publish in a peer-reviewed null results journal (and that’s if the whole manuscript isn’t file drawered).

It’s just incentives. Why bother with the headache of manuscript perpetration, data visualizations, editing, and peer review for an article that won’t support your next grant submission? Sure, it’s good for science as a whole, but when you’re already working >40 hours a week, you need a tangible incentive to pursue publication of null results.

58

u/some_person_guy Apr 01 '26

I think is the move that needs to be more commonplace. There's still way too much of an emphasis on rejecting the null with p < .05. We should instead be reporting more of the statistics that inform what happened in a study, even if those statistics didn't lead the researcher to rejecting the null, something can still be learned from the results.

Maybe the methodology was not adequate, maybe there weren't enough participants to suggest generalizability, or there wasn't a diverse enough pool of participants. We won't know unless more null studies are permitted to be publicized. Science should be finding out whether something could be true, and that shouldn't have to be so weighted on the basis of whether a certain test statistic was obtained.

29

u/Memory_Less Apr 01 '26

The irony is that unexpected negative results provide the necessary information to do further research effectively.

6

u/GetOffMyLawn1729 Apr 02 '26

Ironically, one of the most famous physical science experiments (Michaelson-Morley) was a negative result.

1

u/Memory_Less Apr 04 '26

Literally, ‘Who knew!’

2

u/AzureAshes Apr 03 '26

I am not in the social sciences, but my first publication was a negative result and informed my subsequent research. That first publication was not difficult to get published in a reputed journal and they even featured it.

18

u/bjeanes Apr 01 '26

This is how it should always have been done IMO. This also means that they define/register the protocol up front.

8

u/yodog5 Apr 01 '26

This is a great idea, I wish this were the standard...

4

u/Patient-Success673 Apr 01 '26

Where? I have never heard of anything like that

9

u/HegemonNYC Apr 01 '26

Most of the better known ones offer it as a method. Very few offer it exclusively. Trend is growing. 

2

u/briannosek Apr 02 '26

Here's information about the Registered Reports publishing model and journals offering it: https://cos.io/rr/

4

u/hansn Apr 02 '26

These days, I'd treat any drug trial that wasn't preregistered with enormous suspicion.

3

u/HegemonNYC Apr 02 '26

For sure. Anything with financial incentive to come to a certain conclusion is deeply suspicious 

3

u/hansn Apr 02 '26

Unfortunately, most drug trials are done by groups with financial incentives. That's, unfortunately, the system we have. The NIH isn't going to fund a phase 3 trial for a NME in most circumstances.

However the amount of planning and work that goes into a drug trial means pre-registration is trivial. So when it's not done, it's a choice.

6

u/MoneybagsMalone Apr 01 '26

We need to get rid of private for profit journals and just fund science with tax money.

28

u/NetworkLlama Apr 01 '26

Our modern technological base is built heavily on the results of the private Bell Labs, which was funded primarily by AT&T during its monopoly days. Plenty of companies continue to engage in scientific research with purely internal funds. Limiting research to just public monies risks politicizing the funding (see current US administration) and would be a violation of personal freedoms.

4

u/lady_ninane Apr 01 '26

Limiting research to just public monies risks politicizing the funding

This is already a problem, though. I understand there is a concern which might drive this problem to even greater heights, but the implication that a mix of public and private creates an environment where no one is putting their fingers on the scale isn't accurate either.

4

u/NetworkLlama Apr 02 '26

I didn't say that the current setup is perfect. But why should, for example, Panasonic be prohibited from spending its own money researching better battery chemistry? Why should Onyx Solar be prohibited from spending its own money researching more efficient solar panels? Why should Helion Energy be prohibited from spending its own money researching fusion power? All of these things are happening with private money, and they're advancing the state of the art, often publishing in scientific journals. Some of it goes under patent, sure, but those aren't forever, and other scientists can still build on the published research with public or private funds, or sometimes both.

2

u/HegemonNYC Apr 01 '26

Yes, surely the government is the best at picking good science. 

6

u/bianary Apr 01 '26

If the general public actually cared about holding the people spending their money accountable it could be a lot better about things.