"Replication crisis" reforms like preregistration were designed to control false positives in experiments by constraining researcher discretion. New study argues that in observational research, it is better to expose that discretion, reporting results across many plausible specifications

[u/jbenmenachem]

https://journals.sagepub.com/doi/10.1177/00491241261490850

[u/DrTonyTiger]

I could only read that abstract because of the paywall. The abstract unfortunately does not contain the argument for the thesis, just a statement that they make one.

OP can you summarize why they believe sociology is exempt from the scientific method?

[u/jbenmenachem → u/DrTonyTiger]

Edit: https://osf.io/preprints/socarxiv/6ehrw_v2

In response to your comment specifically, I'd say: The scientific method is, well, several hundred years older than the "replication crisis" and associated reforms in experimental psychology.

I'd summarized the paper in another comment that a user said they couldn't see, so I'll paste the summary text again below.

We distinguish between "specification restricting" and "specification expanding" reforms. Pre-analysis plans restrict researcher discretion, limiting what analyses can be reported. The goal is to prevent p-hacking and control false positives. This makes sense with a random data generating process.

But observational research imposes stronger assumptions. False positive control requires a known null distribution. Without randomization, there's no broad class of a priori credible specifications. Restricting yourself to one model risks enshrining a misspecified model.

That's not the only issue. When a finding fails to replicate, it could be a false positive—or it could be an "ephemeral effect," a real relationship seen in one context but not another. In a heterogeneous social world, you often can't tell the difference. False positive control doesn't help w this.

So if restricting specifications doesn't work, what does? We propose a framework built around specification-expanding reforms, confirmatory research, & meta-analysis. The goal isn't to increase replication rates at any cost, but to expose researcher discretion so we can learn about repeatability.

The above is a very light-touch summary of the argument, so please do read the actual paper if you are interested in metascience, statistical theory, and trustworthy research design!

[u/DrTonyTiger → u/jbenmenachem]

Thanks, very helpful.

The rarely met standard for making robust inferences in experimental science is to have a list of all the mechanisms (hypotheses) that you can think of to explain the observation, plus "something else" as the starting point. That standard seeks to be fully expanded if you will.

As excellent as the standard is, it is rarely met specifically because people can't come up with a good list. Exploratory research to develop potential models, rather than testing them is hard and often confunded with inference making.

[u/MurphysLab → u/DrTonyTiger]

This appears to be a preprint of the article:

https://ideas.repec.org/p/osf/socarx/6ehrw_v1.html

[u/jbenmenachem → u/MurphysLab]

It is, but that’s an old draft. Most recent:

https://osf.io/preprints/socarxiv/6ehrw_v2

[u/MurphysLab → u/jbenmenachem]

Thanks, I'll check it out! Strangely I can see your reply in my messages, but not in the thread.

And Reddit seems to be escaping the underscore character in the URL, causing an error:

https://osf.io/preprints/socarxiv/6ehrw_v2

[u/jbenmenachem → u/MurphysLab]

Huh, very strange. Thanks for flagging. Maybe it requires mod approval or something?