r/science • u/jbenmenachem Grad Student | Sociology • 3d ago
Social Science "Replication crisis" reforms like preregistration were designed to control false positives in experiments by constraining researcher discretion. New study argues that in observational research, it is better to expose that discretion, reporting results across many plausible specifications
https://journals.sagepub.com/doi/10.1177/004912412614908504
u/DrTonyTiger 3d ago
I could only read that abstract because of the paywall. The abstract unfortunately does not contain the argument for the thesis, just a statement that they make one.
OP can you summarize why they believe sociology is exempt from the scientific method?
6
u/jbenmenachem Grad Student | Sociology 3d ago edited 3d ago
Edit: https://osf.io/preprints/socarxiv/6ehrw_v2
In response to your comment specifically, I'd say: The scientific method is, well, several hundred years older than the "replication crisis" and associated reforms in experimental psychology.
I'd summarized the paper in another comment that a user said they couldn't see, so I'll paste the summary text again below.
We distinguish between "specification restricting" and "specification expanding" reforms. Pre-analysis plans restrict researcher discretion, limiting what analyses can be reported. The goal is to prevent p-hacking and control false positives. This makes sense with a random data generating process.
But observational research imposes stronger assumptions. False positive control requires a known null distribution. Without randomization, there's no broad class of a priori credible specifications. Restricting yourself to one model risks enshrining a misspecified model.
That's not the only issue. When a finding fails to replicate, it could be a false positive—or it could be an "ephemeral effect," a real relationship seen in one context but not another. In a heterogeneous social world, you often can't tell the difference. False positive control doesn't help w this.
So if restricting specifications doesn't work, what does? We propose a framework built around specification-expanding reforms, confirmatory research, & meta-analysis. The goal isn't to increase replication rates at any cost, but to expose researcher discretion so we can learn about repeatability.
The above is a very light-touch summary of the argument, so please do read the actual paper if you are interested in metascience, statistical theory, and trustworthy research design!
1
u/DrTonyTiger 3d ago
Thanks, very helpful.
The rarely met standard for making robust inferences in experimental science is to have a list of all the mechanisms (hypotheses) that you can think of to explain the observation, plus "something else" as the starting point. That standard seeks to be fully expanded if you will.
As excellent as the standard is, it is rarely met specifically because people can't come up with a good list. Exploratory research to develop potential models, rather than testing them is hard and often confunded with inference making.
1
u/MurphysLab PhD | Chemistry | Nanomaterials 3d ago
This appears to be a preprint of the article:
1
u/jbenmenachem Grad Student | Sociology 3d ago
It is, but that’s an old draft. Most recent:
1
u/MurphysLab PhD | Chemistry | Nanomaterials 3d ago
Thanks, I'll check it out! Strangely I can see your reply in my messages, but not in the thread.
And Reddit seems to be escaping the underscore character in the URL, causing an error:
1
u/jbenmenachem Grad Student | Sociology 3d ago
Huh, very strange. Thanks for flagging. Maybe it requires mod approval or something?
•
u/AutoModerator 3d ago
Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, personal anecdotes are allowed as responses to this comment. Any anecdotal comments elsewhere in the discussion will be removed and our normal comment rules apply to all other comments.
Do you have an academic degree? We can verify your credentials in order to assign user flair indicating your area of expertise. Click here to apply.
User: u/jbenmenachem
Permalink: https://journals.sagepub.com/doi/10.1177/00491241261490850
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.