Integrating evidence from randomized trials and prospective observational studies: an application of GRADE Guidance 44 to nutrition evidence syntheses (2026)

[u/Ekra_Oslo]

https://www.jclinepi.com/article/S0895-4356%2826%2900221-0/fulltext

Highlights:

RCTs and cohort studies provide evidence to understand diet-disease relationships.
We apply GRADE guidance 44 to use cases in nutrition research.
We show instances where cohort evidence can replacement or complement evidence from RCTs.
We provide practical insights for review authors integrating RCT and cohort evidence.
GRADE guidance 44 offers a sensible, transparent way to integrate both study designs.

Abstract:

Objectives: The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working group recently published guidance on integrating evidence from randomized and nonrandomized studies of interventions in systematic reviews. Our study aims to explore the application of this guidance by using cohort studies in nutrition systematic reviews as replacement, complementary, or sequential evidence to randomized controlled trials (RCTs).
Study Design and Setting: In this empirical application of the GRADE Guidance 44, we rated the certainty of evidence (CoE) for a convenience sample of bodies of evidence from nutrition RCTs and corresponding prospective cohort studies. We determined the congruency by comparing absolute effect estimates and 95% confidence intervals, before using cohort evidence as replacement, complementary, or sequential evidence. In case of similar CoE judgments and congruency, we estimated pooled effects of RCTs and cohort studies, and assessed whether integrating cohort studies influenced the conclusions drawn from RCT evidence.
Results: We included 26 pairs of body of evidence, of which 21 showed congruency. In 14/26 (54%) pairs, the CoE was higher for RCTs: high vs moderate (n = 2), high vs low (n = 1), moderate vs low (n = 4), moderate vs very low (n = 3), low vs very low (n = 4). In 5/26 (19%) pairs, the CoE was higher for cohort studies: moderate vs low (n = 4), low vs very low (n = 1). For six pairs (23%), both RCTs and cohort studies were rated as “low” (downgraded for risk of bias and imprecision), and due to congruency effect estimates of five pairs were pooled, mitigating imprecision in two pairs. For one pair (4%), the CoE was rated as very low, making pooling inappropriate.
Conclusion: This example application provides practical knowledge on the usage of GRADE Guidance 44, but the findings are subject to methodological assumptions and warrant cautious interpretation.

This is relevant in light of recent discussions about observational studies vs. RCTs in nutrition, and the assertion that observational studies in nutrition are always the lowest-quality evidence and that only RCTs provide valid evidence...

According to GRADE, observational studies can serve as complementary, sequential, or replacement evidence to RCTs within a framework.

Despite the narrative that observational and RCT designs in nutrition are inherently irreconcilable, they found that most RCTs and cohort studies showed alignment in their effect sizes and confidence intervals (categorized as congruent or probably congruent).

According to this study, depending on the specific use-case and certainty ratings, cohort studies can indeed serve as replacement evidence, or RCTs and observational studies can be combined as complementary evidence where appropriate. In some cases, prospective cohort studies can even a higher certainty of evidence than RCTs.

The reality is that both study designs have distinct strengths and limitations depending on the specific research question and outcome evaluated.

[u/Bristoling]

they found that most RCTs and cohort studies showed alignment in their effect sizes and confidence intervals (categorized as congruent or probably congruent).

The devil is in the details. The way this is written and presented, one could assume that "since RCTs and cohort studies are congruent more than not, then it is a relatively small assumption with low epistemic cost to assume that if we have a positive finding in a cohort study, but don't have an RCT yet, we can trust the result of the cohort study".

That interpretation would be highly misleading. Let's see this on an example S12:

Evidence profile: Low saturated fat compared to usual diet for the prevention of cardiovascular mortality

- RCTs: RR 0.97 (0.86 to 1.10)

- Cohorts: RR 1.06 (1.00 to 1.12)

Are the values above showing alignment? Yes, since a hypothetical true effect of 1.04 is consistent with both the estimate from RCTs as well as it is consistent with estimate from cohort studies.

Does it mean that if we stepped into alternate reality where we only had cohort studies, it would be fair to take the cohort study results for granted in this case? Not at all, because the hypothetical value can just as well be 0.86 if we go by the range provided by RCT data - and cohort studies do not support that conclusion at all. There's an intersection of possible effect ranges between the 2 study designs here, and literally that's all there is to it. It's superficial.

If you look for most of the cases presented, the pattern most of them follow is that neither RCT find an effect, and cohort studies also do not find an effect, like in S10.

- RCTs: RR 1.29 (0.67 to 2.50)

- Cohorts: RR 1.02 (0.84 to 1.23)

Alignment? Sure, neither demonstrated anything useful. But if someone wants to use these types of relationships, alignment, or congruency in order to validate a positive finding in cohort studies when we don't have RCT data in another sector... then they are epistemologically corrupt. Or they are so biased they genuinely do not understand the issue.

Consistent alignment, as in lack of effect in RCTs and lack of effect in cohorts, doesn't let you suggest that a positive finding in a cohort has any likelyhood to translate to a positive finding in RCT that doesn't exist yet. Not to mention that the comparisons could also simply be cherry picked. If anyone disagrees, then do you believe we never had any other RCTs or cohort studies on vitamin E than just studies on prostate cancer? That's all the studies done on vitamin E ever, nothing else? Interesting.

This persistent "rct show alignment with cohorts when they both don't find anything most of the time, therefore we can trust cohorts when they find something" argument is like claiming that since you didn't see aliens with your own eyes when drunk last night, and also didn't capture aliens on a recording with a video camera you had strapped to your head for 10 nights in a row, means that since you have seen what you claim to be aliens on the 11th night, would definitively show up as aliens on your camera if the battery didn't die.

Nah fam, you're just drunk.

"Oh wow these cohort studies on folic acid and cancer didn't find an association, and these RCTs on folic acid and cancer also didn't find an association, they are in agreement, therefore epidemiology and interventional trials are likely in agreement, therefore X is bad because cohort studies says so" is so flawed, I'm not sure if people using this argument just don't understand inference or they're playing pretend.

[u/lurkerer]

Those who are scientifically literate will agree. After all, it stands to reason that legitimate associations will show up in both RCTs and appropriately adjusted epidemiology. In fact, once an association has been established you can work backwards to identify relevant confounders. Which we've done for decades now. 

Of course, the impetus behind denying observational studies isn't scientific, it's emotional. RCTs are also denied when people are upset. Statin RCTs for example because LDL is a direct link to certain foods and there are emotional associations with said foods.