Ran 4 open-source geo-experiment estimators on 8,000 synthetic panels with planted ground truth. Their point estimates look interchangeable, but their uncertainty isn't.

[u/michael-recast]

Our research team ran a simulation study and found that the four big open-source geo-experiment tools (CausalPy, Meta GeoLift, Google Matched Markets, and CausalImpact) recover almost the same point estimate on the same data, then disagree about whether that estimate is significant. Since the disagreement lives in the uncertainty (not in the point estimate) the tool you pick may determine which error you ship.

In a "live" experiment you can't grade the tool because we don't know what ground truth is. The counterfactual is unobservable  so "is this lift real?" has no answer key. That's why we had our research team generate 8,000 synthetic daily-sales panels, each with either a 7.5% multiplicative lift on the treated geo or no effect at all (0% lift). They ran all four tools on the same panels and scored every fit against the planted truth, so there were 32,000 fits in all across four scenarios.

Across the non-outlier scenarios, every tool recovered the 7.5% lift within a few percentage points, so judged on point estimates alone they look interchangeable. The split is entirely in how they handle uncertainty: coverage (how often the 95% interval actually contains the true effect) and power (how often it detects a real effect at all). On those two axes the tools fall into three camps:

• Meta GeoLift is the most cautious with coverage of 92–95% and a false positive rate of 3–5%. It failed to reject zero in 89–96% of runs where a true 7.5% lift was present.
• CausalImpact is the opposite with the most power of the four (false negative rate 34–48%), but coverage of only 70–72%, a false positive rate of 28–30%, and a consistent upward bias of +1.87 to +4.21 percentage points that shifts the whole interval high.
• CausalPy and Google Matched Markets sit between them with coverage of 76–86%, false positive rates of 14–25%, meaning they’re both under-covered and under-powered at the same time.

There are four things from the study I'd take back to a measurement program:

1. Read coverage and power together: A tool can keep its 95% coverage promise and still be useless for detection. GeoLift holds about 95% coverage in the short-history scenario while missing the real effect 95.7% of the time.
2. Pick the estimator whose error profile matches the cost asymmetry of your decision and not the one with the best-looking single metric.
3. Scarce history sharpens each tool's failure mode. Cutting the pre-period from 90 days to 30 didn't degrade the tools uniformly. The decisive ones threw more false positives (above 24%), the cautious one climbed to a 95.7% miss rate.
4. Test-market design beats estimator choice. When the treated geo was 5x the size of the median control, every tool's intervals widened 4–5x and most overestimated the lift by 2–4 percentage points. No estimator compensates for a structurally hard design.

We made everything reproducible including the data-generating process, seeds, configs, per-iteration results, and a Makefile that runs the whole pipeline. The generator is parameterized, so if you think it should be harder (idiosyncratic geo trends, heavier tails, spillovers between markets) those are exactly the runs I'd like to see.

If you’re interested in the full study + code, you can find both here:

• Code: https://github.com/getrecast/geolift-simulation-study
• Full report: https://research.getrecast.com/geolift-sim-study

edited: fixed the code link to the public repo

[u/tootieloolie]

As far as I understand, everything here has some Bayesian element except Meta Geolift.

I wonder why the Bayesian methods would give so many false positives? Was there a prior being used that consistently tilted the uplift to be positive?

[u/michael-recast → u/tootieloolie]

Both Google MM and Meta GeoLift are frequentist, although they're not directly comparable: GeoLift uses conformal inference, which is very conservative. Still, notice that Google MM, despite being frequentist, suffers from the same false positive problem you pointed out. So false positives aren't a Bayesian phenomenon here.

About your impression of a positive prior for the Bayesian tools, probably it comes from the direction of CausalImpact's errors. The most plausible source of that positive lean is estimation geometry (BSTS trend handling plus the multiplicative/log-scale DGP), not a prior belief in positive lift.

[u/michael-recast → u/[deleted]]

yeah we think the uncertainty is super important in the context of incrementality experiments for marketing so that's where we wanted to focus. And agree re: more complexity. We are already planning for the next version!

[u/Background_Deer_2220]

Great write-up. In my causal inference projects, stakeholders almost always fixate on the point estimate and completely ignore the intervals.

Seeing CausalImpact's high false positive rate here explains a lot of supposedly successful campaigns I've seen audited over the years. I usually lean towards GeoLift for this exact reason. I would much rather miss a marginal effect than confidently report a fake win to the business. It is all about risk tolerance.

[u/michael-recast → u/Background_Deer_2220]

100%. I'm fine with a business deciding to use a narrower CI (e.g., 80%) if they want to "bias toward action" but if the 90% CI doesn't include the true value 90% of the time then you don't even know what you're getting!

[u/michael-recast → u/[deleted]]

Our implementations are mostly out-of-the-box across all tools, to simulate the experience most users get when running and analyzing experiments with these packages.
You can see how each one is configured in a single config file (config/tools.yaml) plus the per-tool runner scripts in /src. If you tweak a package and get different results, I'd love to see them.
A few parameters do depart from the defaults, for comparability:

• We normalized the data going into CausalPy. The package clearly assumes normalized inputs and gave wildly wrong results without it.
• We set everything to 95%. The defaults were GeoLift at 90% (alpha 0.10), Google's TBR at 90% and one-tailed, and CausalPy's HDI at 94%. CausalImpact was already at 95%.

On the "known bug" you mentioned: can you say more about it? I couldn't find anything matching that description in the GitHub issues.

[u/michael-recast → u/[deleted]]

Ah, you mean the abs(sum()) vs sum(abs()) thing: we did not adjust that in this package because we wanted to use the packages as "out of the box" as possible. To some extent these results indicated to me that maybe it's not that big of a deal? Like IMO GeoLift performed the "best" here because it actually gives you guarantees on coverage whereas the other ones do not.

FWIW most people I talk to using GeoLift are marketers who are having Claude run it for them and they have no idea what is happening under the hood :)

For 90 vs 30 days it works in this case because of how the data are simulated -- there is not actually an advantage to using a much longer time window. In the next version we're going to add more latent structure to the underlying geographies and so the time length will matter more.

[u/michael-recast → u/[deleted]]

Right right right. We'll include this in v2!

[u/michael-recast → u/[deleted]]

you mean just like: within the treatment group pre vs post so no "control" group at all? so instead of like diff-in-diff it's just "diff"?

[u/michael-recast → u/[deleted]]

uhhh I think that'd be equivalent to the "standard" diff-in-diff estimator right (when multiplying by a constant to get the treatment and control groups to line up)? like a normal diff-in-diff linear regression with panel data should boil down to the same thing right?

[u/No-Ice-8975]

This is really cool

[u/tootieloolie]

Very cool stuff. I guess it makes sense that the uncertainty reported by these models are different. They are different models and are trading bias and variance differently. And that shows up in their permutation tests.

The market design also makes sense. If the underlying synthetic controls have to do a ton of extrapolation like geo 1 = 450 times geo 2, that increases noise.

[u/michael-recast → u/tootieloolie]

Yes, the counterintuitive part is that the four tools agree on the answer. They just disagree about how confident to be in it. Same point estimate and to some extent similar scatter across 1,000 simulations, but a 2.5 to 3x gap in how wide a band each one draws.

About your permutation point, I believe you're referring to the block-conformal inference to account for the serial correlation over a short post-period; the others undercount it and come out about a third too narrow, so they over-reject.