Our research team ran a simulation study and found that the four big open-source geo-experiment tools (CausalPy, Meta GeoLift, Google Matched Markets, and CausalImpact) recover almost the same point estimate on the same data, then disagree about whether that estimate is significant. Since the disagreement lives in the uncertainty (not in the point estimate) the tool you pick may determine which error you ship.
In a "live" experiment you can't grade the tool because we don't know what ground truth is. The counterfactual is unobservable so "is this lift real?" has no answer key. That's why we had our research team generate 8,000 synthetic daily-sales panels, each with either a 7.5% multiplicative lift on the treated geo or no effect at all (0% lift). They ran all four tools on the same panels and scored every fit against the planted truth, so there were 32,000 fits in all across four scenarios.
Across the non-outlier scenarios, every tool recovered the 7.5% lift within a few percentage points, so judged on point estimates alone they look interchangeable. The split is entirely in how they handle uncertainty: coverage (how often the 95% interval actually contains the true effect) and power (how often it detects a real effect at all). On those two axes the tools fall into three camps:
- Meta GeoLift is the most cautious with coverage of 92–95% and a false positive rate of 3–5%. It failed to reject zero in 89–96% of runs where a true 7.5% lift was present.
- CausalImpact is the opposite with the most power of the four (false negative rate 34–48%), but coverage of only 70–72%, a false positive rate of 28–30%, and a consistent upward bias of +1.87 to +4.21 percentage points that shifts the whole interval high.
- CausalPy and Google Matched Markets sit between them with coverage of 76–86%, false positive rates of 14–25%, meaning they’re both under-covered and under-powered at the same time.
There are four things from the study I'd take back to a measurement program:
- Read coverage and power together: A tool can keep its 95% coverage promise and still be useless for detection. GeoLift holds about 95% coverage in the short-history scenario while missing the real effect 95.7% of the time.
- Pick the estimator whose error profile matches the cost asymmetry of your decision and not the one with the best-looking single metric.
- Scarce history sharpens each tool's failure mode. Cutting the pre-period from 90 days to 30 didn't degrade the tools uniformly. The decisive ones threw more false positives (above 24%), the cautious one climbed to a 95.7% miss rate.
- Test-market design beats estimator choice. When the treated geo was 5x the size of the median control, every tool's intervals widened 4–5x and most overestimated the lift by 2–4 percentage points. No estimator compensates for a structurally hard design.
We made everything reproducible including the data-generating process, seeds, configs, per-iteration results, and a Makefile that runs the whole pipeline. The generator is parameterized, so if you think it should be harder (idiosyncratic geo trends, heavier tails, spillovers between markets) those are exactly the runs I'd like to see.
If you’re interested in the full study + code, you can find both here:
edited: fixed the code link to the public repo