Why most incrementality tests are invalid

Attribution asks which touchpoints preceded a conversion. Incrementality asks a harder question: would the conversion have happened anyway? The only way to answer it is to compare people who saw the advertising against comparable people who did not, and the entire difficulty is in the word comparable.

This is why retargeting flatters itself so reliably. It reaches people who already visited your site, many of whom were going to convert regardless, and last-click attribution credits it for all of them. An incrementality test is the mechanism that separates the ones it caused from the ones it caught.

For channels where person-level tracking is impossible — cinema, out-of-home, television — a geographic experiment is the only credible causal design available. That makes the quality of the design the whole ball game.

Statistical power is the probability that a test detects a real effect of a given size. The convention is 80% power at 5% significance, and a design that falls short of it will usually miss a genuine effect — and, more dangerously, any positive result it does produce carries little information.

The number that makes this actionable is the minimum detectable effect: the smallest true lift the design could reliably find. If your design's minimum detectable effect is a 30% lift and the realistic effect of your campaign is 5%, no amount of patience fixes it. The test cannot answer your question and should not be commissioned.

The inputs are your baseline conversion volume, the variance in that baseline, the number of independent markets and the duration. All of them are known before the campaign runs, which is why this is a pre-flight check rather than a post-hoc caveat. AdBuyMCP computes it before any readout and refuses to produce a lift figure when the design falls short, returning the required sample days and the minimum detectable lift instead — its own phrasing is that reporting a number from it would be reporting noise.

A holdout only works if it is genuinely held out. The common failure is that the campaign reached the control markets — through national inventory, spillover from an adjacent region, or a channel whose targeting is coarser than the test design assumed — so the comparison is between heavily exposed and lightly exposed rather than exposed and unexposed.

The result is an underestimate of the true lift by an unknown amount, presented with the same confidence as a clean result. Nothing in the output flags it. This is why the check has to be structural: verify that no candidate control market received delivery before running the comparison.

The fix, when contamination is found, is specific rather than general: exclude those markets from targeting for the duration of the test window, or nominate a control the campaign does not buy. AdBuyMCP refuses to read out a contaminated design and names the affected markets rather than adjusting silently.

The intuitive way to match a control market to a test market is on demography: similar size, similar income, similar age profile. It is also the wrong way. Two cities that look alike on paper frequently behave differently commercially, because of competitor presence, distribution, local seasonality or a dozen other things nobody is modelling.

The right basis is the pre-campaign outcome series itself. If two markets tracked each other closely on the metric you care about before the campaign, a divergence afterwards is meaningfully attributable. If they did not, no amount of demographic similarity rescues the comparison.

This is also what makes the analysis robust to shared shocks. Difference-in-differences compares the change in the test markets against the change in the controls, so anything affecting both — seasonality, a competitor's national campaign, the weather — cancels. What does not cancel is anything that hit only the test markets, which is the residual risk in any geo experiment and the reason a result should carry a confidence interval rather than a single number.

Four questions separate a result worth acting on from one worth ignoring. What was the minimum detectable effect, and is the reported lift comfortably above it? How were the control markets chosen, and on what series were they matched? Was contamination checked, and what did the check find? And what is the confidence interval, rather than the point estimate?

A result presented as a single percentage with no interval and no design detail is not evidence. It may be true, but nothing about how it was presented allows you to tell.

The corollary is worth stating plainly: a vendor who will tell you when a test cannot work is more useful than one who always returns a number. The refusal carries information; the number might not.