Designing a geo-lift test that can actually be powered

Pick markets that behaved alike before the campaign. Buy media in some of them and deliberately buy none in the others. After the flight, scale the control markets by the ratio the two groups sat at before the media started, and treat that scaled series as the counterfactual: what the exposed markets would have done anyway. The gap between what they actually did and that counterfactual is the estimate, and it is a genuine causal estimate because the holdout was chosen by you rather than selected by exposure.

That last clause is the whole reason this design is worth the trouble. Almost every other measurement in media compares people who saw the ad with people who did not, and the two groups differ in the one way that matters: one of them was targeted. A geo holdout is decided before delivery, by the buyer, on a geography rather than on a person, which is why it survives the collapse of identity-based measurement and why it works on channels with no user-level data at all.

The sample-size calculation for a two-sample comparison of daily observations is short enough to write down: the required number of days is roughly two times the squared sum of the two critical values, times the variance of your daily baseline, divided by the square of the absolute change you are trying to detect. At the conventional settings — a 5% significance level and 80% power — that sum of critical values is a shade over 2.8.

Two consequences follow, and both of them are counter-intuitive to anyone who has planned a flight by budget. Volatility enters squared, so a conversion series that swings wildly day to day needs dramatically more days than a smooth one at the same volume. And the lift you are chasing enters squared in the denominator, so a design that can detect a 20% lift in three weeks needs roughly twelve weeks to detect a 10% one. Chasing a small effect is expensive in a way that is not obvious until the calculation is done.

The reason to run it before the flight is that every fix is a design change. Once the media has run, the volatility is what it was and the days are what they were, and the only remaining move is to report a number the design cannot support. AdBuyMCP therefore runs the calculation at planning time and returns one of two things: a powered verdict with the smallest lift the design could detect, or an underpowered verdict with the number of days it would actually need and no lift figure at all.

The candidate list is twelve UK conurbations with their populations and broadcast regions: Greater London, Greater Manchester, the West Midlands, West Yorkshire, Greater Glasgow, Liverpool City Region, Sheffield City Region, Tyneside, Greater Nottingham, Bristol, Edinburgh and Cardiff. That set is large enough to give a national campaign a real holdout and small enough that each market carries meaningful weight.

Matching ranks the candidates on two things: how strongly each one's pre-period conversion series correlates with the test market's, weighted at 70%, and how similar the two populations are, weighted at 30%. The top three become the control group. Correlation does most of the work because the design's assumption is about parallel trends rather than about equal levels: a control market half the size is fine if it moves in step.

One implementation detail is worth stealing whatever tool you use. The two series are joined on date before they are correlated, not aligned by position in the array. Real conversion series have gaps — a day with no conversions often produces no row at all — and index-aligning two series with different gaps pairs Tuesday with Thursday and can invert the sign of the correlation. A candidate sharing fewer than two dates with the test market cannot be correlated and is skipped rather than scored at zero.

The most common way a lift test quietly measures nothing is contamination: the control markets received some of the media. It happens by accident more often than by design — a targeting fallback widening to national, a programmatic line delivering outside the intended polygons, a search campaign with no geographic restriction at all — and it biases the result towards zero, which makes a working campaign look ineffective.

The defence is procedural rather than statistical. Before matching runs, every market the campaign actually delivered into is excluded from the candidate list. If that leaves nothing, the honest response is to refuse: AdBuyMCP returns a "no clean control" refusal naming the contaminated markets, rather than matching against a second test cell and calling it a holdout.

There is a second, quieter failure. Where the buy reports no geography at all — which is normal on several rails — you cannot verify that the control markets were clean, only assume it. Control integrity is then reported as unverified rather than passed, along with the count of delivery rows carrying no geography. A check that cannot see the delivery has not passed; it has not run.

The readout is a difference-in-differences with pre-period ratio scaling. The control series is multiplied by the ratio the two groups sat at before exposure, that scaled series is the counterfactual, and the daily deviations after exposure give the point estimate, the confidence interval and the p-value.

Two details in that calculation are worth insisting on wherever you have it run. Inference should be Student-t rather than normal, because post-exposure windows in media are short and the normal approximation understates the tails badly at two weeks. And the standard error has to carry both sources of uncertainty: the noise in the post-period, and the fact that the scaling ratio is itself an estimate from a finite pre-period. Treating that ratio as a known constant is how a lucky fortnight before the campaign turns into a tight, spuriously significant lift.

Then there is the grade, which is separate from the statistics. Passing the power gate is necessary and not sufficient. A result is only allowed to call itself causal when every geographic outcome is observed, every delivery row used in the control check maps to a line that executed live, and control exposure is verified. Where inputs are modelled or the delivery is sandbox, the result is graded a modelled demonstration; where the evidence is mixed or exposure could not be checked, it is graded indicative. The grade travels with the number.

When the calculation says no, there are four real moves and one fake one. The fake one is running it anyway and reporting whatever comes back.

Some campaigns cannot be proved causally, and the number of them is larger than the incrementality industry likes to admit. A national buy with no geographic dimension in its reporting has nothing to hold out. A thin conversion series at a small budget cannot clear the arithmetic at any sensible flight length. In both cases the useful response is to say so at planning time and offer what the campaign can actually evidence: delivery, observed response on your own properties, and modelled reach and frequency, each under its own label.

That is a smaller claim and it is the one the campaign can carry. It is also a considerably less pleasant conversation to have at the point of sale than at the point of reporting, and a considerably more useful one, because the buyer still has the option of changing the design.

If you take one habit from this guide, take this one: ask whether the campaign can be proved before you fund it, in the same meeting where you agree the budget. The answer takes about ten minutes to compute and it changes what you buy.