Designing a geo-lift test that can actually be powered

Pick markets that behaved alike before the campaign. Buy media in some of them and deliberately buy none in the others. After the flight, scale the control markets by the ratio the two groups sat at before the media started, and treat that scaled series as the counterfactual: what the exposed markets would have done anyway. The gap between what they actually did and that counterfactual is the estimate, and it is a genuine causal estimate because the holdout was chosen by you rather than selected by exposure.

That last clause is the whole reason this design is worth the trouble. Almost every other measurement in media compares people who saw the ad with people who did not, and the two groups differ in the one way that matters: one of them was targeted. A geo holdout is decided before delivery, by the buyer, on a geography rather than on a person, which is why it survives the collapse of identity-based measurement and why it works on channels with no user-level data at all.

The sample-size calculation for a two-sample comparison of daily observations is short enough to write down: the required number of days is roughly two times the squared sum of the two critical values, times the variance of your daily baseline, divided by the square of the absolute change you are trying to detect. At the conventional settings of a 5% significance level and 80% power, that sum of critical values is a shade over 2.8.

Two consequences follow, and both of them are counter-intuitive to anyone who has planned a flight by budget. Volatility enters squared, so a conversion series that swings wildly day to day needs dramatically more days than a smooth one at the same volume. And the lift you are chasing enters squared in the denominator, so a design that can detect a 20% lift in three weeks needs roughly twelve weeks to detect a 10% one. Chasing a small effect is expensive in a way that is not obvious until the calculation is done.

The reason to run it before the flight is that every fix is a design change. Once the media has run, the volatility is what it was and the days are what they were, and the only remaining move is to report a number the design cannot support. AdBuyMCP therefore runs the calculation at planning time and returns one of two things: a powered verdict with the smallest lift the design could detect, or an underpowered verdict that gives the number of days it would actually need and reports no lift figure.

Meta's open-source GeoLift tool makes the same point in its own walkthrough: a prospective power analysis comes before the test, not after it. If your conversion volume is small, the worked tables in the small-budget guide show where your design lands.

The candidate list is twelve UK conurbations with their populations and broadcast regions: Greater London, Greater Manchester, the West Midlands, West Yorkshire, Greater Glasgow, Liverpool City Region, Sheffield City Region, Tyneside, Greater Nottingham, Bristol, Edinburgh and Cardiff. That set is large enough to give a national campaign a real holdout and small enough that each market carries meaningful weight.

Matching ranks the candidates on two things: how strongly each one's pre-period conversion series correlates with the test market's, weighted at 70%, and how similar the two populations are, weighted at 30%. The matcher returns the top three by default, and the product's post-flight readout takes the top two and sums their series into one control. Correlation does most of the work because the design's assumption is about parallel trends rather than about equal levels: a control market half the size is fine if it moves in step.

Whatever tool you use, join the two series on date before correlating them rather than aligning them by position in the array. Real conversion series have gaps (a day with no conversions often produces no row at all), and index-aligning two series with different gaps pairs Tuesday with Thursday and can invert the sign of the correlation. A candidate sharing fewer than two dates with the test market cannot be correlated and is skipped rather than scored at zero.

The most common way a lift test quietly measures nothing is contamination: the control markets received some of the media. It happens by accident more often than by design: a targeting fallback widening to national, a programmatic line delivering outside the intended polygons, a search campaign with no geographic restriction at all. It biases the result towards zero, which makes a working campaign look ineffective.

The defence is a procedure: before matching runs, every market the campaign actually delivered into is excluded from the candidate list. If that leaves nothing, the test should not run: AdBuyMCP returns a "no clean control" refusal naming the contaminated markets, rather than matching against a second test cell and calling it a holdout.

Where the buy reports no geography at all, which is normal on several rails, you cannot verify that the control markets were clean, only assume it. Control integrity is then reported as unverified rather than passed, along with the count of delivery rows carrying no geography. A check that cannot see the delivery has not run, so it cannot count as a pass.

Google's CausalImpact documentation states the same condition from the other side: the control series must not themselves be affected by the intervention, or the estimate can be substantially under- or overestimated.

The readout is a difference-in-differences with pre-period ratio scaling. The control series is multiplied by the ratio the two groups sat at before exposure, that scaled series is the counterfactual, and the daily deviations after exposure give the point estimate, the confidence interval and the p-value.

Two details in that calculation are worth insisting on wherever you have it run. Inference should be Student-t rather than normal, because post-exposure windows in media are short and the normal approximation understates the tails badly at two weeks. And the standard error has to carry both sources of uncertainty: the noise in the post-period, and the fact that the scaling ratio is itself an estimate from a finite pre-period. Treating that ratio as a known constant is how a lucky fortnight before the campaign turns into a tight, spuriously significant lift.

Then there is the grade, which is separate from the statistics. Passing the power gate is necessary and not sufficient. A result is only allowed to call itself causal when every geographic outcome is observed, every delivery row used in the control check maps to a line that executed live, and control exposure is verified. Where inputs are modelled or the delivery is sandbox, the result is graded a modelled demonstration; where the evidence is mixed or exposure could not be checked, it is graded indicative. The grade is reported with every result.

When the calculation says no, the design has to change, and the moves below are how to change it. Running the test anyway only produces a number the design cannot support.

Some campaigns cannot be proved causally, and the number of them is larger than the incrementality industry likes to admit. A national buy with no geographic dimension in its reporting has nothing to hold out. A thin conversion series at a small budget cannot clear the arithmetic at any sensible flight length. In both cases the useful response is to say so at planning time and offer what the campaign can actually evidence: delivery, observed response on your own properties, and modelled reach and frequency, each under its own label.

That is a smaller claim and it is the one the campaign can carry. It is also a considerably less pleasant conversation to have at the point of sale than at the point of reporting, and a considerably more useful one, because the buyer still has the option of changing the design.

If you take one habit from this guide, take this one: ask whether the campaign can be proved before you fund it, in the same meeting where you agree the budget. The answer takes about ten minutes to compute and it changes what you buy.