procedure

Run a geo-lift test

The engine refuses underpowered or contaminated tests and names the fix.

A geo-lift test runs advertising in some markets, withholds it from comparable others, and compares the outcomes. It is the causal design that needs no login, pixel or match rate, which makes it the only credible option for out-of-home, cinema and television. Most such tests fail before they start, for two reasons: not enough power to detect a plausible effect, or a control group that received some of the campaign. Both failures are steps in this procedure rather than errors in it, and the engine returns each one with the specific thing that would fix it.
steps
6
things needed first
4
roughlyAn hour's setup, plus weeks of testing
Get my plan and price Book a 30-minute call with the founder

Free, with no card. We read your homepage, draft who buys from you and show your monthly price before you join the waitlist. Nothing is bought until you've approved the plan and the ads.

what you need

You need these before you start

  • Enough geographically separable markets, since a national campaign in one market cannot be tested this way
  • An outcome you can observe by geography: bookings, leads, point-of-sale, or conversions with a location
  • Pre-campaign history in those markets, so they can be matched on behaviour rather than on demography
  • Enough spend that a plausible effect would be detectable, which is what the power step establishes

the procedure

The procedure has six steps

Every step comes with a note on what commonly goes wrong there.

  1. step 01

    Nominate a test market

    Choose where the media will run. AdBuyMCP matches candidate control markets against it on the pre-campaign outcome series rather than on demographic similarity.

    what goes wrong here

    Two cities that look alike demographically often behave differently commercially, and it is the behaviour you need to match.

  2. step 02

    Let the power analysis run first

    A two-sample calculation on daily observations runs before any readout, defaulting to 5% significance and 80% power. It returns either a powered verdict with the minimum detectable lift, or an underpowered one.

    what goes wrong here

    Checking power first stops you paying for a test that cannot answer, because an underpowered test produces a random result.

  3. step 03

    If it refuses, read what it needs

    An underpowered design returns the number of days of post-exposure data required and the minimum lift the design could detect, and reports no lift figure. Its own words are that reporting a number from it would be reporting noise.

    what goes wrong here

    If the minimum detectable lift is 30% and a realistic lift is 5%, the test as designed cannot answer your question. The engine returns the sample days that would close the gap instead of a number, and if that is longer than the flight you can fund, the test should not be commissioned.

  4. step 04

    Check the control is genuinely held out

    If every candidate control market received delivery from the campaign, none of them is a holdout and the engine refuses on that basis, naming the contaminated markets.

    what goes wrong here

    A contaminated control is the most common way a lift test quietly measures nothing, because it still produces a confident-looking number. Exclude those markets from targeting for the test window, or nominate a control the campaign does not buy.

  5. step 05

    Run the flight without touching the design

    Let the campaign run. Changing the targeting, the budget split or the market list mid-flight invalidates the comparison.

    what goes wrong here

    The temptation to optimise into the control markets when the test markets are performing is strong and fatal.

  6. step 06

    Read the difference-in-differences result

    A powered design reports the lift with its confidence interval, comparing the change in exposed markets against the change in control markets so that anything affecting both cancels out.

    what goes wrong here

    A result is graded causal only when every geo outcome is observed, every delivery row maps to a live line, and control exposure is verified. Sandbox delivery is labelled modelled by construction, so a sandbox campaign can never reach a causal grade.

what you end up with

Either a lift figure with a confidence interval that can survive being questioned, or a documented refusal naming exactly what would have to change.

other procedures

You might need these procedures next

Talk your market and your plan through with James Rooney, who built AdBuyMCP and wrote its code.

Book a 30-minute call with the founder

These questions come up when you run a geo-lift test

How much spend does a geo-lift test need?

It depends on your baseline variance, your conversion volume, how many independent markets you have and how large an effect you would need to detect, so no single figure applies. The power analysis computes exactly that before the test runs, which is why it runs first. A multi-catchment retailer can often power a test that a single national campaign cannot at any budget.

Has AdBuyMCP produced a causal result for a customer?

No. Nothing has run live, and a geo-lift is graded causal only under conditions sandbox data cannot meet by construction. The engine, the power gate and both refusals run in the product today, and none of their output is yet evidence about a real campaign.