--- title: "Why most incrementality tests are invalid | AdBuyMCP" description: "Two failures account for most of them: a design with too little power to detect a plausible effect, and a control group that received some of the campaign." url: "https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid" --- [Home](https://adbuymcp.com/) / [Guides](https://adbuymcp.com/guides) / Why most incrementality tests are invalid Guide · knowing what you bought # Why most incrementality tests are invalid An underpowered test returns a random number that looks like a result. Most incrementality tests fail before the media runs, and for two reasons that are both checkable in advance. The first is power: if the design cannot detect an effect of a plausible size given the spend, the conversion volume and the variance in the data, then any number it produces is noise that happens to look like a result. The second is contamination: if the control markets received some of the campaign, the comparison is exposed against slightly-less-exposed and the measured lift is an underestimate of unknown size. Neither failure announces itself in the output: both produce a confident-looking percentage. That is why the checks have to happen before the money is committed rather than after. [Get my plan and price](https://adbuymcp.com/start) [Book a 30-minute call with the founder](https://adbuymcp.com/contact#book) Free, with no card. We read your homepage, draft who buys from you and show your monthly price before you join the waitlist. Nothing is bought until you've approved the plan and the ads. By [James Rooney](https://adbuymcp.com/about) Updated 29 October 2026 - **Standard significance**: 5% - **Standard power**: 80% - **Failure one**: Underpowered design - **Length**: 6 min read Sources 2 linked and dated On this page [What to remember](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#takeaways) [The question an incrementality test answers](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#the-question-an-incrementality-test-answers) [The first failure is too little power to answer](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#the-first-failure-is-too-little-power-to-answer) [The second failure is a control that was not held out](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#the-second-failure-is-a-control-that-was-not-hel) [Match markets on their pre-campaign outcomes](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#match-markets-on-their-pre-campaign-outcomes) [What to ask when you are handed a result](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#what-to-ask-when-you-are-handed-a-result) [Limits that apply](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#limits) [Where this came from](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#sources) [Questions](https://adbuymcp.com/guides/why-most-incrementality-tests-are-invalid#questions) Who this is for Anyone who has been handed an incrementality result and is not sure how much weight it carries, and anyone about to commission one. Key points ## Remember these points if you read nothing else These are the points most likely to change your decision, and the full guide follows below. 1. 01 Power is the probability a test detects a real effect. A design with low power usually misses genuine effects and its positive findings are as likely to be noise as signal. 2. 02 The minimum detectable effect is the number that tells you whether a test is worth running. If it is 30% and a realistic lift is 5%, the experiment as designed cannot answer the question, and the engine returns the sample days it would take to close that gap rather than a lift figure. 3. 03 A contaminated control is the most common silent failure. If the campaign reached the control markets, the comparison measures the wrong thing and still returns a number. 4. 04 Both checks are arithmetic and both can be done before spending. The reason they usually are not is that the answer is often 'this test cannot work', which nobody wants to hear. 01 ## The question an incrementality test answers Attribution asks which touchpoints preceded a conversion. Incrementality asks a harder question: would the conversion have happened anyway? The only way to answer it is to compare people who saw the advertising against comparable people who did not, and the entire difficulty is in the word comparable. This is why retargeting flatters itself so reliably. It reaches people who already visited your site, many of whom were going to convert regardless, and last-click attribution credits it for all of them. An incrementality test is the mechanism that separates the ones it caused from the ones it caught. For channels where person-level tracking is impossible (cinema, out-of-home, television), a geographic experiment is the only credible causal design available, so the result depends on how well that experiment is designed. 02 ## The first failure is too little power to answer Statistical power is the probability that a test detects a real effect of a given size. The convention is 80% power at 5% significance, and a design that falls short of it will usually miss a genuine effect. More dangerously, any positive result it does produce carries little information. The number that makes this actionable is the minimum detectable effect: the smallest true lift the design could reliably find. If your design's minimum detectable effect is a 30% lift and the realistic effect of your campaign is 5%, no amount of patience fixes it. The test cannot answer your question and should not be commissioned. The inputs are your baseline conversion volume, the variance in that baseline, the number of independent markets and the duration. All of them are known before the campaign runs, which is why this is a pre-flight check rather than a post-hoc caveat. AdBuyMCP computes it before any readout and refuses to produce a lift figure when the design falls short, returning the required sample days and the minimum detectable lift instead. Its own phrasing is that reporting a number from it would be reporting noise. Meta's open-source GeoLift walkthrough puts the same step first: run a prospective power analysis, covering the detectable effect, the duration and the choice of markets, before the test begins. Source: [Meta Open Source, GeoLift walkthrough](https://facebookincubator.github.io/GeoLift/docs/GettingStarted/Walkthrough) (open-source docs). - [A worked power table for small budgets](https://adbuymcp.com/guides/incrementality-testing-on-a-small-budget) 03 ## The second failure is a control that was not held out A holdout is valid only if the campaign did not reach it. The common failure is that the campaign reached the control markets through national inventory, spillover from an adjacent region, or a channel whose targeting is coarser than the test design assumed. The comparison is then between heavily exposed and lightly exposed rather than exposed and unexposed. The result is an underestimate of the true lift by an unknown amount, presented with the same confidence as a clean result. Nothing in the output flags it. This is why the check has to be structural: verify that no candidate control market received delivery before running the comparison. The fix, when contamination is found, is specific rather than general: exclude those markets from targeting for the duration of the test window, or nominate a control the campaign does not buy. AdBuyMCP refuses to read out a contaminated design and names the affected markets rather than adjusting silently. Google's CausalImpact documentation makes the same warning about any design that leans on control series: if the controls were affected by the intervention, the estimate can be substantially under- or overestimated. Source: [Google, CausalImpact](https://google.github.io/CausalImpact/CausalImpact.html) (open-source docs). 04 ## Match markets on their pre-campaign outcomes The intuitive way to match a control market to a test market is on demography: similar size, similar income, similar age profile. Demographic matching is unreliable, because two cities that look alike on paper frequently behave differently commercially, owing to competitor presence, distribution, local seasonality or a dozen other things nobody is modelling. The right basis is the pre-campaign outcome series itself. If two markets tracked each other closely on the metric you care about before the campaign, a divergence afterwards is meaningfully attributable. If they did not, no amount of demographic similarity rescues the comparison. This is also what makes the analysis robust to shared shocks. Difference-in-differences compares the change in the test markets against the change in the controls, so anything affecting both (seasonality, a competitor's national campaign, the weather) cancels. What does not cancel is anything that hit only the test markets, which is the residual risk in any geo experiment and the reason a result should carry a confidence interval rather than a single number. 05 ## What to ask when you are handed a result Ask these questions before you act on a result. What was the minimum detectable effect, and is the reported lift comfortably above it? How were the control markets chosen, and on what series were they matched? Was contamination checked, and what did the check find? And what is the confidence interval, rather than the point estimate? A result presented as a single percentage with no interval and no design detail is not evidence. It may be true, but nothing about how it was presented allows you to tell. A vendor who tells you when a test cannot work is more useful than one who always returns a number, because a refusal tells you the design has to change, and a number from an underpowered test may be noise. Type your company website or work email to see who we think buys from you, how big your market is and what it costs at 3, 5 or 12 times a week. [Get my plan and price](https://adbuymcp.com/start) Free, with no card. Limits ## These limits apply to everything above This guide explains why tests fail and what to check. It does not claim AdBuyMCP has produced a causal result for a customer, because it has not. Nothing has run live here, and a geo-lift is graded causal only when every geo outcome is observed, every delivery row maps to a live line and control exposure is verified, which sandbox data cannot satisfy by construction. The engine, the power gate and both refusals run in the product today, and none of their output is yet evidence about a real campaign. A 30-minute call with the founder will tell you whether any of these limits matters for your market. [Book a 30-minute call with the founder](https://adbuymcp.com/contact#book) Where this came from ## Every figure above comes from one of these Outside figures link to the page that published them, with the day each was opened. Product facts name the part of AdBuyMCP they come from. ### Published sources 1. 01 [Meta Open Source, GeoLift walkthrough](https://facebookincubator.github.io/GeoLift/docs/GettingStarted/Walkthrough) Open-source docs · opened 24 September 2026 2. 02 [Google, CausalImpact](https://google.github.io/CausalImpact/CausalImpact.html) Open-source docs · opened 24 September 2026 ### AdBuyMCP’s own product facts - 01 The power analysis defaults, the refusal behaviour and the contamination check are read from AdBuyMCP's geo-lift engine and its campaign measurement routes. - 02 The difference-in-differences readout and the claim grading are from the same source. Read next ## These pages pick up where this guide stops Each one goes further into a part of this guide. ### [Run a geo-lift test](https://adbuymcp.com/how-to/run-a-geo-lift-test) The how-to walks through a test, including both refusals. ### [Measurement](https://adbuymcp.com/measurement) What the weekly report shows for each channel, and when a geo-lift test can say more. ### [Statistical power](https://adbuymcp.com/glossary/statistical-power) Why the engine refuses a test without enough power. The other 19 guides - [Most B2B buyers have vendors in mind before they search](https://adbuymcp.com/guides/uk-media-buying-for-b2b-marketers) - [Why cross-channel frequency capping rarely happens](https://adbuymcp.com/guides/frequency-capping-across-channels) - [Buying UK DOOH without a movement-data licence](https://adbuymcp.com/guides/buying-uk-dooh-without-movement-data) - [How to buy podcast ads in the UK](https://adbuymcp.com/guides/how-to-buy-podcast-ads-in-the-uk) - [The minimum budget for a startup to test TV advertising in the UK](https://adbuymcp.com/guides/minimum-budget-to-test-tv-advertising) - [How to read a media plan](https://adbuymcp.com/guides/how-to-read-a-media-plan) - [What a DSP does and when you need one](https://adbuymcp.com/guides/what-is-a-dsp-and-do-you-need-one) - [How UK addressable TV targeting actually works](https://adbuymcp.com/guides/how-uk-addressable-tv-targeting-works) - [How Sky AdSmart works](https://adbuymcp.com/guides/sky-adsmart-explained) - [How UK cinema advertising is booked](https://adbuymcp.com/guides/how-uk-cinema-advertising-is-booked) - [Choosing between cinema and CTV](https://adbuymcp.com/guides/choosing-between-cinema-and-ctv) - [Designing a geo-lift test that can actually be powered](https://adbuymcp.com/guides/designing-a-geo-lift-test) - [Incrementality testing on a small budget: a worked power table](https://adbuymcp.com/guides/incrementality-testing-on-a-small-budget) - [What a fidelity score measures](https://adbuymcp.com/guides/what-a-fidelity-score-is) - [First-party data in UK media buying](https://adbuymcp.com/guides/first-party-data-in-uk-media-buying) - [What Clearcast and the CAA actually check](https://adbuymcp.com/guides/what-clearcast-and-the-caa-actually-check) - [Running paid media from an AI agent without losing control](https://adbuymcp.com/guides/running-paid-media-from-an-ai-agent) - [MCP for marketers: what the protocol actually changes](https://adbuymcp.com/guides/mcp-for-marketers) - [The ad platforms' own MCP servers, and planning one budget across them](https://adbuymcp.com/guides/ad-platform-mcp-servers-and-cross-channel-planning) Bring your market ## We'll price an awareness plan for your own market. In a 30-minute call with the founder, we work out how big your market is, price 3, 5 and 12 times a week, and talk through how we'd balance the four channels for your market. [Get my plan and price](https://adbuymcp.com/start) [Book a 30-minute call with the founder](https://adbuymcp.com/contact#book) Free, with no card. Talk your market and your plan through with James Rooney, who built AdBuyMCP and wrote its code. ## Questions this guide gets asked ### Can I run an incrementality test on a small budget? Sometimes, and the budget is the wrong place to start. The power check reads the test market's daily conversions and how much they swing, and never the budget, so a business with steady daily conversions in one region can power a test that a larger one with lumpy conversions cannot. The small-budget guide has worked tables for finding your row before you spend. ### Is marketing mix modelling a substitute? No. Marketing mix modelling answers a different question: it estimates each channel's contribution from historical patterns without needing identifiers, which makes it valuable for channels that cannot be tracked. It remains correlational, hungry for history and sensitive to specification: two competent modellers can produce materially different answers from the same data. It is evidence, but it is not the same evidence as a controlled experiment, and presenting a model output as proof of causation is the failure that separate evidence labels exist to prevent.