Measurement and evidence
What is a holdout group?
Free, with no card. We read your homepage, draft who buys from you and show your monthly price before you join the waitlist. Nothing is bought until you've approved the plan and the ads.
what it means here
Here is what the term means for a UK buyer
The definition above holds anywhere. Buying media in the UK adds the detail below, including where AdBuyMCP does something different.
The most common way a lift test quietly measures nothing is a contaminated holdout: the control markets received some of the campaign, so the comparison is exposed against slightly-less-exposed. AdBuyMCP checks for this before the readout and refuses with a specific instruction: exclude those markets from targeting for the test window, or nominate a control the campaign does not buy.
How big should a holdout be?
Big enough that the exposed and held-out series can each move against their own daily noise. In a geo test that means more than one market on each side, with enough conversion volume that a plausible lift clears the power calculation. AdBuyMCP takes the top three matched controls per test market, and two well-matched, high-volume exposed markets usually beat six thin ones.
A national campaign that buys everywhere has no holdout at all, so its incrementality cannot be measured this way. Withholding media somewhere is what turns a report into a test.
How do you keep a holdout clean?
Exclude every held-out market from targeting for the test window, and check the delivery after launch. A targeting fallback that widens to national, a programmatic line delivering outside its polygons, or a search campaign with no geographic restriction will each leak spend into the control, and all three bias the result towards zero.
Where a rail reports no geography at all, a clean control can only be assumed. AdBuyMCP then reports control integrity as unverified, with the count of delivery rows that carried no geography, and an unverified check never counts as a pass.
Type your company website or work email to see who we think buys from you and what reaching them 3, 5 or 12 times a week would cost.
Free, with no card.
related terms
These terms sit next to this one
Most confusion in media buying comes from two adjacent terms being used interchangeably, so these are the neighbours worth reading next.
Geo-lift test
An experiment that runs advertising in some geographic markets and withholds it from comparable others, then compares outcomes between them. It can measure what the advertising caused without a login, a pixel or a match rate.
Incrementality
The share of an outcome that happened because of the advertising, measured against what would have happened anyway. It is measured by comparing an exposed group with a comparable unexposed one, which counting the conversions that followed an advert cannot do.
Matched market
A control market chosen because its historical outcome pattern closely tracks the test market's, so that a later divergence can be attributed to the campaign.
same group
These terms are also in Measurement and evidence
What you can prove afterwards, and how to tell a number that means something from one that does not.