An ad platform can report conversions that would have happened anyway. An attribution tool can assign credit to a click that captured existing demand. Incrementality testing addresses a different question: how many additional outcomes occurred because the marketing activity ran?
What incrementality means
Incremental sales are the outcomes above what would likely have happened without the activity being tested. The missing outcome is the counterfactual, so a sound test needs a comparison group or another defensible way to estimate it. The stronger the design, the more confidently a team can separate campaign impact from seasonality, promotions, and unrelated changes.
Choose a test design
| Method | How it works | Best fit | Watch-out |
|---|---|---|---|
| Audience holdout | Randomly withhold ads from a portion of eligible users | Platforms and audiences with reliable randomization | Contamination across devices or channels |
| Geo experiment | Run different spend levels in matched regions | Large markets with regional variation | Regions may differ in demand or distribution |
| Switchback test | Alternate treatment and control periods | Some channels with controllable timing | Seasonality and carryover effects |
| Matched-market analysis | Compare treated areas with similar untreated areas | When randomization is unavailable | Matching cannot remove every hidden difference |
Step 1: Write the hypothesis
Be specific. For example: “Increasing prospecting spend on Channel A by 20% in the treatment markets will increase net new-customer contribution margin over four weeks.” Define the channel, audience or geography, budget change, outcome, time window, and what result would change your decision.
Step 2: Select the outcome before launch
Revenue is easy to understand but can hide discounts, returns, and product costs. Where possible, use incremental contribution margin or new-customer gross profit. Choose one primary metric and a small number of guardrails such as total orders, average order value, and new-customer share.
Step 3: Protect the comparison
Randomize where possible. Keep pricing, promotions, inventory availability, and other media activity as similar as practical across groups. Record any unavoidable differences. Avoid changing the test design after seeing early results; that creates a temptation to stop when the numbers look favorable.
Step 4: Calculate lift
A simple lift calculation is: (treatment outcome per eligible unit − control outcome per eligible unit) ÷ control outcome per eligible unit. Multiply the estimated incremental outcomes by the relevant margin per outcome, then subtract the incremental media and operating cost to estimate incremental profit.
For geo tests, compare the change in treatment regions with the change in control regions, not just the raw totals. Use an appropriate statistical method and report uncertainty. A small positive point estimate with a wide confidence interval is not proof of success.
Test planning checklist
| Before launch | During the test | After the test |
|---|---|---|
| Define hypothesis and primary metric | Keep budgets and eligibility stable | Report lift and uncertainty |
| Choose randomization or matched groups | Monitor tracking and stockouts | Calculate incremental profit |
| Estimate sample size and duration | Log promotions and external events | Decide scale, stop, or retest |
| Set a stopping rule in advance | Avoid peeking-driven changes | Document what the test cannot prove |
Common mistakes
Testing during a major sale without accounting for the promotion can confuse demand changes with ad impact. Running a test that is too short can miss delayed purchases. Allowing control users to receive the same campaign through another channel contaminates the comparison. Measuring only platform-attributed ROAS can also reintroduce the very bias the experiment was designed to avoid.
Frequently asked questions
What is an incrementality test?
It is a controlled comparison designed to estimate the additional outcomes caused by a campaign or channel, relative to a credible holdout or control group.
How long should an incrementality test run?
Long enough to capture the campaign’s expected response and conversion lag while collecting an adequate sample. Set the duration and stopping rule before launch.
What should I measure in the test?
Choose a primary outcome such as incremental orders, revenue, or contribution profit. Track spend and guardrail metrics, and keep the measurement window consistent across groups.
Can attribution software replace an incrementality test?
No. Tools such as ThoughtMetric can help inspect attributed channel performance, but causal lift requires a valid experimental or quasi-experimental design.
ThoughtMetric can be included in the measurement stack when reviewing channel-level e-commerce attribution before and after a test. Use it to inspect attributed performance, but define the test’s control group, primary outcome, and measurement window in advance; the experiment—not an attribution dashboard—determines whether the campaign caused incremental lift.
Leave a Reply