Incrementality testing measures how many sales a channel or campaign actually caused, as opposed to how many it was present for. You split your audience or your markets into a group that sees the ads and a group that does not, run the test long enough to see a difference, and compare the two. The gap between them is the incremental effect. Attribution tells you which touchpoints were in the path to a sale. Incrementality tells you whether taking them away would have lost the sale.
How is it different from attribution?
Attribution assigns credit for sales that happened. It cannot tell you about sales that would have happened anyway, because there is no comparison group. That is why retargeting and branded search often look excellent in attribution reports, since they reach people who were already on their way to buying. A holdout test exposes this by showing what those people did when they did not see the ad. I went through the branded search version in brand search is the easiest thing to test and the hardest thing to be honest about.
What kinds of tests are there?
Three main types. Platform lift tests run inside Meta or Google and split users into exposed and held-out groups. They are built into the platforms, but they are measured by the platform selling you the ads. Geo tests turn spend off or down in a set of regions and compare sales against matched regions, which works for any channel, including ones you cannot target by user. On/off tests pause a channel entirely for a period. They are the crudest version and only work when the channel is big enough to move total sales visibly.
What does incrementality testing cost?
Anywhere from nothing to a significant line item. A geo holdout you design yourself costs the sales you might lose in the holdout regions plus the time to analyze it. Several attribution platforms now bundle testing. Triple Whale includes incrementality testing and GeoLift in Compass, which comes with its Enterprise plan and is an add-on for eligible Foundation and Automate customers. TrueROAS sells True Horizon, which includes incrementality reports, from $449 a month. Klar and Admetrics both list incrementality or experimentation as part of their products. Dedicated testing platforms are another route, and I compared several in Haus alternatives.
When is it worth paying for?
When the decisions it informs are bigger than its cost. If you spend on one or two channels at modest budgets, a self-run geo test once or twice a year is usually enough. Once you spend across several channels and a wrong call on one of them costs more than a test, regular testing starts to pay for itself. Triple Whale’s own help center recommends Compass for businesses with at least $10M in annual GMV so the models and experiments have enough data to work with, which is a reasonable marker for where paid tooling starts to make sense. I mapped this out by spend level in MTA, MMM, or incrementality, depending on what you spend.
What do you need before running a test?
Enough volume that the difference between groups is bigger than normal week-to-week noise. A clear outcome, usually total orders or new-customer orders, and not attributed revenue. A test long enough to cover your typical purchase cycle. And a decision written down in advance for each possible result, so the test actually changes what you do. My simplest version is in running a cheap geo holdout to sanity-check a channel.
Does it replace attribution?
No. Tests are slow, and you can only run a few a year. Attribution is what you use every day to rank campaigns, ads, and creative within a channel. The two work best together, with attribution telling you where to look and a test telling you whether what you see is real. ThoughtMetric, which sponsors this blog, sits on the attribution side of that line, and I use attribution data to decide which channel to test next rather than as proof that a channel is incremental. If you want to know how far to trust attribution in the meantime, I looked at one popular tool in is Triple Whale’s attribution accurate.
Where to start
- One or two channels at modest budgets? Run a self-designed geo holdout once or twice a year.
- Big retargeting or branded search lines? Test those first, since they are the most likely to be overcredited.
- Already paying for a platform with built-in testing? Use it before buying a dedicated tool.
- Decide what you will do with each result before the test starts.
Leave a comment