The Ecomm Analyst

Every attribution argument eventually runs into the same wall. Platform ROAS, blended ROAS, multi-touch models, all of them are trying to assign credit for sales that happened. None of them can tell you what would have happened if you had turned the channel off. That counterfactual is the whole game, and the only honest way to get at it without an enterprise budget is a geo holdout.

Why your reporting can’t answer the real question

Say Meta tells you a campaign drove $40,000 last month at a 4x return. The question that actually matters is how much of that $40,000 you would have collected anyway, because those customers were already coming. Brand search is the classic example. A big chunk of the revenue Google claims on your branded keywords is people who typed your name in because they already decided to buy. Your reporting cannot separate the incremental sales from the ones that were going to happen regardless, because it only ever sees the sales that did happen. A holdout fixes that by deliberately creating a version of reality where the ad did not run.

What a geo holdout actually is

You split the country into two groups of regions that look similar in revenue and seasonality. In one group you keep running the channel you want to test. In the other you turn it off completely for a few weeks. Then you compare total sales between the two groups, not platform-reported sales, total store revenue. If the regions where you killed the channel sell roughly the same as the ones where you kept it on, the channel was not driving much incremental volume. If they drop, you now have a real number for what the channel is worth. The expensive incrementality platforms run a fancier version of exactly this with synthetic control groups, but the core idea is something a careful operator can run on a spreadsheet.

The part that trips people up is the measurement. You need clean total revenue by region for both groups, before and during the test. I pull that through ThoughtMetric because it keeps regional order data in one place and I am not stitching exports together by hand. Worth disclosing here that I work with ThoughtMetric, so treat that as a biased recommendation. Any tool that gives you trustworthy revenue by geography will do the job. The method matters more than the software.

Running one without a six-figure platform

Pick a channel you genuinely doubt, usually a prospecting campaign or branded search. Choose two sets of states that have historically tracked each other in revenue. Hold everything else steady, no new promos, no big creative swings, during the test window. Turn the channel off in one group for two to four weeks, long enough to see a signal but short enough that you are not bleeding sales on a hunch. Then compare the revenue trend between groups against how they tracked in the weeks before. The gap, if there is one, is your incremental lift.

Reading the result without fooling yourself

A holdout gives you a directional answer, not a lab result. Noise is real, especially for smaller stores where a single wholesale order or a viral post can swamp the signal. Do not over-read a 5% difference. Do look hard at a 30% one. And resist the urge to run the test once, get an inconvenient answer, and explain it away. The whole reason you ran it was that your reporting could not be trusted on this question. If the holdout says the channel is not incremental, that is the finding, even when it contradicts a ROAS you have been proud of for a year.

When it’s worth the trouble

• You suspect a channel is taking credit for sales that were already coming, branded search and prospecting retargeting being the usual suspects.
• A channel reports a great ROAS but scaling it never seems to grow total revenue the way the math says it should.
• You are about to make a real budget shift and want one piece of evidence that is not self-reported by the platform receiving the money.

A geo holdout is blunt, a little nerve-racking, and more honest than any attribution model you can buy. You do not need to run them constantly. Running one or two a year on the channels you are least sure about will teach you more about your real marketing efficiency than another quarter of staring at dashboards.