When to stop arguing about attribution and run a holdout

Attribution tells you which touchpoint to credit. A holdout tells you what would have happened if you had not spent the money. Only one of those is a fact, and it is not the first one.

If your last three marketing meetings ended in an argument about whether the platform numbers or the analytics numbers are right, the argument is not resolvable. Both are models. Google Ads reports modelled conversions where consent is missing. GA4 fills gaps with behavioural modelling. Meta reports on its own view of a click and a view. None of them can see the counterfactual, because the counterfactual does not exist in their data.

A holdout creates it.

What a holdout is

You deliberately withhold advertising from a slice of your market, keep everything else the same, and compare. The slice that gets no ads is the control. The difference between the two, once you allow for baseline differences, is incrementality. That is the only number in marketing that does not change when a platform changes its reporting.

There are two practical versions.

Geo. Turn off spend in a set of postcodes, regions or metro areas. Best for anything where you cannot identify the individual, which is most brand, video, PR and broad-reach media.

Audience. Hold back a randomised percentage of a known list, usually for email, SMS, remarketing or a customer-base campaign. Easier to run, narrower in what it tells you.

The geo version, step by step

1. 1.Pick your unit. In Australia, capital-city metro areas are usually too few and too different to compare cleanly. Use statistical areas, or groups of postcodes, so you have at least twenty units to work with.
2. 2.Pull 12 months of weekly conversions and revenue per unit. You need history, because you are matching on trend, not on size.
3. 3.Split so control units total roughly 20 to 30 per cent of baseline conversions. Match on the shape of the trend line, not population. Two regions with the same seasonality behave alike; two regions with the same population often do not.
4. 4.Freeze everything else. No new landing pages, no pricing changes, no PR push in one half.
5. 5.Run for at least four weeks, longer if your sales cycle is longer than a fortnight. A test shorter than one full purchase cycle measures timing, not effect.
6. 6.Compare actual to predicted. Use the pre-period to build the expected line for the control, then read the gap.

Google's Meridian is now embedded in Google Analytics 360 with natural-language scenario planning, announced at Marketing Live on 20 May 2026. It is a paid-tier feature. If you are not on 360, the open-source version and the geo-test libraries that sit alongside it do the same job with a data analyst and a week of work.

How much volume you need

This is where most tests fail, and almost nobody says it out loud.

A holdout can only detect an effect that is bigger than your week-to-week noise. As a rough working rule, if the treated area produces fewer than a few hundred conversions a month, you will not resolve anything smaller than a very large swing, so the test will come back inconclusive and everyone will go back to arguing. Low-volume, high-value businesses should hold out on a leading indicator instead: qualified enquiries, booked meetings, quote requests. More events, same direction.

If you genuinely have only forty deals a year, do not run a holdout. Run a switch-back test on a leading metric, or accept that you are steering on judgement and say so.

What breaks a holdout

Spillover. Radio, national PR, out-of-home and anything with a metro footprint bleeds into your control regions. Your measured effect comes back smaller than the truth. Either accept the bias and treat the result as a floor, or exclude contaminated regions from both sides.

Brand search. People see the ad, then search your name, then land through organic or a branded paid click. If you only count non-brand paid conversions, you will conclude the campaign did nothing. Count all conversions in the region, from every channel. That is the whole point.

Someone turning things back on. A well-meaning person un-pauses the control after nine days because the numbers look bad. Lock it. Write the end date down. Tell the team what the dip means before it happens.

Seasonal cliffs. Do not run a four-week test that straddles Christmas or the end of financial year unless both halves straddle it equally.

When not to run one

If the channel is under about 10 per cent of your total spend, the answer will not change your budget, so the test is not worth the lost revenue in the control. If you are still in the first six weeks of a campaign, you are measuring a learning phase. And if the honest reason for the test is that someone wants a channel killed, decide that in the open instead of dressing it up as measurement.

There is a real cost. You are switching off working advertising in part of your market for a month. Budget for that as the price of knowing.

What you do with the answer

You get an incrementality multiplier: the ratio between what the platform claimed and what actually moved. Apply it as a standing correction to that channel's reported numbers, and re-test twice a year or after any large change in mix. Now the platform report is useful again, because you know how much to discount it.

The next time a platform changes its attribution, its reporting windows or its conversion modelling, your holdout still means what it meant. That is what makes it worth the month.