Incrementality Testing
The Complete Guide to True Causal Measurement
🔬 The Test That Proved Everyone Wrong
The VP of Display was confident. “Our display campaigns drive $4 in revenue for every $1 we spend. The ROAS is incredible.”
The CFO approved a budget increase. Then I suggested a simple test: turn off display for 20% of users for 30 days.
The results were devastating. Revenue didn’t drop. Not $4 less. Not $1 less. It dropped $0. That’s right—those “high performing” display campaigns were driving zero incremental revenue.
The customers would have bought anyway. The display ads were just getting credit for sales that would have happened regardless. Once we turned off display, the ROAS plummeted because there was no increment to measure.
That’s the power of incrementality testing. It answers THE most important question in marketing: “What would have happened if we didn’t run this campaign?”
What is Incrementality Testing?
Incrementality testing (also called holdout testing or geo-experiments) is a controlled experiment that measures the TRUE causal impact of a marketing campaign by comparing results between exposed and unexposed audiences.
Here’s the critical distinction:
- Attribution measures correlation: “This customer saw our ad and converted.”
- Incrementality measures causation: “This customer converted BECAUSE they saw our ad.”
The difference? Everything. A customer who searches for your brand and sees a display ad was going to convert anyway. Attribution gives the display ad credit. Incrementality shows the display ad added nothing.
How Incrementality Testing Works
There are several approaches:
1. Holdout Testing
Randomly select a percentage of your audience (typically 10-20%) and exclude them from seeing your campaign. Compare their conversion rate to the exposed group.
2. Geo-Experiments
Turn off campaigns in specific geographic regions (DMAs or states). Compare sales in test vs. control regions.
3. Switchback Tests
Turn campaigns on and off in alternating time periods. Measure the difference in each period.
Increment = Exposed Conversions – Control Conversions
Why Incrementality Testing is Critical in 2025
1. Attribution is fundamentally broken
Attribution conflates “caused” with “correlated.” If a customer would have converted anyway, attribution gives false credit. Incrementality separates signal from noise.
2. Privacy changes are here
With less individual-level tracking, aggregate incrementality testing becomes more important, not less.
3. Marketing budgets need justification
When budgets are tight, you need to know what actually drives incremental revenue. Incrementality testing provides proof.
💡 Pro Tip
Start with your highest-spend channels. It’s counter-intuitive, but incrementality tests are most valuable for channels that “look” like they’re working well. If you’re spending $1M/month on a channel with “great” attribution, run an incrementality test. You might be shocked.
Common Mistakes in Incrementality Testing
⚠️ What to Avoid
Mistake #1: Running tests too short
You need 2-4 weeks minimum for statistical significance. Shorter tests have too much variance from normal fluctuations.
Mistake #2: Not having enough sample size
If your test group is too small, you won’t reach statistical significance. Calculate your sample size needs before starting.
Mistake #3: Ignoring external factors
Seasonal changes, promotions, competitive activity—all can skew results. Account for these in your analysis.
Mistake #4: Testing too many things at once
You need clean test design. Test one campaign or channel at a time. Otherwise you can’t attribute the impact.
Types of Incrementality Tests
| Type | How It Works | Best For |
|---|---|---|
| Holdout | Exclude random audience % | Digital campaigns |
| Geo | Turn off in test regions | TV, radio, OOH |
| Switchback | Turn on/off alternatingly | Seasonal businesses |
| PSA | Ad blackout periods | Brand campaigns |
Implementing Your First Incrementality Test
Step 1: Choose your test
Select the channel or campaign to test. Start with highest spend.
Step 2: Determine sample size
Calculate how many conversions you need for statistical significance. Many tools online calculate this.
Step 3: Set up test/control
In digital platforms, create exclusion audiences. For geo-tests, select regions.
Step 4: Run the test
Run for minimum 2-4 weeks. Don’t make changes mid-test.
Step 5: Analyze results
Calculate conversion lift in test vs. control. Apply statistical significance tests.
FAQ: Incrementality Testing Questions
Q: How much does incrementality testing cost?
A: The test itself is “free” in terms of platform costs—you’re just not showing ads to a group. But you’re “losing” the potential revenue from that group. Budget for 5-10% revenue impact during tests.
Q: How long does a test take?
A: 2-4 weeks minimum for reliable results. Longer is better—aim for 4-8 weeks.
Q: Can I test ALL my channels?
A: No—don’t starve your business. Test one channel at a time. Run tests sequentially or in different geo regions if you need to test multiple.
Q: What if results aren’t statistically significant?
A: That’s still valuable information. If you can’t prove incrementality, assume the channel might not be adding value. Run larger tests or accept that the channel’s contribution is minimal.
✅ Incrementality Testing Checklist
- Identify highest-spend channel to test
- Calculate required sample size
- Set up test/control groups
- Run test for 2-4+ weeks
- Collect and analyze data
- Apply statistical significance
- Calculate true ROAS
- Present findings to stakeholders
- Make budget decisions with proof
- Schedule regular tests
Start Testing Incrementally
The only way to truly know if marketing drives incremental revenue is to test. Pick your highest-spend channel, set up a simple holdout test, and see what the data tells you.
Be prepared to possibly be proven wrong. That’s the point.
