Influencer Hub homeDiscuss a pilot
Menu

Measure Creator Campaign Incrementality

Measure creator campaign incrementality with a holdout. Compare exposed and withheld groups, account for spillover and explain the uncertainty in pilot lift.

Two ink wash samples with a small difference in tone.

Measure creator campaign incrementality by comparing an outcome in a group eligible for the campaign with the same outcome in a comparable holdout. Choose the assignment unit before launch and account for any spillover between groups.

What can a holdout answer?

Incrementality is the change caused by the campaign relative to what would have happened without it. A holdout is a group withheld from the specific intervention being tested. An experiment on paid amplification can estimate the impact of that amplification while both groups still encounter organic creator posts. IAB and IAB Europe incremental measurement guidelines

  1. Choose a primary outcome, such as first paid orders per eligible customer, and a minimum effect that would influence the next decision.
  2. Choose an assignment unit you can control. Geographic experiments can assign regions and compare outcomes across them. Check audience overlap, planned promotions and whether you have enough independent units. Google Research on geo experiments
  3. Define the observation period, exclusions and analysis before allocating campaign delivery. Keep the assignment and observed delivery available for review.

What can a small pilot measure?

Use the observed rates to describe the result while allowing for sample uncertainty. Fictional example. Both groups contain 4,000 eligible people, with rates of 2.0% in a comparable baseline period. During the test, the eligible paid-exposure group records 100 orders and the holdout records 88.

Measure Eligible for paid exposure Holdout
Test order rate 2.5% 2.2%
Difference 0.3 percentage points Comparison group
Relative difference About 13.6% Based on 2.2%

The 12-order observed difference could include ordinary variation. Estimate uncertainty using a method that respects the assignment unit. A geo test needs geographic units in the analysis.

What goes in the readout?

Report the intervention, outcome, sample, group counts and uncertainty interval. Log any delivery contamination, surprise promotion or ambiguous eligibility. Keep creator link and code sales in the attribution report as a separate descriptive result.

If you evaluate tools to run the test, include a separate test for exporting assignments and outcomes in the software trial.

Use the holdout planning worksheet to decide whether the test can detect the effect you care about. Add the test result to the campaign KPI report with its own scope and decision.

Define the experiment you can act on

Begin with a decision. If the next question is whether to fund paid amplification of creator posts, test that intervention where assignments can be controlled. If you want to measure all effects of a public creator partnership, organic exposure may cross groups. Be explicit about what will vary and what stays available in both groups. An experiment that adds paid delivery to an existing campaign answers the effect of adding that delivery under that setup.

Choose a primary outcome and a measurement period ahead of time. A primary outcome might be paid first orders over an eligible audience. Secondary outcomes might include merchandise revenue or landing page visits. Define the eligible population from pre-treatment information and retain the original assignments. When visitors are excluded because they didn’t see the ad, the comparison no longer matches the randomized groups as assigned.

Test orders per 100 eligible people
  1. Eligible for paid exposure2.5%
    2.5%
  2. Holdout2.2%
    2.2%
Fictional results for two groups of 4,000. Use observed order rates and assess uncertainty separately.

Plan allocation around interference

Use a unit you can control, such as an account or geography. Check whether shopping behavior, promotions and public creator content will cross units. Store-level messaging could reach multiple cities. A regional announcement might affect everyone, including your holdout. Document the reach of other activity before and during the run. A good design protects the comparison and makes known interference visible in the final report.

Check the unit size, number of units and observed background variability before booking the test. A national lift percentage alone cannot tell you if your sample can detect the effect you care about. If available, use historic period data to examine natural demand variation and choose a method appropriate to the assignment. The Google geo experiments paper describes a design where regions are assigned to conditions and observed over time.

Explain what uncertainty means in the example

In a simplified fictional example where people were assigned independently and outcomes are binary, the difference between 100 of 4,000 and 88 of 4,000 is 0.3 percentage points. A normal approximation to the unadjusted difference has a standard error of about 0.34 percentage points. An illustrative 95% interval is roughly −0.36 to +0.96 percentage points. The interval includes both a small negative difference and a lift larger than the point estimate.

Use that calculation only for the stated assumptions. A geo or household assignment needs analysis that reflects clustering, and an adjusted estimator can yield a different interval. Explain the inference in context. The team saw a positive observed difference and the simple sample still leaves substantial uncertainty. Estimate what additional units, longer exposure or a stronger intervention would be needed before making a budget decision.

Write the decision before looking at the first result

Agree on what an inconclusive test will change. A team might keep spend steady while improving logging, commission new creative for a second test, or choose a bigger intervention. Record the expected effect that would justify additional investment and distinguish it from statistical significance. Provide dates, assignments and outcome definitions so another reviewer can interpret the result.

Report assignment counts, actual delivery, missing outcomes, method and the uncertainty interval. Show attributed orders in their own section. Those records help the finance team calculate creator pay even if the experiment is underpowered. Keep the reporting period and return treatment visible so marketing and finance understand the differences between the two figures.

Build a result summary you can read without the research file

Include the original assignment, actual delivery, outcome rate, difference and uncertainty in one view. If an eligible group included people who received no ad, keep them in the assigned analysis where the design requires it. A group formed by observed exposure can differ in important ways from the group that received randomized assignment. Give the reader both figures where available and explain which one answers the planned question.

Fictional reporting scenario. A group of market areas was assigned to receive the added creator ads. One area delivered little because a scheduled offer was unavailable, and another received heavy organic spillover. State both events and when they occurred. Ask the analyst to show the primary analysis and any pre-planned sensitivity check separately. The decision maker can then see what happened and what additional test conditions would make the comparison cleaner.

A result can be useful when the smallest plausible difference is uncertain if it helps diagnose why a proposed measurement failed. For example, clean assignment but near-identical delivery makes a contrast too weak to answer the original question. Good delivery and highly variable outcomes may call for more independent units or a different period. Record expected and observed delivery before scheduling another experiment.

Diagnose a small test that produces an attractive point estimate

Fictional pilot. A subscription brand can assign 4,000 eligible households to access a creator campaign and 4,000 to a holdout. The outcome is the number of assigned households making at least one qualifying purchase within a fixed period. Suppose 124 campaign households and 108 holdout households purchase. The observed difference is 3.1% minus 2.7%, or 0.4 percentage points. Under independent household assignment and simple binomial assumptions, the rough standard error of that difference is about 0.38 percentage points. An approximate 95% interval is −0.34 to +1.14 percentage points. A different assignment unit or dependence between households would require a corresponding analysis.

Measure Campaign access Holdout
Assigned households 4,000 4,000
Households purchasing 124 108
Purchase rate 3.1% 2.7%

Evaluate assignment first, including participants who never see the creator content. Some of the holdout may see an organic repost. Record that exposure risk and explain what the test did and did not constrain. Excluding people based on whether they were actually exposed after assignment can distort the comparison. If the creator package ran in regions rather than households, analysis should use the assignment design and account for variability between regions.

Select the next decision before looking at the result

Say the brand would consider expanding only if the improvement is at least 0.5 percentage points at a sustainable cost. The example interval covers effects on either side of that threshold. Management can keep the uncertainty in its decision instead of treating the lower result as a proof of failure or the upper estimate as promised sales. A larger study may help when the anticipated gain justifies the cost. A narrower creative test might be more useful if the biggest unknown is whether the content can be accepted and delivered consistently.

Retain the written test specification, assignment data, outcome counts and deviations behind the summarized result. Reconcile ineligible or duplicate records before analysis and report any missing outcomes by group. Preserve the rule for return-adjusted conversions if the business uses them. Include a separate reporting table for creator links and codes when those fields have a role in payouts. On the closeout call use the tests to answer distinct questions: which sales the ledger credits under the agreement, and whether the assigned opportunity plausibly changed total purchases.

Write the analysis plan for an imperfect launch

Before the first publication, identify common events that could affect the design. If a brand-wide sale begins in the middle of the test, document whether its reach differs by assignment group. If one area has different inventory, flag the discrepancy and preserve the planned eligibility rules. Keep a log of deviations and when the team learned about them. Choose which deviations require a sensitivity analysis, and make that analysis distinguishable from the pre-specified result.

Agree what “purchase” means. A household that orders twice can count once for a purchase-rate outcome and twice in an order-volume outcome. A return-adjusted revenue outcome requires a cutoff and may need more time to mature. If reporting includes both orders and revenue, choose one primary measure to steer the test and use secondary measures to explain the result. Specify what happens when a household has missing assignment information.

After a small test, share three practical numbers with their denominators: participation or delivery, the primary outcome and the uncertainty range. A single return-adjusted total may conceal whether effects were driven by order volume or by the value of a few purchases. If any small group has sensitive or potentially identifying results, use a level of aggregation the business approves. Keep the raw evidence accessible to the people who can validate the design.

Discuss a pilot