How DoorDash Ads keep consumers first with budget A/B experimentation

Ads play an important role at DoorDash in keeping delivery fees affordable while surfacing sponsored results consumers actually want. It is notoriously tricky, however, to test new ad features in a three-sided marketplace without disrupting the consumer’s in-app experience. 

The standard A/B testing playbook wasn't cutting it for our marketplace dynamics. This is why the team at DoorDash Ads has implemented the budget A/B framework, a powerful tool that ensures unbiased and reliable results for the benefit of consumers, restaurants, and consumer product goods (CPG) advertisers alike, all while ensuring ads are relevant and fees are low. This post discusses how the budget A/B framework works, the problems it solves, and why it was chosen over other options.

Marketplace experimentation challenges

Experimentation is a core pillar of DoorDash’s ads marketplace. If a new feature isn’t rigorously vetted, consumers may be shown off-target ads, or restaurants may face unexpected fee spikes. To avoid those pitfalls, we run tightly scoped experiments designed to capture reliable insights while keeping risk at a minimum.

DoorDash’s ads marketplace is a three-sided ecosystem featuring restaurants and CPG partners, consumers, and the platform all interacting and affecting one another. It's like trying to measure the ripple effects in a pond while someone else is also making waves, rendering it impossible to isolate what caused what. In this environment, classic A/B tests often fall short because such interactions violate A/B testing’s core assumption — the stable unit treatment value assumption. 

Specifically, two major problems arise:

• Cannibalization effect: In a typical A/B test on a new ad relevance model, ad creative, or bidding strategy, the treatment group may appear more effective simply because it consumes more of the shared budget. This doesn't mean it's genuinely better; it just means it’s capturing a larger share of the available spend, leading to inaccurate results. This is because the treatment group cannibalizes budget or impressions from the control group.
• Network effect: In marketplace experiments, network effects happen when different system components alter each other's results. For instance, if we are testing a new ranking algorithm, it changes auction dynamics and costs for other advertisers. This makes it hard to measure the true impact of any changes we test.

These issues become particularly critical when experimenting with budget pacing, bidding strategies, or any feature that impacts how advertising budgets are allocated. In these cases, standard A/B tests often produce misleading results.

Introducing the budget A/B framework

The budget A/B framework is DoorDash Ads' solution to these challenges. Pioneered by LinkedIn, it's an approach to online experimentation that ensures fair and unbiased results by creating separate universes within a single campaign, splitting the budget across treatment groups. This framework is designed to isolate experimental groups from each other. The key concept is to divide the overall budget into distinct pools, one for each experimental variant. The budget for each treatment group is thus segmented as seen in Figure 1.

Here's how it works:

1. Consumer-level randomization: Experiments are configured at the individual consumer level. This allows for the smallest minimum detectable effect (MDE), which will be discussed later.
2. Budget splitting: The total budget of a campaign is divided among the experiment groups. This is done proportionally based on a pre-defined split — for example 50/50 for a simple A/B test.
3. Independent universes: Each experimental group essentially operates in its own miniature marketplace with a dedicated budget and no interaction with other experimental groups.
4. Accurate measurement: By eliminating the cannibalization effect and feedback loops, the framework ensures that overall ad performance metrics truly reflect the impact of each experimental variation.

In essence, the budget A/B framework allows for accurate observation of what would happen if a feature rolls out to the entire consumer base.

Feature development benefits 

The budget A/B framework offers several important feature development benefits: 

• Unbiased results: This is perhaps the most important benefit. With the budget A/B framework, advertisers can be confident that the results of experiments are not skewed by budget cannibalization or interference. This means a more accurate evaluation of ranking algorithms, ad creatives, bidding strategies, and other features. Consumers also benefit because new features now only ship after we’re sure they don’t affect the overall DoorDash consumer experience.
• Clear campaign insights: By splitting the budget among treatment groups within a campaign, the framework allows for the accurate computation of campaign-level metrics, such as budget utilization rate and throttle rate.
• Faster innovation: With a more robust experimentation setup, DoorDash can iterate and innovate more quickly, bringing new features and improvements to the platform at a faster pace.
  • Although we focus on ads here, the same budget-split idea can power promotions, logistics incentives, or any feature where spend is the interference channel.

Assessing alternatives to budget splitting 

Before settling on the budget A/B framework, we considered a few alternatives. The most prominent were:

• Switchback experiments: In this design, the entire marketplace is treated as a single unit, with treatments switched back and forth over time. This method prevents interference between groups at any given time point, but allows interference across time points. While simpler to implement initially, switchback experiments have several drawbacks:
  • Carryover effects: Changes in one period may affect outcomes in subsequent periods, making it difficult to isolate the impact of the treatment.
  • Inefficiency: Switchback experiments require a significant amount of time for the effect of a new feature to become measurable because the switchback interval cannot be smaller than one day. Also, the experimental period has to be long enough to observe the true impact of the treatment.
  • Low power: Switchback experiments are often low-powered, meaning that a large sample size is required to detect effects. The MDE for switchback experiments is determined by the number of regional-time units and not the sample size.
• Analysis approach: This approach involves using modeling to adjust for biases observed in Bernoulli randomized experiments. However, this requires making untestable assumptions and is not as reliable as the budget split design.
• Comparable submarket experiments: This approach tries to compare different submarkets, but makes it difficult to eliminate pre-experimentation bias.
• Cluster-based randomization: This method randomizes units in clusters. However, it is difficult to avoid interference among clusters.

The budget A/B framework was chosen over these alternatives for several reasons:

• Unbiased: The budget split design is unbiased in any marketplace where buyers have a defined budget.
• High power: Budget A/B experiments have significantly higher statistical power than switchback experiments and comparable submarket experiments, allowing DoorDash to detect even small effects with confidence. The MDE in consumer-based experiments is six to seven times stronger than in switchback experiments.
• Practicality: While requiring architectural changes, the budget split design is practical to implement and does not require significant changes to the existing experimentation platform, or EP.
• Flexibility: The framework supports a wide range of experiments, including testing of bidding algorithms, pacing rules, and various ad creatives.
• No carryover effects: Unlike switchback experiments, the budget split framework does not have carryover effects because the budget is split simultaneously and the treatment is applied at the same time across all segments.
• No feedback loops: Unlike traditional A/B tests, the framework eliminates feedback loops between treatment groups, which tends to produce more accurate estimates.

Budget split design examples

To illustrate how the budget split design works, consider these practical scenarios:

Example 1: Testing a new bidding strategy

DoorDash Ads wanted to experiment with a new automated bidding strategy for all of its relevant ad campaigns. Without budget A/B, the new strategy, also known as a treatment, might inadvertently consume more budget than the old strategy, used as the control, making it look better simply because it spent more. With budget A/B:

• The total campaign budget — say, $200 for a single campaign — is split into two pools: $100 for the treatment and $100 for the control.
• Each strategy operates independently, using only its own $100 budget.
• The results can then show the true impact of the bidding strategy without budget interference.

Example 2: Testing intraday spend pacing strategies

DoorDash Ads needed to evaluate a new, more sophisticated pacing algorithm against our existing ASAP — as soon as possible — spending approach. The existing ASAP algorithm aggressively spent a restaurant's daily budget early on, potentially missing valuable advertising opportunities later in the day. Our proposed improvement was an intelligent intraday-pacing algorithm that would distribute budget based on historical impression patterns throughout the day.

It would be difficult to evaluate these dramatically different pacing strategies without budget A/B testing. The ASAP algorithm could consume most of the available budget before the intraday-pacing algorithm had a chance to execute its more measured strategy. Here's how budget A/B enabled a rigorous comparison:

• For a restaurant with a $400 daily advertising budget:
  • $200 was allocated to the ASAP spending universe
  • $200 was allocated to the intraday pacing universe
• Each pacing strategy operated in complete isolation:
  • The ASAP algorithm could aggressively spend its $200 during the morning hours
  • The intraday-pacing algorithm could strategically distribute its $200 based on historical impression data
  • Neither algorithm could access or compete for the other's budget pool
• This clean separation revealed the true performance implications of each strategy:
  • We could measure how many valuable impressions were missed by ASAP spending
  • We could quantify the benefit of aligning spend with peak consumer activity periods
  • Most importantly, we could evaluate authentic return on investment differences without budget interference effects

Key metrics to measure experimental performance

DoorDash tracks several key metrics to assess the performance of budget A/B experiments, including:

• Budget utilization rate: Measures how effectively the allocated budget is being spent.
• Throttle rate: Indicates the frequency of budget throttling.
• Impression and click-through rates: Uses impressions and click-throughs to assess the effectiveness of ads.
• Minimum detectable effect: Quantifies experimental power; the framework aims to minimize MDE, which is the smallest statistically detectable effect. The smaller the MDE, the more powerful the experiment is.
• Conversion rate: Measures the percentage of users who complete a desired action such as placing an order.
• Cost per action (CPA): Quantifies the cost of a desired action.
• Effective cost per mille: Quantifies cost per 1,000 impressions.
• Revenue: Measures revenue generated via the experiment.

System architecture and implementation

Implementing the budget A/B framework required significant changes to the DoorDash ads system architecture, particularly around tracking ad spend between different experimental categories.

We can model a campaign’s spend with a budget experiment running as shown in Figure 2.

When we need to evaluate a campaign’s current spending for a given budget experiment arm, we apply the following formula:

Experimentation platform integration

We leverage EP to split both the campaign space — to allocate a given ad campaign to a budget experiment — and the consumer space — to split the consumer universe per the experiment configuration. 

One of the challenges of splitting the DoorDash ads marketplace into discrete universes is that many restaurant advertisers operate with a low campaign budget, which means each transaction’s CPA represents a large proportion of the budget.

Thus, we had to be judicious about the splits to deal with multiple budget A/B experiments concurrently. The system currently supports only one campaign per budget experiment at a time; we leverage the platform to establish targeting rules to determine which campaigns should be part of which budget experiment.

Ads read-path modifications

Each ad auction request is annotated with budget experiment Information that indicates to which experiment and treatment arm a given ad candidate is allocated, as shown in Figure 3.

The ad pacing and bidding modules were updated to operate at the campaign-treatment level by using segmented budgets and providing bidding/pacing signals accordingly. 

Ads write-path modifications

Our ad attribution pipeline, as shown in Figure 4, needed to be changed to track budget spending at the experiment-segment level in addition to the existing campaign level. Each auction is labeled with an experiment context, ensuring that an experiment arm uses only its allocated budget.

This was needed to ensure that all systems requiring reads of the aggregated spend for a given campaign could also read aggregated spends for an experiment arm. It was also necessary to make schema changes to our transactional database to account for the new dimensions.

Creation of new experiment metrics

Changes were made to allow for campaign-level metrics to be calculated at the experiment-segment level. This lets us calculate budget utilization rates, throttle rates, and other KPIs within each treatment arm.

The modified system creates independent feedback and control paths for each treatment, enabling each to operate as a standalone marketplace.

Key implementation considerations

Several critical factors must be considered when setting up budget A/B experiments:

• Experiment duration: The experiment should run long enough to collect sufficient data. The time required to achieve MDE was drastically reduced with budget A/B. However, as with any other A/B experiment, the process should run sufficiently long to capture enough coverage and any seasonality effects.
• Minimum split: The minimum budget split must be determined to avoid potential revenue loss with small budget splits. This is specific to the DoorDash Ads restaurant business because the CPA to daily budget ratio tends to be large.

Conclusion

The budget A/B framework acts as an invisible safety mechanism for consumers, ensuring a smooth, pertinent, and interesting app experience. By testing ad changes on small, random portions of traffic, the framework safeguards the user experience from the potentially negative impacts of unproven changes until they are thoroughly validated.

That same rigor benefits restaurant and CPG partners, too; with cannibalization and cross-network noise stripped away, they get crystal-clear performance data, smarter budget allocation, and stronger return on ad spend. Internally, the DoorDash Ads team can innovate with confidence that every idea is measured against sound evidence before it reaches the wider marketplace.