Skip to content

DoorDash

In a marketplace, risks inevitably converge on two key underlying questions: Who is this, and how confident are we that this action is legitimate? Some examples of what we mean by this: A support agent evaluating a refund request needs to know these answers. A promo engine deciding whether to extend an offer can’t go ahead without this information. A bot-mitigation system screening a login attempt for automated traffic is hamstrung without this data. The specifics differ — the channel, the signal, the threshold — but the core questions don’t change. 

While these questions aren’t new, AI increasingly has introduced new pressure from both sides of the equation; the same advances that make it easier to generate convincing fake behavior at scale also make it possible to identify the actors with richer signals than ever before.

Yet identity has never been treated as its own horizontal capability. Instead, most systems have been built product-first — a risk engine for orders, a model for promo targeting, or a rule for account-takeover detection. Each one implicitly answers a version of the who-is-this question for its own purposes, but none of them share that answer. This leaves a set of partial views of the same person, each accurate for its own purpose but walled off inside in its own system.

Identity intelligence starts from a different premise: If the question is shared, the signal that answers it should be, too.  That’s a platform problem — solved once, centrally — that’s distinct from how any given system chooses to act on the answer, which looks different depending on where it’s applied. Treated as a first-class, horizontal capability — not a byproduct of any one product — identity becomes a common thread running through nearly every integrity use case. That’s our central thesis here; identity is one of the highest-leverage but least appreciated capabilities in integrity precisely because no one builds it as its own thing.

Identity comes into its own

If that who-is-this question is so universal, why hasn’t identity been treated as core infrastructure from the start?

In the past, every integrity surface — order risk, account risk, or promo risk — has solved its own problem first and in its own way, with its own device signals, linkage logic, and working definition of “the same person.” That’s how specialized problems tend to be solved, with a rule here and a model there, each purpose-built and quick to ship. A shared identity layer is a different kind of investment: It doesn’t belong to any single surface, so no one surface is naturally positioned to build it, even though every one of them would benefit.

The result to date has been an identity signal that isn’t reusable off the shelf. Every new team that wanted that answer had to onboard it from scratch with a bespoke, one-off integration instead of plugging into a standard interface. For years, identity signals have existed everywhere but were owned nowhere; each team paid to re-derive a partial answer while no one captured the compounding value of answering the question once and sharing it.

Building an identity intelligence platform

At DoorDash, we decided to close that gap by designing new infrastructure. Instead of each system building its own linkage logic on top of its own signals, we wanted identity to become something built once and consumed everywhere.

In practice, that meant building three layers on top of one shared signal store:

  • First, identity resolution: An entity-resolution graph with 1B+ edges, built mostly with unsupervised clustering and link-prediction techniques, that investigates which accounts might be the same person. 
  • Second, relationship intelligence: A related but different graph question: Which people, accounts, devices, and assets are associated with each other? This layer acknowledges that two accounts can be meaningfully connected without being the same human. 
  • Third, decisioning models: Typically supervised learning models, such as gradient-boosted classifiers, trained on known outcomes for a specific use case. This layer consumes the first two layers to return a real-time answer to the question: Is this particular event, for this particular decision, legitimate?
FIGURE 1: A SHARED SIGNAL STORE FEEDS TWO GRAPHS — WHO SOMEONE IS, AND WHO THEY’RE CONNECTED TO — WHOSE OUTPUT ONE SCORING LAYER TRANSLATES INTO A SINGLE DECISION.

We’ll use one of the hardest cases for identity resolution as an example — a customer who just signed up, places a first order, and files a support request. There’s no history to compare against, nor is there likely to be a strong match to an existing account; identity resolution alone doesn’t have much to go on. What the account does have, from the moment it exists, is a handful of identity attributes, such as a device, a phone number, an address, or a card on file. The platform already tracks which accounts each of those attributes has appeared on, so an overlap on any of them is visible immediately, without waiting for a single order or support request. A shared address, for example, is a start, but a bare shared-or-not flag is too coarse to act on; it doesn’t say whether that address sits under two accounts or 200, whether they all showed up on the same day or over years, or whether that pattern is even unusual. That’s relationship intelligence’s job: Not resolving who someone is, but characterizing how connected they are. It combs through more than 100 derived signals computed per account, each one capturing how many other accounts share a given attribute, how recently, and how anomalous that looks next to an ordinary customer.

But even all of this isn’t a decision. A simple fact about the graph — such as, this address has 40 accounts attached to it, most created in the last week — is not a verdict on this particular support request. Turning one into the other is the decisioning model’s job. It is trained on outcomes to know how much that pattern should matter for this specific kind of request; its conclusion is made available in real time so that it’s ready the moment a request comes in.

That’s why the platform needs all three layers, not just the first. Identity resolution answers whether two accounts are the same person, while relationship intelligence gives a useful answer even when resolution comes up empty; neither is a decision until a model built for that specific use case turns it into one.

The platform separates signal computation from scoring (Figure 2). Features are aggregated asynchronously and ready within minutes of whatever changed. A checkpoint retrieves what it needs in 10 to 15 milliseconds rather than waiting on any of the work that produced it. Scoring then sits wherever the surface’s latency budget wants it; it’s called inline when there is room, or can be triggered as soon as the features land so that the score is stored and read at decision time.

Figure 2: IDENTITY SIGNALS ARE BUILT ON THEIR OWN CLOCK AND STORED ONCE, SO NO DECISION WAITS ON THE AGGREGATION BEHIND THEM.

One foundation, many use cases

Downstream systems — for instance, a support flow, a promo engine, or a checkout gate — don’t all need the same layer. Some systems can get an answer straight from the identity and relationship graph; others need it distilled further through the appropriate decisioning model into a real-time score built for that specific decision. These never need to know what the graph underneath looks like.

Consumer identity clustering is the clearest example of that platform in action. Such accounts look like separate people but are actually the same person. This specifically is identity resolution, determining whether two accounts collapse into one human, not just whether they’re somehow connected. Under the hood, the identity cluster weighs the reliability of each signal connecting two accounts and how unlikely it would be for two different people to share it by coincidence. The more improbable a shared signal is, the more weight it carries on its own; more common or easier-to-share signals only carry weight in combination, when several of them add up. Because it runs continuously against DoorDash’s identity graph rather than as a periodic batch job, it can be checked the moment a login, checkout, or refund request comes in.

Once that cluster exists, the same underlying identity and relationship signal — reused rather than rebuilt — shows up across the rest of integrity’s surface area. In support workflows, that same clustering helps discern a genuine service failure from one person filing similar-looking claims across several accounts. In promotions, it decides eligibility by discerning a new customer from a person who already claimed an offer under a different identity. In account security, it flags a login with the right password that doesn’t behave like the account owner. Scaled up from one person to many, a related but distinct layer — relationship intelligence, not identity resolution — is what surfaces coordinated abuse; this may take the form of a ring of accounts that look independent individually but can be seen to move as a single operation after their connections are mapped. In bot mitigation, it distinguishes a human working through friction from a script working at scale. 

None of these needed a new kind of signal, just the same one, routed to a different decision each time. That’s what a shared platform makes possible: One substrate, many decisions. And once that platform exists, it doesn’t have to stop at direct identifiers.

Identity is more than an identifier

Direct identifiers were the obvious place to start, of course — a device ID, a phone number, an email, or a card on file. These are powerful signals that are hard to fake and often enough on their own to confidently answer the who-is-this question. But a sufficiently determined bad actor can still work around them with a new device, a new number, a new email, or a new card — making the direct trail go cold.

Behavior helps close the gap. How someone moves through a marketplace — the rhythm of their actions, their browsing patterns, checkout actions, or interactions with support — also provides a form of identity, one that tends to persist even when the identifiers around it are swapped out. It doesn’t replace direct signals; it’s a complementary layer for the cases where direct signals alone fall short.

This reframe matters because it changes the definition of “the same person.” Two accounts with completely different devices and payment methods can still be recognized as the same person if behavior enters the picture. Two accounts that look unrelated by every direct identifier can still warrant a closer look, if their behavior lines up too closely to be a coincidence.

Identity’s AI future

The clearest way to measure a platform like this isn’t in any single win; it’s in the customers who never notice anything happened, yet who benefit from a more streamlined experience. Identity intelligence removes friction that honest customers have been forced to absorb; a more confident system that moves more quickly to a conclusion about who it’s dealing with relieves them of that burden. 

This same idea can be applied to richer inputs. Today, our behavioral signals still rely heavily on numerical summaries. But LLM and generative AI models can enable us in future to absorb insights from unstructured text, images, and free-form conversations at much lower development costs than before. 

Unlike bootstrapping text and image sub-components from scratch in traditional neural network models, we can now either use readily available genAI endpoints or leverage the output layer of pretrained large models to generate text/image-related features such as issue description sentiment analysis or duplicate image checks. Additionally, we can explore transformer-based model architectures to curate user identity embeddings that could potentially revolutionize how we define clusters in the identity graph. In the architecture above, these show up as two separate paths (Figure 1): an LLM extracting features directly from free-form appeals, and text embeddings becoming a new kind of edge in the identity graph. 

That said, the same AI advances that make it easier to build a trustworthy identity signal also make it easier to fake one with more convincing synthetic behavior, automated variation across accounts, or faster adversarial response to our new policies. Treating identity as a platform rather than a patchwork of one-off checks makes it possible to keep pace by offering a single place to fold in a new kind of signal instead of relying on a dozen separate systems each waiting their turn to catch up.

Ultimately, we aim to benefit from fewer bespoke, one-off identity answers by developing one consistent identity and relationship substrate to serve every decision point that needs it.

Stay Informed with Weekly Updates

Subscribe to our Engineering blog to get regular updates on all the coolest projects our team is working on

When a consumer asks Ask DoorDash to “reorder my usuals,” the request sounds simple. DoorDash already has their order history; why not retrieve the products they buy most often and put them into a shopping list?

Reordering is one of our most popular use cases, accounting for roughly one in every three orders. But the original agentic path could take tens of seconds to produce a list. By moving stable historical reasoning ahead of the request and using deterministic execution where the product already knows the consumer’s intent, we reduced one reorder path from roughly 16 seconds to roughly 2 seconds and doubled the rate at which consumers viewed the resulting list. 

Moving the historical reasoning offline also relaxed the latency constraint on generation, allowing us to use more capable models to produce higher-quality usuals. That improvement translated into a ~34% increase in order rate for the flow.

The challenge is that making reorder faster cannot mean simply replaying history. Order history records what someone bought, not what they need next.

Milk purchased once a week is probably a recurring need. A large pack of paper towels purchased at roughly the same frequency may not be. Six units bought in one delivery are different evidence from one unit bought across six deliveries. A yogurt a consumer buys repeatedly may be a staple; three flavors bought once each may be experimentation.

Answering “shop my usuals” therefore requires judgment over historical behavior. At the same time, the final answer has to be grounded in a catalog that changes continuously: items go out of stock, listings change, and the exact product a consumer bought previously may not be available at the store they are shopping today.

For a typical free-form reorder request, the Grocery agent loads the Reorder skill, retrieves the consumer context and order history needed for the request, decides which historical purchases belong in the next basket, resolves those needs against the current catalog, and assembles the shopping list. This flexibility lets the same flow handle a range of reorder requests while bringing together several different parts of the Ask DoorDash system to produce a seemingly simple shopping list.

Earlier posts in this series covered Intelligence, Evaluation, Platform, and User Experience one at a time. Reorder is the flow where all four have to agree at once: the list has to be personal, verifiable, fast, and shoppable. This post follows a single request through each of them.

Reorder is more than replay

Ask DoorDash supports a range of reorder requests. A consumer might type “What groceries do I usually get?”, “Reorder my usual groceries”, “Which milk did I buy last time?”, or “Reorder my usuals, but cheaper”.

These requests sound similar, but they require different decisions underneath. Looking up a product from the last order is different from deciding which purchases represent a consumer’s usuals; asking for cheaper versions adds another set of constraints.

Ask DoorDash’s multi-agent architecture, described in our engineering overview, first routes a grocery request to the Grocery agent. When the request involves reorder, the agent can load the Reorder skill.

The Reorder skill gives the agent task-specific instructions for handling these requests using the context and tools available to it. Rather than defining a separate workflow for every possible phrasing, it gives the agent a general process for deciding what the request requires and turning the result into an actionable shopping list.

Conceptually, that process has five stages:

  1. Establish context. Determine the store to shop at when one is not already known and retrieve relevant context about the consumer.
  2. Retrieve history. Read the grocery order history needed for the request and reshape it into a representation the model can reason over.
  3. Select what matters. Decide which parts of that history answer the consumer’s request. For “usuals,” this means distinguishing recurring needs from one-off purchases.
  4. Resolve products. Search the current DoorDash catalog, preserving details such as brand, size, and variant wherever possible.
  5. Build the result. Ask the consumer a clarifying question when a genuinely ambiguous choice remains, then create the editable shopping-list artifact.

Other skills can compose with that flow. If a consumer asks for their usuals “but cheaper,” for example, the Reorder skill can establish what they typically need while affordability reasoning considers the products and prices available now.

This flexibility matters for free-form language. A consumer typing into Ask has not selected an API operation; they have expressed an intent, and the agent has to determine what work that intent requires.

Every one of those five stages assumes something the skill does not contain. Which store to shop, which brand to prefer, whether this consumer even wants olive oil this week — none of that is a fact about reorder. It is a fact about the consumer.

Intelligence: What the agent already knows about you 

Order history tells us what a consumer bought, but it does not tell us how to interpret every purchase. As described in Building Ask DoorDash (Part 2): Intelligence, Ask DoorDash can bring in context such as store preferences, brand affinities, dietary preferences, and pantry state from long-term, conversational, and in-session memory.

For reorder, those signals are not another source of products. They help interpret the consumer’s purchase history. If someone consistently buys one milk brand, for example, that preference should carry into the resulting list. If they recently told Ask they are well stocked on olive oil, that context can keep olive oil out of their next reorder even if it appears frequently in their history.

That matters because consumers already know what their own “usuals” are. Small mismatches in store, brand, or pantry state are immediately visible and can quickly undermine trust in the rest of the list.

Intelligence therefore gives the Reorder skill a better starting point, but it does not answer the core question: which purchases in the consumer’s history actually count as recurring needs? That decision still has to be made — and evaluated consistently.

Evaluation: Testing whether the list is right

We needed a way to evaluate the “usuals” decision before it reached the consumer. As described in Building Ask DoorDash (Part 3): Evaluation, agent evaluation looks at both the execution path and the result. For reorder, that means checking whether the agent retrieved the right context and history, followed the expected flow, and produced an actionable shopping list — as well as whether the items on that list actually reflect the consumer’s recurring needs.

Those selection checks are concrete. If a consumer repeatedly orders three yogurts, does the list preserve that pattern and quantity? Does an item that appeared only once stay out of their “usuals”? If the consumer has said they already have something at home, is that reflected in the result?

Because live order history changes over time, we use the fixture-based Conversation Simulator to replay reorder scenarios against a fixed purchase history. Checklist-style rubrics then let us test recurring items, one-off purchases, quantities, and known preferences against the same evidence as the implementation changes.

A correct list is only useful if it arrives while the consumer is still shopping. Our original flow took about 16 seconds.

Platform: The decision doesn’t need to happen while you wait 

That latency came partly from the way Ask DoorDash handles agentic requests. As described in Building Ask DoorDash (Part 4): A Platform for Building and Evolving Agents, within a single consumer turn, the model can make a decision, call a tool, inspect the result, and use that information to decide what to do next. In a representative Grocery flow, that can require 6–8 LLM calls along with several tool calls.

That flexibility matters when each step depends on information discovered during the request. However, reorder showed us that several of those steps relied on historical data that changes slowly.

Consumers often return to a relatively small set of grocery stores. Instead of selecting a store, fetching its order history, and determining the consumer’s recurring needs on every request, we can do that work ahead of time using historical data.

Other parts of the flow have to stay live. Inventory and prices change, products become unavailable, and catalog listings can change between visits.

That gave us a natural boundary: precompute the consumer’s likely stores and recurring needs, then use the live request path to resolve those needs against the current catalog and build the shopping list.

That split became the basis for our offline-to-online pipeline.

Figure 1: Ask DoorDash assigns each reorder request to an execution path based on where its judgment belongs. For “reorder my last cart,” a typed session scope allows a backend plugin to retrieve the previous cart without using the model for tool selection. For recurring purchases, a daily pipeline combines 90 days of order history with task-relevant profile signals, selects consumers with sufficient evidence, runs batch LLM inference, and validates the output into a reusable usuals bundle. “Reorder my usuals” reads that bundle directly, while “reorder my usuals, but cheaper” supplements it with current prices, promotions, package sizes, inventory, and consumer constraints. The three paths converge on live catalog resolution, which attempts the store item ID first, then the catalog product ID, and finally a search term before creating an editable shopping list. The Grocery agent remains available to clarify ambiguous choices, narrate substitutions, and recover from missing or unusable results.

Generating usuals before the consumer asks. 

Cohort selection and input materialization

Not every order history contains enough evidence to infer recurring needs. A single grocery delivery tells us what a consumer bought once, not whether any of those items are staples. Running generation across every consumer would therefore add cost while producing low-confidence results.

We treat cohort selection as both a quality gate and a cost-control mechanism. A daily Spark job identifies consumers with enough recent grocery or retail activity to support generation.

For each selected consumer, we prepare two inputs: a compact profile containing the long-term signals most relevant to reorder, such as item-category interests and brand affinities, and item-level order history from the previous 90 days.

The two serve different purposes. The profile provides context about the consumer; the order history provides the event-level evidence needed to infer recurring needs. A brand preference may tell us which yogurt a consumer is likely to want, for example, while purchase history tells us whether yogurt belongs in their usuals at all.

Following the task-specific context trimming approach from our work on offline LLMs for online personalization, we send only the profile signals relevant to this task rather than the consumer’s full profile. That reduces input cost and keeps unrelated context from competing with the purchase evidence the decision depends on.

Defining “Your usuals”

Producing a ranked list is straightforward once “usual” is defined. Defining it was the harder part.

Frequency is a reasonable baseline, but it answers what a consumer bought most often rather than what they are likely to need again. Milk and a large pack of paper towels can appear in the same number of orders while having very different replenishment patterns. Six bottles in one delivery are also weaker evidence of a recurring habit than one bottle purchased across six separate deliveries. Recency alone does not resolve the ambiguity either.

We use an LLM because these signals interact in ways that are difficult to capture in a single ranking rule. The model follows an explicit qualitative ordering: frequency across distinct deliveries is the primary signal of a recurring need, recency indicates whether that need is still active, and replenishment characteristics help distinguish quickly consumed products from durable ones. Brand affinity can refine the choice, but it does not outweigh direct purchase evidence.

Stock-on-hand adds another layer. A recently purchased multipack of a durable product may move down the list, while a perishable that appeared regularly in the past but has not been purchased recently may move up.

Turning the decision into a bundle

The model applies that reasoning independently for each relevant store and returns the recurring items it identifies. The maximum number of items is a ceiling, not a target: if the evidence supports only a handful of usuals, the model returns the smaller set rather than filling the list with weaker guesses. If the history does not support a useful bundle, it can abstain entirely.

The output is constrained to evidence from the input. Store and item identifiers must come from the consumer’s history, repeat purchases are preferred over one-offs, and near-duplicate variants are collapsed rather than filling multiple positions in the bundle. For each item, we retain signals such as purchase frequency, recency, and typical quantity so the result can be served without redoing the historical reasoning online.

Represent the need, not the SKU

A well-ranked item is still unusable if the catalog changes before the consumer comes back. The exact listing may be relisted under a new merchant-specific identifier or disappear entirely. Persisting only the historical store item ID would make the bundle brittle; persisting only a generic term such as “oat milk” would lose the specificity of what the consumer actually buys.

We therefore represent each need at three levels: the store item ID for the most precise match, a catalog-level product identifier for the same underlying product across listings, and a generic search term as a fallback. At request time, the serving path tries them from most specific to most general and stops at the first eligible current product.

That hierarchy captures the offline-online split cleanly. The offline model decides which need from the consumer’s history is worth carrying forward; the online system decides which product can satisfy that need right now.

Before a bundle is published, deterministic validation sits between model generation and the serving contract. It distinguishes an explicit abstention from malformed output, removes unusable identifiers, normalizes quantities, and filters results that do not meet the requirements for a useful bundle. The model makes the semantic decision; deterministic code decides whether the result is safe to serve.

Generating and serving at scale

Generating these bundles across a large consumer cohort requires a batch system that can tolerate partial failures. We use a sharded Metaflow infrastructure so batches can run independently and failed shards can be retried without discarding successful work.

After validation, the resulting bundles are published to a low-latency key-value store. The serving path can then fetch a consumer’s bundle for a given store without repeating the historical reasoning, while missing or unusable bundles degrade to an empty response rather than failing the Grocery turn.

Because the bundle is generated independently of Ask, the same signal can also support other personalized surfaces without repeating the inference.

None of this is visible to the consumer. What they see is a shopping list — and depending on how they entered the flow, they may see it in about two seconds.

User Experience: One list, several ways in

A consumer does not always arrive at reorder through the same path. They may type “reorder my usuals” directly into Ask, or they may enter through a contextual surface where DoorDash already knows something about what they are trying to do.

That distinction matters because context supplied by the experience can remove work from the conversation. As described in Building Ask DoorDash (Part 5): A Grounded Interface for Shopping Agents, the client attaches scope to a request, carrying information such as the topic and store the consumer entered from. The system can use that context instead of asking the consumer to restate information the product already knows.

For a free-form request like “reorder my usuals” the Grocery agent still interprets the request through the Reorder skill. But when a consumer enters through a dedicated “Reorder my usuals” action, the product has already established the intent. As shown below, we attach a preset scope that is invisible to the consumer and route the request through a deterministic path.

Combined with the precomputed bundle from the Platform section, that reduces the live work to retrieving the bundle and resolving its items against the current catalog. The path no longer needs the LLM to interpret the request or reconstruct the consumer’s usuals from history, bringing latency from roughly 16 seconds to roughly 2 seconds.

Whether the consumer arrives through free-form text or a contextual entry point, the result is the same grounded shopping-list artifact. From there, they can change quantities, remove something they already have, swap a product, or add the finished list to their cart. Those structured edits happen directly against the artifact rather than requiring another LLM round trip.

The agent remains available when the request requires more judgment. A consumer can ask for “my usuals, but cheaper” for example, and the agent can start from the same recurring needs while reasoning over current prices and alternatives.

Different entry points can trigger very different amounts of work underneath while still converging on one continuous reorder experience.

Conclusion

“Reorder my usuals” looks like a simple request, but making it useful requires putting different decisions in the right place. Consumer context helps interpret purchase history; recurring needs can be computed ahead of time; inventory, prices, and catalog resolution stay live; and contextual entry points can remove work the product has already done.

The broader lesson is that making an agent better does not always mean asking the model to do more. It means deciding what should be reasoned about at request time, what can be known ahead of time, and what should be deterministic.

When those boundaries are right, the complexity disappears for the consumer. They ask DoorDash to reorder their usuals and get a list that feels personal, current, and ready to shop.

Stay Informed with Weekly Updates

Subscribe to our Engineering blog to get regular updates on all the coolest projects our team is working on

How a reusable benchmark fleet made Triton configuration measurable across hundreds of production and shadow models, with mean QPS lift rising from 1.75x at 4 vCPU to 6.78x at 32 vCPU.

Part 1 of a multi-part series. This installment covers CPU-backed Triton models. A follow-up post will cover how we are extending the system to GPU inference.

On one production model, the same artifact running on the same 32-vCPU pod delivered either 68 QPS or 863 QPS depending only on its serving configuration. After tuning, p99 latency fell from 126 ms to 14 ms and CPU utilization fell from 67% to 16%.

1.75-6.78xmean QPS lift from instance scaling, by CPU tierHundredsproduction and shadow models supported20.1% vs 0%severe regression rate, one instance vs one per vCPU252reusable benchmark pod slots

When More CPU Made Models Slower

At DoorDash, PyTorch models run on NVIDIA Triton Inference Server across use cases such as search, ranking, dispatch, fraud, and ads. Each model has several serving choices: CPU architecture, pod size, Triton instance count, OpenMP thread count, and request concurrency. A reasonable default for one model can waste capacity or violate latency targets for another. 

The refreshed analysis included 264 models with usable sweep data. Among models with comparable single-instance results, 44 of 219 fell to half or less of their 4-vCPU throughput at the largest swept tier. Among 206 models with comparable one-instance-per-vCPU results, none did. 

The mistake was treating pod size as the main performance decision. In practice, pod size interacts with the number of Triton model instances and the number of threads used by each instance. More cores can create more contention when those settings are not coordinated. 

Which complete serving configuration delivers the required throughput and latency at the lowest resource cost?

Answering that question for one model is a benchmark. Answering it consistently across hundreds of models requires a system. 

A Reusable Tuning System

We built an auto-tuning platform that benchmarks production model artifacts and representative inputs against a controlled configuration matrix, then selects the lowest-cost configuration that meets the model’s latency and capacity requirements. 

  1. Orchestration
      An AutoTuneFlow resolves the model artifact, representative inputs, production traffic, latency target, and candidate configuration matrix. 
  2. Measurement
      A shared benchmark pool runs the candidates on the same Triton runtime used in production and records throughput, latency, and cgroup CPU utilization. 
  3. Selection
      The flow rejects candidates that miss latency or CPU limits, combines the remaining capacity measurements with production QPS, and emits deployment overrides. 

Model teams do not need to understand the mechanics of the benchmark pool, and the measurement layer does not need to know the business context behind a model’s latency target. The orchestration layer joins those concerns into a production-ready recommendation. 

The benchmark data plane contains 252 pre-provisioned pod slots across 21 deployment groups on amd64 and arm64.

A Fixed Pool Instead of Per-Model Infrastructure

Each deployment group contains 12 dedicated slots covering four CPU tiers (4, 8, 16, and 32 vCPU) and three Triton instance counts for each tier. A sweep acquires one available group and runs the tier and instance-count combinations in parallel. Different models can use different deployment groups at the same time. 

An S3-backed lock coordinates access. If a group is busy or unhealthy, the orchestrator moves to another available group. This gives us two kinds of parallelism: candidates run concurrently within a sweep, and multiple models can be tuned concurrently across the shared fleet. 

The fixed 252-slot pool supports tuning across hundreds of production and shadow models. Growing the number of models increases utilization of the platform, not the infrastructure required for each individual run. 

Benchmark with NVIDIA’s perf_analyzer

Each benchmark pod invokes NVIDIA’s perf_analyzer through a small benchmark model running alongside the target model. perf_analyzer sends asynchronous gRPC requests over localhost at each requested concurrency level, while the runner records throughput and latency and reads the pod’s cgroup CPU counters. This produces one consistent measurement of QPS, p99 latency, and the same CPU signal that Kubernetes HPA sees. 

The in-pod runner combines traffic generation, model measurements, and cgroup CPU utilization into one result for each candidate configuration.

Reconfigure Triton Without Restarting Pods

The benchmark pods run Triton in explicit model-control mode. For each candidate, the tuner loads the target model through Triton’s model-control API and passes the candidate ModelConfig as an override. It can then unload the model and reuse the pod for the next sweep. 

That lifecycle avoids an S3 edit, repository polling delay, or pod restart for every candidate. A single measurement follows the same sequence: 

  1. Acquire a benchmark group.
  2. Load the benchmark runner and target model.
  3. Validate the inputs and warm the model.
  4. Run perf_analyzer across concurrency levels.
  5. Record QPS, latency, and CPU measurements.
  6. Unload the model and release the group.
Each tier and instance-count slot follows this lifecycle in parallel. The results return to the same flow for configuration and cost selection.

The exact number of measurements depends on the CPU architectures, tiers, instance counts, batch sizes, and concurrency levels requested. Because the expensive tier and instance-count combinations run in parallel, expanding the search matrix does not translate directly into wall-clock time. 

Why Configuration Changed Performance So Much

PyTorch CPU inference has two forms of parallelism. Triton can run several copies of a model in parallel, while each copy can use several OpenMP threads within an inference. These controls are useful individually, but they must be tuned together. 

The out-of-the-box pattern on an N-core pod is often one Triton instance using N threads. For many of our models, a better pattern was N single-threaded instances: 

instance_group.count = vCPU
OMP_NUM_THREADS = 1
OMP_DYNAMIC = FALSE
INTRA_OP_THREAD_COUNT = 1

This turns a single highly threaded inference worker into multiple independent workers. It also avoids the worst case, where multiple Triton instances each create a full-size thread pool and compete for the same cores. 

The tuner therefore treats instance count and thread count as one configuration decision. In general, it keeps: 

instance_group.count x OMP_NUM_THREADS ~= vCPU

The choice between more instances and more threads is still model-dependent, which is why we measure it rather than hard-code one fleet-wide answer. 

ConfigurationQPSp99 latencyCPU
1 instance, 32 threads68126 ms67%
32 instances, 1 thread each86314 ms16%

Across the refreshed analysis, comparing one Triton instance per vCPU with a single instance at the same tier produced mean raw QPS lifts of 1.75x at 4 vCPU, 2.31x at 8 vCPU, 3.65x at 16 vCPU, and 6.78x at 32 vCPU. The model artifact and hardware tier stayed the same; only the serving configuration changed. 

The measured configuration benefit increased with pod size. Severe throughput regressions appeared in 44 of 219 comparable single-instance results and zero of 206 one-instance-per-vCPU results.

PyTorch’s CPU threading documentation describes the interaction between inter-op and intra-op parallelism in more detail. 

What We Learned Across the Fleet

Let the Sweep Choose the CPU Tier

The refreshed 264-model analysis selected 4 vCPU most often at 71.8%, followed by 8 vCPU at 14.9%, 16 vCPU at 10.2%, and 32 vCPU at 3.1%. Larger tiers won for 28.2% of models. 

Every available CPU tier was optimal for at least some models.

Four vCPU is a useful starting candidate, not a production default. The tuner evaluates the available tiers and selects the one that delivers the best cost-adjusted capacity while meeting the model’s latency target. 

CPU Alone Does Not Always Describe Saturation

Kubernetes HPA commonly estimates capacity by scaling observed throughput in proportion to CPU utilization: 

predicted QPS = current QPS x (target CPU / measured CPU)

That works when CPU is the limiting resource. It fails when throughput plateaus first. In one 8-vCPU test, CPU leveled off at 26%, so a 40% HPA target implied 1,252 QPS of capacity. The model saturated at 829 QPS, a 1.5x overestimate. 

Triton exposes queue signals such as nv_inference_pending_request_count and queue duration, which describe saturation more directly. We are moving toward an autoscaling signal that combines CPU with queue pressure. 

From Benchmark Result to Production Configuration

The tuner does more than rank benchmark rows. It combines measured per-pod capacity with the model’s production QPS range, calculates replica requirements and projected cost, validates the proposed change, and produces the deployment overrides consumed by the delivery workflow. 

  • CPU architecture and vCPU tier
  • Memory allocation
  • Triton instance count
  • OpenMP thread settings
  • Expected QPS and p99 latency
  • Minimum and maximum replicas
  • Serving image
  • Projected cost

This closes the gap between a benchmark and an operable production configuration. It also makes the recommendation reproducible: the measurements, selected configuration, and cost inputs are stored together instead of living in an engineer’s notebook. 

Supporting Platform Migrations and Continuous Training

The tuner now supports two recurring platform workflows. During platform migrations, it gives each model a measured configuration for the target serving environment instead of carrying forward assumptions from the previous platform. The same integration lets continuous-training workflows re-evaluate new model artifacts as they are produced, so serving settings can evolve with the model. 

Both workflows use the shared benchmark pool and the same selection logic. This gives migration and retraining a consistent performance-validation path without requiring model teams to build separate benchmarking infrastructure. 

The Broader Lesson

Serving configuration is part of model performance. Retraining and code optimization matter, but neither is required to recover capacity lost to runtime defaults. 

At fleet scale, the important capability is not a single rule such as “use 4-vCPU pods” or “set OpenMP to one.” It is a repeatable system that can test those assumptions against every model, under consistent conditions, and turn the result into a deployable configuration. 

For our fleet, that system found mean QPS lifts ranging from 1.75x to 6.78x across CPU tiers, showed why CPU tier must be measured per model, and replaced manual sizing with a shared tuning path that scales across hundreds of models. The benchmark pool stayed fixed as the number of models using it grew. 

What’s Next

This first post focused on CPU-backed Triton models. In the next installment, we will cover how we are extending the same auto-tuning workflow to GPU-backed models, including the GPU configuration space, accelerator-specific capacity signals, and the changes required in the measurement layer. The orchestration and selection layers remain reusable, giving GPU models the same repeatable path from model artifact to validated production configuration. 

Acknowledgments

  • ML Serving Platform Arvind Kalyan, Kyoungah Kim, Haifeng Geng, Emmanuel Khayat, Jeffery Ni, Songze Li, Teer Ying, Feijin Deng, Zhe Liu, Zhengyang Wang
  • Ads ML Team Utsaw Kumar, Kin Sum Liu, Benjamin Yang, Praveen Gondhi

Stay Informed with Weekly Updates

Subscribe to our Engineering blog to get regular updates on all the coolest projects our team is working on

The DoorDash Data team is responsible for generating accurate data insights and answers to key business questions: 

  • Why did the order volume dip in California last week? 
  • How has a new product been pacing since launch? 
  • Which merchants are at risk of churning from our marketing products?
  • Show me canceled orders by hour for the last 90 days at Arby’s

Frontier agentic models combined with agent harnesses such as Codex and Claude Code (CC) are powerful tools for interacting with our data and answering such questions. However, they often perform suboptimally when facing the large semi-structured data footprint common in large-scale companies. It is a challenge to ensure that all business functions — including operations, engineering, marketing, and product — have access to current, correct data from which they may draw the right conclusions.

“The problem is actually that our upstream data sources are super-fragmented,” noted a senior analytics manager looking at our previous tools. Those tools were either deep in one domain or broad and shallow. 

As a result, we built Vera, an internal conversational data agent. We found that building and tuning all parts of the pipeline — prompting, ranking and retrieval, and domain-specific skills — led to significant performance improvements over standard agents with simple data connectors. “It has worked well for strategic finance questions across difficulty levels. I’ve been very impressed,” said a finance director after several weeks of testing Vera.

Why we built in-house

Contemporary agents generally can answer data questions in a clean environment. But a strong semantic data layer is key to ensuring these agents pull truthful answers that the business can trust. Disambiguating and deconflicting semantic meaning across the organization is not trivial; it’s a pressing challenge to grow this data layer while maintaining accuracy. 

The key to scaling Vera, in our view, is to let domain owners — the people closest to the data — contribute and vet evals. A domain owner is the subject-matter expert accountable for how a business area is measured. Building that tight loop is easier when we have firm control over the stack. 

In this post, we step through Vera’s overall design and performance measurements, discussing architectural choices and providing comparative studies across model types and reasoning efforts.

Table 1: One full benchmark run, every model in the same harness. The frontier models land within a few points of each other on score, so latency, steps, and tokens are where they separate.

Our evaluation dataset spans 19 vertical domains across sales, finance, product, and operational domains at DoorDash with more than 1,900 contributed evaluations. These vary in complexity, with some questions requiring many tool calls and extended reasoning. 

The current production model – GPT 5.5 – scores highest overall on the benchmark. This is unsurprising as we have optimized prompts and skills for this model specifically, which we aim to replicate across model families soon.  

The final responses are scored with an LLM judge across three criteria: tool usage, table selection, and overall answer correctness. Where evals do not have an unambiguous SQL-backed query, domain experts provide detailed judging rubrics. The maximum score in this design is 3.0; as Table 1 shows, the frontier models from both Anthropic and OpenAI land within a few points of each other.

We also report tool calls and completion rates. Because the models author their own SQL, a full run fires many concurrent heavy queries. Some hit database timeouts which count as failures in the pass rate, not as wrong answers.  

The run-to-run spread of the benchmark is roughly four percentage points of pass rate, independent of the model; differences smaller than that are considered noise. One contributor keeps the evaluations themselves current. An eval that specifies a relative time window — for instance, “What was the order throughput of this store last week?” — goes stale quickly.

While frontier models perform well with our harness on this benchmark, there are different trade-off considerations for latency, tool-use, and cost depending on the model and provider. We choose Vera’s production model on that trade-off rather than on the headline score alone, re-running the benchmark as new models ship. These ablations run on a 100-question subset, so differences of a few points are within our definition of noise. Across model types, more reasoning effort improves the score.

Figure 1: Accuracy against tokens per question on the 100-question subset. Every line rises with effort and they converge toward the top, so effort matters alongside model choice and is paid for in tokens.

The performance gap at the frontier is small, with both OpenAI and Anthropic models achieving similar scores with extended effort. Naturally, increasing the reasoning budget leads to an increase in latency.

Figure 2: Median and p90 answer time for the same models and effort levels. Higher effort buys accuracy with wall-clock time.

While more reasoning effort monotonically increases the benchmark score, the effect can be deleterious to the system as a whole in the form of longer responses to the user and more frequent tool calls to the data store that increase the likelihood of timeouts and partial results. We observed the following timing distributions across models, as shown in Figure 3:

Figure 3: Answer time distribution per model on the full run. The tail, not the median, drives timeouts and partial results.

In our harness, the tail latency of the Anthropic models was longer than that of their OpenAI counterparts. We hosted an internal instance of GLM 5.3 Flash via Modal endpoints as a coarse comparison, although its unoptimized nature meant significant timeouts under heavy load. 

Claude Fable 5 needed fewer tool steps than the GPT models, yet spent longer per step and roughly doubled the runtime. This is evident when comparing the tool trajectories for the various models, as shown in Figure 4:

Figure 4: Average tool calls and agent steps per question. Fewer steps does not mean a faster answer, since the models that take fewer steps spend longer on each one, as Figure 3 shows.
Although historically we have seen stepwise improvements on benchmark scores for new model variants, the impact of releases has progressively plateaued, as shown in Figure 5:
Figure 5: Pass rate by model from March to September, with the changes that moved it.

This is a strong indicator that raw model intelligence is no longer the biggest contributing factor to Vera’s improvements, and that the model harness — adding business context, knowledge retrieval, data modeling, SQL validation, and execution — is the dominant vector for improvement going forward. 

To validate this hypothesis, we tested both Claude Code and Vera with a common model, Opus 4.8. We gave CC our SQL access tools and the same skills that Vera uses.

We ran 100 representative samples through each system. Vera scored 2.60/3.0, a 73% pass rate, while CC scored 2.16/3.0, a 48% pass rate. This gap fell well outside the estimated noise spread. We found that CC struggled to query modeled tables effectively to draw correct inferences, which emphasized the need for an effective data layer, including a catalog of modeled tables, join pattern heuristics and the like. Coming up: How we build and query that layer. 

Retrieval and ranking

More than 10,000 people at DoorDash rely on a data ecosystem that has grown to include more than 200,000 datasets, 350 petabytes of data, and 10 institutional knowledge sources, from code repositories to wikis. With that much data available, the challenge can be knowing what to use and when. The retrieval and ranking step provides those current, accurate, deconflicted data sources. 

Figure 6: Vera’s path from a question to a grounded answer. Retrieval runs two searches side by side, over human-authored knowledge and over the modeled tables, and everything downstream is constrained by that model.

For simplicity, a single embedding model is used across all knowledge stores. Core data tables are indexed offline on a regular cadence, which lets us filter out low-signal queries and strip boilerplate. We learn join patterns and other semantics from the remaining usage patterns, table metadata, code references, and wiki pages to build a rich data model. 

The data model constrains runtime behavior; access to unmodeled or unverified sources is blocked at the harness level. One downside is the lag between when data becomes available and when Vera can query it. New tables are discovered and indexed both agentically at inference time and periodically in the background. The accuracy tradeoff is worthwhile because the vast majority of core business tables are long-lived and already indexed. 

We also need to consider how to manage the addition of new information. How do we decide when new join patterns or document embeddings are additive for user queries? By default, these sources are marked inactive and are subject to passing an evaluation threshold before a domain owner promotes them. Domain owners can also turn on fully automated promotion for their area. 

Additionally, every knowledge source and skill encodes its provenance back to the source material; a change-detection pass regularly polls for source diffs. Material changes result in a change proposal that flows through an approval queue and is subject to the same gating criteria as the evals.

Evals and grading

More than 1,900 evaluations have been contributed by both domain owners and everyday users; a domain owner vets every user contribution before it counts. Each eval is graded by an LLM judge on three questions: 

  1. Did the agent use the right tools?
  2. Did the approach use the correct tables?
  3. Was the final answer correct?

We report the weighted aggregate of these answers as the overall score. The judging model was iteratively calibrated at several milestones. The most notable deviation was a tendency to over-penalize correct answers when tables were equivalent but not exact. The recalibrations ensured the judge was robust to numerical imprecisions, accepted equivalent data sources, and ignored minor formatting differences. 

The core operational question: How do we source evals across the company that are non-trivial, non-duplicative, and representative of real analytical work without burdening domain owners? 

We built tooling into Vera that supplements manual evals by mining everyday work outputs such as Slack threads and Jira tickets. A frontier model reads each work product and drafts a full eval proposal, including the question, a SQL query or rubric, and a complexity rating. 

Domain owners then review each proposal and either add it to or drop it from the evaluation set. We have found this to be a better use of expert time than having them write evals from scratch. This has resulted in significantly more evals providing broader coverage, as shown in Figure 7: 

Figure 7: Growth of the evaluation set, with the three additions that shaped it. The two jumps are bulk imports reviewed by domain owners, and the steady climb after July is the mining and promotion pipeline.

Ensuring quality at this scale becomes more challenging. We automatically deduplicate near-duplicate questions with a similarity measure; domain owners accept 95% of the remaining candidates during review. 

Addressing complexity

Although Vera’s focus is answering conversational questions, the world knowledge and post-trained behaviors of the underlying models allow them to address significantly harder questions. We use a coarse taxonomy of problem difficulty:

  • Tier 1: Descriptive questions with known definitions and curated patterns, including metric pulls, trends, splits, and sanity checks
  • Tier 2: Analyses that could influence business or product decisions, including prioritization, targeting, opportunity sizing, and resource allocation
  • Tier 3: Questions requiring deeper interpretation and accountability, including causal analysis, experiment readouts, committed forecasts, and high-stakes decision support

Because the overlap between the tiers is high, it can be challenging to classify queries into discrete buckets, but it is illustrative when considering model effort expenditure for the more complex queries, as shown in Figure 8: 

Figure 8: Answer time against the data each run pulled back from tools. Tier 2 and 3 questions cluster toward larger payloads and longer runs, so harder questions cost more in data and time, not just reasoning.

For the more complex queries in Tier 3, we noticed that models tended to stop early and optimistically. There was relevant information to answer the query, but the model often returned superficial — albeit plausible — answers without diving deeper. 

For these questions, we built an /analysis flow, which prompted the model to do problem discovery and planning before execution as shown in Figure 9: 

Figure 9: Vera’s /analysis flow on a merchant trend question. The plan is shown before any query runs, and nothing executes until the user approves or adjusts it.

This is where the agent resolves timeframes, desired metrics, relevant scope, and any constraints the user wants to enforce. We’ve found that a structured planning step improves the performance of these Tier 3 evals, but with a large variance — between 64% and 100% pass rate — across a small sample of 14, showing the need for broader and deeper expert-driven feedback in these areas. 

Going forward, we will address as a core focus these harder, more subjective questions in Tier 3. 

Takeaways

Over the course of the project, the pass rate moved from 43% to 90% while the evaluation set size doubled twice. Improvements in the intelligence of the underlying models helped drive performance, but it was data engineering around the harness that proved to be most impactful. 

There remain multiple open questions:

  1. How do we continue to maintain quality while reducing cost per answer?
  2. How do we build evals around more complex data analytics questions and reporting?
  3. How do faster, better analytics move core business metrics?

As we build data analytics intelligence at DoorDash, we will take up these open questions in later posts in this series.

Contributors: Pavel Astakhov, Andy Fang, Jash Radia, Naveen Srinivasan, Helen Lin, Lokesh Sharma, Adnan Sheikh, Gun Johnson, Harsha Venkat Annapa Reddy

At DoorDash, we have many Engineering teams that support our Engineering domains. This document provides an overview of our domain areas and teams that support them:

  1. Dasher & Logistics
  2. Foundations
  3. LaunchPad & AI Research
  4. Marketplace
  5. Merchant Consumer Platform
  6. New Verticals
  7. Operational Excellence
  8. Platform Services
  9. DashOS
  10. Money
  11. Ads & Promotions

It is our hope that these profiles can serve as helpful references for you to respond to our team matching survey. 

Since our teams are ever-evolving, this document is a snapshot of our current Engineering organization. Please keep in mind that it’s likely to change between now and your start date. Therefore, as we make team matching decisions, we will focus on the type of work you are interested in. In addition to your preferences and technical background, we will take team capacity and other organizational factors into consideration.

Note: Abbreviations for our primary customer pillars are sprinkled throughout the document – Merchant (Mx), Consumer (Cx), Dasher (Dx) – in addition to Restaurants (Rx).

Dasher & Logistics

The Dasher & Logistics (DxLx) team powers the heart of DoorDash’s delivery network – driving the execution of every order with intelligence, precision, and scale. From the logistics “brain” that routes Dashers and balances supply and demand, to the platforms and products that create a cohesive and rewarding Dasher experience, this team ensures every delivery happens efficiently and reliably.

Our vision is to build the highest-quality, lowest-cost fulfillment network, making it the first choice for gig workers and merchants around the world. DxLx is built around two primary pillars:

  • Logistics: Builds a world-class, global last-mile logistics platform that operates at the frontier of engineering, science, and infrastructure. The team continuously pushes the boundaries of speed, cost, efficiency, and reliability—powering millions of deliveries every day.
  • Dasher: Focused on empowering Dashers with choice, stability, and opportunity. This team is creating the best gig platform in the world, driving retention, growth, and satisfaction through innovative employment models, rewards, and engagement programs.

DxLx sits at the core of the DoorDash experience – transforming how goods move through cities and making life more convenient for millions across the globe.

Foundations

The Foundations team builds the core infrastructure, data, and developer platforms that power every product and service at DoorDash. As the engineering backbone of the company, they combine infrastructure and data to create a seamless, secure, and scalable foundation on top of our AWS Cloud ecosystem. The organization spans three key pillars:

  • Developer Platform: Empowers engineers to move fast and build with confidence through a first-class developer experience built on modern, reliable, and high-performance platforms.
  • Core Infrastructure: Provides secure, efficient, and resilient systems that enable engineers to deliver great products at speed and scale.
  • Foundations Data: Brings together data engineering and data platform work to ensure the business runs on trusted, high-quality data and to accelerate data-driven product innovation across DoorDash.

In September 2025, the Foundations teams across DoorDash and Wolt unified to scale Pedregal and build the future together, guided by shared principles of keeping the business running, single-threaded leadership, geographic ownership, respecting time zones, and aligning the right talent to the right problems.

Foundations is the platform the entire business runs on, and there’s still plenty of room to shape how it scales as DoorDash grows – driving reliability, performance, and speed across everything we do.

LAUNCHPAD & AI RESEARCH

LaunchPad’s mission is to innovate, experiment, and incubate bold ideas. The team is built to move fast, quickly turning promising ideas into real prototypes so DoorDash can find out what’s worth investing in further. Alongside LaunchPad, DoorDash AI Research is an applied research team exploring how the newest AI capabilities can change the way DoorDash’s business works.

The org covers two main areas:

  • 0-to-1 Projects: LaunchPad takes on new, high-potential ideas across the DoorDash product ecosystem, the kind of projects that could become a major new business or unlock a meaningful new capability for the platform. 
  • AI Research: Explores how the newest AI capabilities can improve DoorDash, from making engineers more productive, to building AI agents that can act on behalf of customers, merchants, and Dashers, to helping the company better understand and use its own data.

Because LaunchPad and AI Research are both about testing what’s next before it becomes an established part of the business, joining this org means working on some of the earliest, most exploratory ideas at DoorDash.

LaunchPad is led by DoorDash co-founder Andy Fang.

Marketplace

Marketplace Overview  

The Marketplace organization powers the exchange of goods between buyers and sellers across multiple verticals—delivering trust, quality, and seamless experiences at massive scale. Their mission is to build extensible, foundational components that enable teams to ship innovative features with confidence and velocity. Marketplace is organized around three core pillars, each driving the consumer experience in unique and impactful ways:

  • Core Consumer: Owns the end-to-end order journey—creating a fast, intuitive, and delightful shopping experience for every customer, everywhere.
  • Consumer Growth: Accelerates DoorDash’s expansion through data-driven insights, experimentation, and continuous optimization—from login to checkout and beyond.
  • DashPass: Oversees our subscription business, focused on growth, engagement, and retention. The Engagement team drives onboarding and early adoption, helping customers experience DashPass value from day one, while the Retention team ensures members stay connected and continue to see value throughout their journey.

The Marketplace team builds the foundation of the DoorDash experience—connecting millions of customers with the local businesses they love, faster and more reliably than ever before.

MERCHANT CONSUMER PLATFORM

The MxCx Platform team builds world-class platforms that accelerate product development and ensure exceptional reliability, observability, and security. Their mission is to empower our customers while scaling DoorDash’s growth across every experience.

The team focuses on both Merchant (Mx) and Consumer (Cx) experiences, powering everything from merchant onboarding to high-performance customer apps. Key areas include:

Merchant Acquisition: Identifies and onboards new restaurant partners using smart data and seamless self-serve tools.

  • Mx Tablet: The hardware and app powering in-store order management for merchants.
  • Business Manager: A mobile app that lets business owners operate and grow their businesses anytime, anywhere.
  • Order Platform: The foundation for managing DoorDash’s order lifecycle, enabling engineers to build new features with speed and reliability.
  • MxCx Client Platform: Empowers Consumer client engineers to safely and efficiently build performant, reliable apps across web, iOS, and Android – driving quality and consistency at scale.

The MxCx Platform team is shaping the backbone of the DoorDash ecosystem – creating the tools, systems, and client platforms that power growth for millions of users and businesses every day.

NEW VERTICALS

The New Verticals (NV) team is expanding DoorDash beyond restaurants – building the platform for local commerce and enabling customers to access anything in their city. From groceries and convenience items to pet supplies, alcohol, home goods, and beauty products, NV is redefining how people shop locally.

Our mission is to make DoorDash the go-to partner for everyday essentials, helping families and individuals get what they need, when they need it. The team tackles complex challenges unique to retail logistics, scaling systems to manage massive, dynamic inventories, improving data accuracy, and optimizing the Dasher experience for diverse fulfillment models like grocery shopping and item-level delivery.

Within NV, there are five main pillars (NV Categories, NV Consumer, NV Fulfillment, NV Merchant + Selection, and NV Machine Learning) with many subteams driving innovation. Additionally, the DashMart team is responsible forDash’s own virtual grocery and convenience store, which is fully owned, operated, and curated to deliver the next generation of “everything on-demand” shopping.

Engineers on the NV team will never be bored to find a challenge, and have incredible opportunities to make a mark on local commerce!

OPERATIONAL EXCELLENCE 

The mission of the Operational Excellence (OpEx) team is to build trust and drive efficiency through innovation and automation. If Consumers, Merchants, or Dashers can’t trust that the marketplace is fair and reliable, we don’t have a business. OpEx is made up of four key areas:

  • Delivery Experience: Ensures smooth, transparent order tracking and support once an order is placed.
  • Fraud: Keeps the marketplace fair by preventing and detecting fraudulent activity.
  • Trust & Safety: Builds confidence across Dashers, Merchants, and Consumers by verifying identity and maintaining compliance.
  • Risk & Identity Platform: Scales these efforts efficiently through platforms for risk, data, accounts, and ML/AI solutions.

As an engineer on the OpEx Engineering team, you’ll power the systems that keep our platform efficient, fair, and safe at massive scale. Whether fighting fraud in real time, ensuring seamless payments, improving delivery experiences, or strengthening trust and safety, your work will impact millions. You’ll ship meaningful code, collaborate cross-functionally, and grow quickly in a fast-paced, high-ownership environment.

PLATFORM SERVICES

The Platform Services team is building the operating system for every restaurant, grocery store, and retailer – empowering local businesses to thrive in the convenience economy. Their mission is to create the software and hardware that help merchants power and scale their own first-party businesses. Engineers on the team work across five high-impact areas:

Digital Ordering: A white-label solution that leverages DoorDash’s technology and scale to help merchants grow their direct online sales.

  • Omni-Channel Loyalty: A merchant-branded loyalty program that lets customers earn and redeem rewards seamlessly – online, in-store, and on DoorDash.
  • Voice Ordering: A high-quality voice service that enables in-store employees to focus on customers while we handle the calls.
  • Drive: Fulfillment-as-a-service that lets merchants tap into the Dasher fleet to deliver orders placed directly with them.
  • DoorDash for Business (DDfB): Enterprise solutions that let companies offer employees food benefits like DashPass for Work, Expensed Meals, Group Orders, and Corporate Gift Cards.
  • The Platform Services team is building the foundation for the next generation of local commerce – helping every merchant connect with customers anytime, anywhere.

DashOS

DashOS is an engineering team built to connect merchants with their consumers end-to-end. The team unifies capabilities that used to be spread across separate products and regions into one team.

DashOS covers three main areas:

  • In-Store Operations: The systems merchants use to run their business day to day — point of sale, kiosks, and kitchen display systems.
  • Merchant Storefront & Loyalty: The tools that let merchants build their own branded storefront and run their own loyalty and rewards programs.
  • Dine-Out & Reservations: The consumer-facing side of the business — helping people discover, book, and enjoy dining and hospitality experiences.

Because DashOS brings together teams that used to work separately, joining this org means helping build shared, global-scale platforms — while still shaping the product experience merchants and diners feel in each local market.

Money

The Money team’s mission is to deliver delightful and innovative payments and financial services products for Cx, Dx, and Mx, and be a catalyst for both top-line and bottom-line business growth.

Operating as both a platform and a product team, their primary responsibility is managing all financial transactions on our platform. This includes everything from charging customers to paying Mx and Dx, all while ensuring that every dollar is accurately accounted for.

The org is built around three pillars:

  • Payment: The core infrastructure behind every transaction on DoorDash — how customers pay, how Dashers and merchants get paid, and how subscriptions like DashPass are billed.
  • Money Products: Financial products that directly serve Dashers, consumers, and merchants — like gift cards, the Crimson card and other Dasher benefits, and merchant financing.
  • Financial Compliance: Ensures every dollar that moves through DoorDash is properly tracked, reconciled, and compliant with financial regulations around the world.

The Money team processes billions of dollars annually, and their payment innovations such as gift cards have contributed millions of dollars in incremental annual Gross Order Value, which highlights the significant impact of the Money team’s financial systems and products on DoorDash’s business!

Ads & Promos

The Ads & Promotions team empowers Restaurants, Merchants, and Brands to reach customers and grow their businesses on DoorDash. The org is built around two halves – ads and promotions.

  • Ads: By combining powerful data with smart placement, the Ads team connects advertisers with the right audiences – driving visibility, conversion, and growth. It’s built around three pillars: Supply (the customer experience), Demand (the advertiser experience), and Delivery (the ML-powered engine connecting the two in real time). The team runs two lines of business – Restaurant Ads, which help restaurants stand out to hungry customers, and CPG Ads, which spotlight everyday brands – backed by the Ads Foundation and Ads Data teams behind the scenes.
  • Promotions: Powers the discounts and deals customers see throughout the app – from the homepage to checkout – helping merchants drive more orders while customers save money.

Looking ahead, the team is working to bring Ads and Promotions closer together into a single, more automated experience for merchants. Both businesses are scaling rapidly, and we’re just getting started – join us as we build the next growth engine for DoorDash.

This post covers the client and platform boundaries behind user experience and interactions with Ask DoorDash: how we moved from hackathon carousels to that model, how one artifact serves three readers, and how context and state stay consistent as consumers move through DoorDash. It follows Building DoorDash Assistant: An Engineering Overview and our deep dives Building Ask DoorDash (Part 2): Intelligence, Building Ask DoorDash (Part 3): Evaluation, and Building Ask DoorDash (Part 4): A Platform for Building and Evolving Agents.

Ask DoorDash turns requests like “build me a $60 dinner for 10 people”, “help me make chicken tacos”, and “snacks for the week” into editable shopping lists. It produces a list with shoppable products, prices, photos, quantity controls, and simple buttons for swapping any item, and checking out. For restaurant requests, stores and dishes are returned that connect into the native shopping flow. Both functionalities run on the same client and platform machinery. This post is focused on the grocery use case, as it puts the most pressure on all of the major complexities of creating an intuitive interface for agent-driven shopping. Grocery lists commonly hold ten or twenty items, go through several rounds of revision, and must easily deal with consumer constraints like budgeting and dietary restrictions.

User sessions from July 2026 showed that in grocery sessions where a shopping list was rendered, consumers averaged close to two UI interactions each. Roughly a third of those sessions went on to apply the list to a cart. The assistant takes the broad intent of consumers and boils it down. The output is native components, which allow for granular refinement by consumers. Simple edits update an authoritative artifact immediately, while changes that require reasoning start a new agent turn.

In order to produce an effective collaborative consumer/agent shopping experience, shopping agents need to integrate with an interactive surface grounded in live commerce data and the consumer’s current app context. When a consumer is shopping in an app, they expect to see at a glance real items they can purchase. Text can describe that environment, but image-based, interactive components let consumers actually traverse it. Context matters too. Consumers opening DoorDash arrive with different expectations than they bring to a general-purpose chat app. They shop by browsing, moving between stores and pages rather than describing what they want. Meeting them where they are is an essential piece of the puzzle.

Big things have small beginnings

The DoorDash grocery assistant began as a hackathon project. We knew from the start that integrating existing DoorDash components would be a key piece of whatever we built. The earliest versions leaned heavily on carousels: first a list of nearby store candidates, then a carousel of search results for each item the agent was looking for. They were rendered inline, each with its search term attached. The result was essentially a chat transcript with commerce results pasted into it.

Functionally, it was close to what we ship today. As an experience it was rougher, and the specifics are worth walking through.

Having found a set of stores, the assistant asked which one to explore and waited for the answer to be typed. Each carousel represented a single item the consumer needed, and picking an option from it added that item to the cart. Every one arrived with its candidates spread out at full width and the choice still open. So building a week of groceries meant working down the stack one selection at a time, with no way to accept a set of items all at once.

Underneath all of it was one failure running in both directions: the interface was built primarily to present the assistant’s work. Every carousel was headed by the query that produced it and candidates sat in the order they were retrieved. Search terms and raw result sets are an agent’s working material. Presenting the results directly is the interface equivalent of printing a stack trace. The assistant deferred just as readily to the interface, working one item at a time because a stack of per-item carousels was what the screen could show. Each had been designed around the other’s limitations rather than around the consumer.

The lesson was not that reusing existing components was wrong; the familiarity they carry is exactly why the assistant feels like DoorDash. We had simply built the wrong container for them. The container is what decides the order they appear in, what they sit next to, and what a consumer can do with them. Above all else, a consumer needs a shopping surface rather than a conversation interface. We had built a conversation interface that happened to contain shopping components. Component reuse gave us the vocabulary. Composing it into a surface was the harder part.

What did survive the hackathon was the foundation: the same streaming infrastructure it established is what the experience runs on today. Meanwhile, the decoding layer that turned tool results into UI did not last. The assistant no longer surfaces tool results at all. Instead, we invested time to build out an artifact architecture that separates out consumer requirements from storage and agent requirements. In other words: we stopped decoding the agent’s working material for display and started to build around the needs of the different components and consumers of the system.

Figure 1: The same grocery request, a year apart. The hackathon build showed a carousel per item, each headed by the query that produced it; selection was left to the consumer. Today, the assistant returns each list as a card in the conversation, with top items already picked.

One artifact, three lenses

A shopping list has three readers: the service that stores it, the agent that revises it, and the person shopping. Each one needs to see something different. Keeping them separate is what lets a consumer’s edit appear immediately and stay authoritative without costing another model turn.

The stored form is the system of record: a JSON blob held as an artifact in Managed Agent Services, described in the platform post. It has to be complete, and it is the only authoritative copy.

The agent does not read that blob directly. An interactive shopping list carries far more detail than a model needs for most follow-ups. Repeating every product name, price, quantity, substitution, and piece of display metadata on every turn would compete with what the consumer actually asked for. So when an agent emits a widget, what stays in conversation context is a compact summary: the store, and what the list contains. When a turn needs exact state, the agent reads the artifact and gets back a reduced view with the display metadata stripped out. That view is unambiguous and small enough to reason over cheaply. A median list persists as 40 to 60 KB of JSON. What stays in the conversation context is a single line of about 240 characters, roughly 250 times smaller.

What the consumer sees is different again, and it takes more than one form. In the conversation the list renders as a compact widget: the store, the name of the list, the ETA, a few item photos, and an “Add to Cart” button carrying the estimated total. That is enough to judge the list at a glance without leaving the chat. A “View List” button opens the full-page view, which presents the complete list and everything needed to work through it in detail. Both views are built to be grasped immediately and acted on directly.

One detail shows how far these views diverge. When the agent chooses a product for an item on the list, it keeps the other products that matched the same search, and the consumer can swap between them with a tap. It keeps ten candidates per item, and 99.7% of items have enough matches to fill that out, so nearly every line in the consumer’s view has one selection and nine alternates behind it. The consumer’s view carries every one of those alternates. The agent’s view carries none of them, because choosing between them is the consumer’s job (unless the assistant is explicitly asked to).

Figure 2: One list, three lenses. The consumer sees prices, photos, and a swap behind every line. The agent sees only the store and what the list contains. The artifact holds all of it, including nine alternates for the lasagna noodles alone.

Staying grounded

Against a cart holding raspberry yogurt, tapping “Recommend cart pairings” returns two suggestions: granola, “because it goes with your yogurt”, and blueberries, “to add fruit to your yogurt” (see Figure 3). The contrast case already exists in our own product, on the same screen: the cart page carries a complementary-item carousel that offers items a consumer may want to add, with no reason attached. Our cart agent could have returned exactly that, a list of plausible items. The items might well have been the same either way. The rationale is what tells the consumer the suggestion was reasoned rather than merely plausible, and since every widget is also a place to act, that distinction does real work.

That is what grounding buys: the difference between a compelling shopping experience and a novelty. If the output of the assistant is not tangible, consumers have little reason to shift away from the rich app experience DoorDash already provides. Text still does the work, but in a supporting role. We deliberately put the assistant’s rationale inside the widgets and around them, instead of showing item cells exactly as they appear elsewhere in the app.

Grounding also has to hold at the level of the data, and there the work is split. An agent chooses a supported content type and supplies the commerce data for it; the iOS client owns presentation and interaction, mapping that content onto native DoorDash design system components. A grocery request produces an editable shopping list with quantities, substitutions, and a subtotal. A restaurant request can produce store cards, item cards, or cart suggestions.

That split is what grounds the data itself. The commerce values in a widget do not come from the model. Prices, inventory, store hours, and cart contents are read from the same systems of record that the rest of the app uses, so a price shown in Ask matches the price on the store page at that moment. Typed content also gives the backend and client a stable contract. Supporting a new widget takes more than a prompt change. The agent has to know when to use it and what to put in it, and the client needs a named schema plus the code to decode and render it. Past that, a new widget reaches the consumer over the same conversation and streaming machinery as every other one.

Figure 3: Two kinds of suggestions against the same cart. Only one of them says why.

Widgets as I/O channels

Each new widget we have integrated has made the assistant feel more functional. The reason has as much to do with input as with output. One boundary governs all of them: a deterministic edit stays in the client and applies to the artifact directly, while a change that requires judgment starts an agent turn. The examples below are organized around where that line falls.

The hackathon build asked which store the consumer wanted, then waited for them to type it. It had no way to choose for itself; today that choice is automatic. Plenty of other questions still need the consumer, and direct text can be a poor way to collect the answers. Text entry is a single-threaded constraint: the consumer may know every answer, but can only supply them as fast as they can type or speak.

This is where widgets earn their place. A consumer may have ten operations to apply to a shopping list, each of them simple. Individually they are trivial; described in prose they become a complex task. It is the difference between looking over someone’s shoulder and telling them which buttons to press, and pressing the buttons yourself.

Structured clarifying questions are the clearest case, and the pattern is borrowed from coding agents: Claude Code’s AskUserQuestion tool does the same job for developers. We ship a widget that interrupts the consumer with a small set of questions (multiple choice and multi-select), letting them steer the conversation at genuinely open-ended points. It has become a core pillar of the interaction model in the grocery agent, and is now being adopted in the restaurant agent to simplify the same class of exchange. We integrate with our consumer memory platform, which narrows choices significantly, but knowledge gaps still remain that need consumer input. Asking is unavoidable. But we can ask in a way that is easy and efficient for the consumer to respond to.

The full-page list view is where the input/output collaboration becomes concrete. A consumer can change the quantity of an item, delete an item, swap an item for one of the alternates the assistant found alongside it, and add the finished list to their cart. Every one of those actions manipulates the artifact directly. None requires the assistant to act on the consumer’s behalf, and none costs an LLM round trip. The swap case puts the hackathon build’s candidate sets to better use. The alternatives a consumer chooses between are the other candidates the assistant retrieved when it selected the item. The list of all options available to the consumer previously filled a carousel. Now they sit one tap behind a decision instead of spread out in front of the consumer as an open question.

Changing the store is the exception, and the exception is the interesting part. It is not a mutation of the existing list. It escalates into an assistant request that rebuilds the list at the alternative store. This interaction is also rare: it was the first action in 0.1% of engaged sessions, which is roughly what we would expect if the line is drawn in about the right place. Put that line in the wrong place and the consumer either waits for the assistant to do something they could have done with a tap, or taps their way through something that could have benefited from the assistant’s judgment.

The artifact makes the split safe, and the two sides write to it differently. A quantity change, a removal, or a swap takes the short path: the client updates the list in place and the edit appears immediately. A store change starts an agent turn instead, because the whole list needs rematching against a different catalog. An agent revision does not overwrite what was there; it produces a new list derived from the previous one.

The effect is that a direct edit is authoritative. Before a turn changes the list, the agent reads the current one rather than working from the transcript, so if the consumer removes oat milk, the next turn begins from a list without oat milk.

The measured behavior over the same period shows where that line sits. Immediately after the first request, nearly three quarters of first next actions came through a component. From the second message onward it inverts, and typed follow-ups make up the majority at every depth that we measured. Controls absorb the immediate, structured refinements; language takes over when the change is larger or harder to express as a tap.

Figure 4: The list across a turn boundary. The consumer removes the apples directly in the list view, with no model round trip. When the store change starts a new turn, the agent reads the artifact rather than the transcript, so the list it rebuilds at the new store is still without them.

Every assistant revision stays live

When the assistant revises a shopping list, the version it replaces stays on screen and stays usable.

That was not our first implementation. The first version hid superseded lists: once a revision arrived, the list behind it could no longer be viewed or edited. One current list, no clutter. It seemed like tidier behavior. What it did instead was make revision feel expensive. If asking for a change might erase a list you were happy with, the safe move is to stop asking, and that undercuts the whole point of a collaborative flow.

So every assistant revision now produces a new artifact derived from the previous one, and each stays where it was created in the conversation. Scroll back and the earlier versions are all still there, still live: a consumer can open an earlier list, edit it, or add it to their cart instead of the latest one. Asking for a change costs nothing, because what they had a moment ago is one scroll away.

Not every list the assistant builds becomes a version the consumer sees. When it assembles a candidate list and concludes the store does not carry what was asked for, that list is never rendered. The consumer is told in text that the store was missing items and that a new list is on the way. It is one of the few places where prose does the work outright, and for the reason behind everything else in this post: interaction is used to simplify, and text is used to justify.

Where a list becomes a cart

An assistant that edits the cart directly can remove an item the consumer wanted, or add one they never confirmed, without being asked. One misstep there is enough to lose a consumer’s trust, because the failure is silent and it lands on the thing they are about to pay for.

So we instead present a shopping list as a proposal. It can hold dozens of items and go through several rounds of revision, whether that means removing a pantry item the consumer already has, changing a quantity, swapping a product, or regenerating part of a recipe, and none of it touches the cart. Committing takes an explicit “Add to Cart”, which is the simplest structure we found that gives the consumer real control without making them supervise every step.

The two objects still have to be reconciled, and the interesting choice is who does it. The assistant reads the cart while building the list, but it does not quietly adjust the list to match: if a recipe needs two bananas, the list asks for two, whether or not there are already four in the cart. Building the list for exactly what was requested, and reconciling it against the cart as a separate step, keeps one decision in one place.

That step surfaces at commit time. When the list and the cart overlap, the consumer is shown the duplicates and chooses whether to replace those quantities or add to them. It is a small moment, but small moments like this are where trust is won or lost.

Carrying context across DoorDash

It is not enough for the assistant to live in one location; it has to be reachable from anywhere in the app. Opening the assistant is already a strong signal of intent, but an ambiguous one. A consumer at that moment may be ordering from a favorite restaurant, buying groceries for the week, shopping for a new pair of headphones, or looking for support on a past order. A consumer who has navigated to a particular grocery store is showing a much sharper signal: it is probable they intend to build a cart at that store. Getting that context to the assistant is what lets it focus on the likely intent instead of asking.

The scope system we designed for this spans the client and the Gateway. The client attaches key pieces of entry context to each request: the topic the consumer entered under (grocery, restaurant), the store they are looking at, and a few related identifiers. The Gateway uses the topic to decide which agent the turn routes to, and the rest to give tools the identifiers they need.

Scope is deliberately a flat set of key-value pairs rather than a typed schema, and the client decides what goes into it. Unrecognized keys are carried through rather than rejected, so a new surface can start sending context without waiting on a Gateway change. The exception is the topic itself: adding a new one means teaching the Gateway which agent it routes to. So, while a new surface is cheap, a new domain is not.

The topic is also the part that applies only to the first message of a context. That gives the assistant solid ground to work from without being overly prescriptive, and it is what lets consumers break out of a scope when they explicitly choose to. A consumer using the assistant inside a restaurant can leave the restaurant scope entirely by asking it to “find me a recipe for this dish, and build me a cart, so I can make it at home.” That request is not free, but the cost is a hop inside the turn rather than a dead end: the restaurant agent has to recognize the request as outside its domain and hand it back before another agent picks it up.

In certain cases, scopes can also allow us to skip the model entirely. Take the “Reorder my last cart” entry point. It could simply send that phrase to the assistant as a message, and while that would work, attaching a scope lets us be more targeted. The “reorder” case is effectively deterministic, so we can bypass the LLM and hand back a shopping list almost immediately. What comes back is an ordinary shopping list widget: the consumer can change quantities, swap products, or ask the assistant to revise it in the chat, exactly as they would with any other. Note that the bypass only holds if the store still carries everything in the previous cart; if an item is unavailable, the request goes to the grocery agent, which rebuilds the list the same way it would from a typed request.

Scoping is also what makes a single continuous chat across an app session possible, but it only works because scope, agent pinning, and session state are kept separate. We treat scope as per-turn context. The pin, described earlier in the blog post series, keeps follow-up turns with the agent that resolved the previous turn. Both the pin and the transcript (the actual conversation) are held against the chat rather than the scope, so a consumer walking from one store to another re-points the pin at a new domain agent without discarding what was said. Several agents specialize underneath, and the whole thing still behaves as one assistant.

Figure 5: One chat, entered from two places. Scope rides along with each request, so asking inside Safeway builds the list at Safeway.

Conclusion

Agent integration into consumer products is still early. However, the foundations are becoming clear. Widgets are where consumers do the work and the consumer/agent conversation is what steers them. There are currently three readers of our shopping list artifact, each needing a different view of it. We use the user-facing part of this to ground the consumer in real products and to make common modifications one-tap actions. As agents take on more of the work, we expect what a consumer needs to see and what an agent needs to reason over will continue to change, and the line between an app interface and a chat interface will become harder to draw.

DoorDash’s experimentation platform manages over 60,000 feature flags across roughly 623 repositories. We identified over a thousand stale feature flags; in other words, the feature they once gated is fully shipped, but the flag and its branching logic still sit in the codebase as dead weight. Cleaning just one up manually takes an engineer one to two hours. With about 2,300 new flags being created every month, the backlog only grows.

So we built a multi-agent LLM system that cleans up stale flags end-to-end from a Jira ticket to a merge-ready pull request (PR) with no human in the loop except an engineer confirming the target value and the PR landing. Of the 50 most recent stale flags it processed, the system produced usable PRs for 45 of them at an average cost of $4.79 and 13.8 minutes per flag, saving countless human engineering hours. The system we built not only had to be effective, but its reliability boundary needed to be completeness, not correctness; if the agent failed, it had to do so safely without making any breaking code changes.

This post covers why off-the-shelf tools couldn’t accomplish this in our codebase, how we designed the system, and what we learned during its deployment. Note: This work was peer-reviewed and accepted for the ICSME 2026 industry track.

Why stale feature flags are worth automating away

Feature flags are a cornerstone of modern software delivery. They let us roll out features gradually, run A/B experiments, and flip kill switches when something goes wrong. But every flag has a lifecycle and once a feature is fully rolled out, its flag becomes stale. stale feature flags are often left as technical debt because an engineer’s time has more value building new features instead of cleaning them. 

Its conditional logic no longer serves a purpose, but it doesn’t just sit there harmlessly:

  • Stale flags can remain in the codebase for years. Prior research found that roughly 25% of feature flags persist for more than eight years in open-source projects; practitioners consistently report that cleanup is burdensome and repeatedly deferred.
  • A stale flag is technical debt. Every stale flag adds a branch, an edge case, and a bit more to read and reason about, which can be especially painful during an incident.
  • Stale flags can be an operational risk. A stale flag preserves a dormant code path that can be reactivated by accident. In one well-documented case, a trading firm repurposed a stale flag during a deployment, triggering $460 million in unintended trades in just 45 minutes.

Multiply tens of thousands of flags and a four-figure monthly creation rate and it becomes clear that manual cleanup simply cannot keep pace. Clearing our stale backlog by hand would cost thousands of engineering hours that would be better spent building.

Dynamic values

Feature flags at DoorDash are called dynamic values (DVs). They are powered by our experimentation platform. Throughout this post, the term DV is synonymous with feature flag.

A DV’s lifecycle follows the pattern creation → experimentation → rollout → stale → removal. We classify a DV as stale when it meets all of the following criteria:

  1. It hasn’t been modified in 90 days.
  2. It’s still referenced in code.
  3. It’s not already in an end-of-life status (archived or retired).
  4. It’s not on an exclusion list.

A daily cron job files a STALEDV Jira ticket for each identified stale DV and assigns it to the DV’s creator. This marks the front of the queue from which our cleanup system works.

Agentic workflow vs. AST-based cleanup tools

The best-known automated approach to flag cleanup is Piranha and its polyglot successor, which simplify flag-gated conditionals by matching patterns in the abstract syntax tree (AST). It works well when a flag is read through a direct API call that a rewrite rule can pattern-match.

But that’s not how DVs are accessed at DoorDash. Our services use a three-layer, dependency-injected wrapper pattern:

// Layer 1: RuntimeKeys.kt – the DV name as a string constant
object RuntimeKeys {
  const val ENABLE_X = “enable_feature_x”
}

// Layer 2: FeatureFlags.kt – a DI wrapper that calls the DV client
@Singleton
class FeatureFlags @Inject constructor(
    private val dvClient: DynamicValuesClient
) {
  fun shouldEnableX(userId: String): Boolean =
    dvClient.getBoolean(ENABLE_X, withContext(“user_id” to userId), false)
}

// Layer 3: business logic – uses the flag via dependency injection
class MyService @Inject constructor(
    private val featureFlags: FeatureFlags
) {
  fun process(userId: String) {
    if (featureFlags.shouldEnableX(userId)) handleNewPath()
    else handleOldPath()
  }
}

This is a clean separation of concerns, but it means the DV name, the API call, and the conditional logic each live in a different file, connected only through method calls and dependency injection. These are semantic relationships, not syntactic ones; AST pattern-matchers can’t follow them. 

Target-value determination

There  is an additional problem a source-code-only tool cannot solve: target-value determination and live rollout querying. A tool that only reads source code could confidently hardcode the wrong value and silently change behavior. Ensuring the tool performs the correct code cleanup requires knowing the flag’s live rollout state, which lives in the experimentation platform, not the repo. For example, DoorDash’s experimentation platform supports the usage of integer, string, and JSON feature flags. A live readout of the target value ensures that the tool is modifying the code with the feature flag’s correct value that represents the system’s current state. Similarly, rollout information is also an important part of the equation. An experiment may have been rolled out to 60% and then abandoned. The tool would need to have the intelligence to determine that this is a partial rollout and ask the user for confirmation before performing any cleanup. This can either trigger the user to complete the rollout/rollback before using the tool to perform the cleanup, or prompt the user to skip the cleanup for now.

Scale and complexity

Finally, even a simple Boolean DV can touch five to 20 files after accounting for the constant, the wrapper, every call site, and updating or deleting relevant tests. This is a job that needs to understand code, not just match syntax. This is exactly why we opted to use LLM agents. But the job wasn’t as simple as writing a prompt and asking the agent to perform the cleanup; that would not achieve a cost- and time-efficient cleanup with guardrails at scale.

A two-phase, human-in-the-loop pipeline

The system is built on Google’s Agent Development Kit (ADK) and runs in two phases with a single human checkpoint between them, as shown in Figure 1.

Figure 1: System architecture. An orchestrator agent runs the Phase 1 analysis, pulls stale DVs from Jira, queries the experimentation platform for rollout state over the model context protocol (MCP), and searches the codebase; an engineer verifies the report. Phase 2 then spawns parallel cleanup agents in isolated git worktrees that edit, test, check coverage, and lint before opening a PR.

Phase 1: Analysis and report

An orchestrator agent, in this case Claude Sonnet, processes stale-DV requests. For each DV, it:

  1. Fetches the stale DV Jira tickets via the Atlassian command-line interface tool.
  2. Queries DoorDash’s experimentation platform over model context protocol (MCP) for the DV’s metadata — its UUID, rollout percentage, and target value.
  3. Discovers the local repository and searches for every code reference to the DV.
  4. Generates a structured report proposing a target value and listing the affected files for each DV.

That MCP integration is the quiet hero of the design. By reading the live production rollout state, the agent determines the correct target value instead of guessing from the code. For example, a DV that was introduced for a feature that ended up being abandoned shows a 0% rollout on the platform, so the agent knows to remove it using the baseline value rather than assuming the feature is on.

Human checkpoint

Before any code changes, an engineer reviews the report and confirms each DV’s target value. This is a deliberate safety valve for ambiguous cases, such as partial rollouts or abandoned experiments, where the right answer needs human judgment. For example, an experiment may have been rolled out to 60% and then abandoned. The human checkpoint allows the user to intervene by deciding, for instance, whether to complete the rollout or roll back the experiment before the cleanup proceeds.

Phase 2: Parallel cleanup

For every confirmed DV, the orchestrator spawns a dedicated removal agent — Claude Opus — with each running in its own isolated git worktree. Up to four agents run concurrently per repository, which is configurable. Each one drives a full cleanup and opens a pull request assigned to the DV’s owner.

Inside a removal agent

Each removal agent runs an iterative workflow as follows:

  1. Search for all references to the DV, including the constant, the wrapper function, and every call site.
  2. Decide a cleanup strategy based on the DV’s type — Boolean, integer, or string — and its usage.
  3. Apply the edits; replace wrapper calls with the target value and simplify the now-constant conditionals.
  4. Remove the wrapper function, the definition, and the constant.
  5. Update or remove affected tests, such as deleting “flag disabled” test branches.
  6. Confirm the build passes.
  7. Confirm the tests pass.
  8. Verify patch coverage is at least 95% on changed lines.
  9. Run the Detekt linter.
  10. If any validation fails, diagnose, fix, and retry.

Autonomy over a codebase only works if the guardrails are real. Four mechanisms keep the agents safe:

  • Git worktree isolation: Each agent works in a dedicated worktree, so concurrent agents never step on each other, and a failed cleanup is discarded by simply removing the worktree.
  • A one-hour hard timeout per agent: This ensures that a stuck session can’t run away.
  • JaCoCo patch-coverage validation: This enforces at least 95% test coverage on the lines the agent changed.
  • Detekt static analysis: This enforces style and catches common issues.

We also run Gradle with –no-daemon to eliminate cross-worktree state pollution during parallel runs. Crucially, an agent is not allowed to open a PR until tests and static analysis pass locally.

Results

Building a system is one thing, but only one question matters when it runs at DoorDash’s scale: How well does it work? In this case, very well indeed.  Of the 50 most recent stale flags the system processed, it produced a merge-ready PR for 45 of them at an average cost of $4.79 and 13.8 minutes per cleanup instead of the one to two hours a manual cleanup would have taken. The system introduced no bugs or regressions along the way.

How we evaluated it

We evaluated the system on the 50 most recent stale DVs it processed across several Kotlin repositories. We labeled each by complexity: 

  • Simple — A single Boolean check in one to two files; n=6
  • Medium — Multiple usages across three to five files with some test changes; n=18
  • Complex — nested conditionals, cross-file dependencies, non-Boolean types, or significant test updates; n=26 

By type, 41 were Boolean, six were string, and three were integer/long. Each PR was judged on three criteria: CI passes — including the 95% patch-coverage gate — code correctness, and developer review. 

Outcomes by complexity

Figure 2: Cleanup outcomes by complexity. Simple DVs are 100% accepted on the first shot; medium DVs reach a 94% cleanup rate; complex DVs show more revisions and failures but still reach 85%.

As shown in Figure 2, 31 of the 50 cleanups were merged in the first shot, 14 needed one minor revision, and five required an engineer to step in. The complexity breakdown maps the system’s reliability boundary:

  • Simple DVs: 100% cleaned up in a single pass.
  • Medium DVs: 94% (17/18) 14 done in a single pass, three after one revision, and one needing engineer intervention.
  • Complex DVs: 85% (22/26), even with cascading parameter removal through more than five method layers, nested conditionals, and heavy test refactoring.

The 14 revisions were minor and self-correcting; six were insufficient patch coverage requiring one more iteration to add tests, and eight were incomplete dead-code removal with a leftover variable or reference deep in a call chain. However, all were fixed in one re-prompt of the LLM.

The most important finding had to do with what kind of thing went wrong. Across all 50 generated PRs, we observed no bugs or regressions introduced by the changed code. Every one of the five intervention cases came down to call-chain depth with cross-interface parameter threading; the agent correctly removed the majority of the dead code, but missed a link or two in a long multi-file chain. In other words, the system’s reliability boundary is completeness, not correctness, alleviating concerns that the system could create larger problems when it was unsuccessful.

Cost and time

Cost and time scale with complexity, but stay firmly in “run-it-at-scale” territory, as shown in Table 1 and Figure 3. 

Table 1 breaks down the average and median performance metrics, specifically time and cost, for the 50 analyzed feature flags categorized by their complexity. This data highlights that while more complex flags naturally require more time and budget, the system remains highly efficient even for the most difficult cleanup tasks.

ComplexitynAvg. timeMedian timeAvg. costMedian cost
Simple67.5 min8.2 min$2.69$2.50
Medium1810.4 min9.9 min$3.46$3.42
Complex2617.7 min14.4 min$6.20$4.50

Table 1: Performance metrics by complexity level across 50 evaluated feature flags.

Figure 3: API cost vs. wall-clock time for each DV, colored by complexity. Simple and medium DVs cluster in the low-cost, low-time region. Complex DVs spread out, with a few outliers approaching 40 minutes and $19. Failed DVs (×) skew toward longer runtimes.

As illustrated in Figure 3, simple and medium DVs cluster tightly in the cheap and fast corner. Complex DVs spread out, with a handful of outliers reaching roughly 40 minutes and around $19. Even at the top of that range, the agent represents a dramatic reduction in engineering effort and reduced costs against even a conservative one-hour estimate of what would be required to remove a single DV manually.

What we learned

  • Specializing models by role pays off. We use Claude Sonnet for orchestration, including broad context, lightweight metadata gathering and planning, and Claude Opus for the removal agents, including deep reasoning about code semantics, call chains, and test dependencies. Reserving the more expensive model for the cognitively demanding editing phase cut our overall API cost without hurting cleanup quality.
  • Worktree isolation is non-negotiable for parallel agents. Our early experiments without worktree isolation produced race conditions where concurrent agents edited the same files and generated corrupted diffs. A dedicated worktree per agent plus –no-daemon Gradle made parallel cleanup safe and made failure disposable.
  • Live experimentation integration was the highest-leverage decision. Without the current rollout state, any automated approach — rule-based or LLM — risks hardcoding the wrong value and quietly preserving the wrong behavior. Reading it over MCP added negligible latency and eliminated a whole class of target-value errors.

Any team considering similar automation should note that our results point to three practical prerequisites: 

  • A reliable source of deployed flag state
  • Worktree isolation for parallel edits, and 
  • Deterministic validation gates before a PR is ever opened.

What’s next

We’re extending the system along two lines:

  1. A phase 1 confidence score to auto-approve low-risk cleanups. For example, a Boolean DV at 100% rollout whose code default already matches production, which reserves the human checkpoint for genuinely ambiguous cases.
  2. A post-cleanup code-quality pass. This would catch semantic issues such as variable names that no longer reflect their purpose once a flag is gone.

Feature flag cleanup is the kind of work that’s individually small, but collectively enormous, particularly at DoorDash’s scale, and perpetually deferred. By combining an orchestrator, parallel removal agents in isolated worktrees, live experimentation platform lookups, and hard validation gates, our feature flag cleanup system transforms a thousands-of-hours backlog into a cheap, reliable background process that produces merge-ready PRs for 90% of the flags it touches at a cost of about $4.79 and 14 minutes each, with no regressions. The remaining boundary is completeness on the deepest call chains, and that’s exactly where we’re focused next.

Interested in building systems like this? DoorDash Engineering is hiring – come help us build at scale.

Flux is DoorDash’s cloud-based agents platform for engineers. In a single month in 2026, we used Flux to automate 130,000 engineering tasks. Rapidly expanding after Q1 2026 debut, Flux already powers high-volume background workflows across DoorDash, including more than 25,000 automated code reviews each week, and more than 300 unique playbooks and 10,000-plus invocations used every week. These workflows can run unattended, in parallel, around the clock.

In this blog post, we’ll walk through the limitations that pushed us beyond local, laptop-based agent workloads, why we chose to build Flux in-house rather than rely solely on hosted coding agents, and the platform primitives such as agent sandboxes, MCP gateway, playbooks, and invocation surfaces to make agent delegation repeatable and secure.

Flux background workflow use-cases

Table 1: A snapshot of Flux usage across DoorDash in a single month, spanning automated code reviews, playbook runs, and background task completions

Where we started

Over the past year, users running agentic workloads on their laptops have quickly hit limitations:

  • Resources and availability. A laptop has a fixed number of CPU cores, limited memory, and a battery, all shared with every application on board. Agentic workflows often need to run compute-intensive tasks such as builds, tests, and large searches in parallel, causing laptops to run out of capacity fast. The workflows also depend on the device being powered on, connected, and available; work pauses when an engineer closes the laptop, loses connectivity, or steps away.
  • Safety controls. Laptops typically have broad access to sensitive credentials and systems, including SSH keys, VPN sessions, and authenticated tools. Giving an autonomous agent that same level of access creates unnecessary risk and a potentially large blast radius. Local environments also make it harder to tightly scope what an agent can access and for how long.
  • Visibility and auditability. When workloads run across individual laptops, execution is fragmented and difficult to monitor. It becomes harder to understand what is running, where it is running, on whose behalf, and which systems or files it has touched.

Our thesis to address these issues is simple: 

Delegate tasks to secure, autonomous coding agents so engineers can dedicate more energy to innovation, critical thinking, and solving complex problems.

Why we built Flux in-house

Hosted coding agents are useful, but they force a hard tradeoff: Either send sensitive code and execution context to a third party, or open a path from that third party back into internal systems. For DoorDash, the harder problem was not just getting an agent to write code; that’s mostly solved. It was giving that agent the right environment, tools, permissions, integrations, and constraints.

Our strategy is to control the primitives around the agent, including orchestration, sandboxes, workflows, permissions, integrations, and the DoorDash-specific context agents needed to work effectively. We also designed those primitives to be modular, giving us the flexibility to use the best third-party tool for each job or to build in-house when deeper security, integration, performance, or UX ownership matters. 

These primitives democratize workflow creation and make systems more adaptable to future use cases. Because they can be composed in different ways, teams can build new agent workflows without reworking the underlying infrastructure or prescribing how each engineer should structure their workflow. For example, we run both the evals for our code review on Flux infrastructure.

Primitives, not workflows

As shown in Figure 1, Flux is built around four platform primitives: sandboxes, the model context protocol (MCP) gateway, playbooks, and invocation surfaces. Together, they make agent delegation repeatable. A playbook defines the work. A cloud sandbox gives the agent a real place to do it. An agent gateway controls what systems the agent can access. And invocation surfaces allow engineers to start and receive work from the places they already use.

Figure 1: The four platform primitives that make up Flux — sandboxes, the MCP gateway, playbooks, and invocation surfaces — and how they connect to turn a task into work an agent can safely carry out

Sandboxes provide the execution environment

Local agents work well for interactive development, but they are a poor fit for unattended workflows. They depend on individual engineers’ laptops, compete for local resources, are difficult to audit, and do not scale efficiently across parallel tasks.

Flux moves execution into isolated cloud sandboxes backed by Firecracker micro virtual machines (microVMs) for hardware-level isolation. Each sandbox is provisioned with the repositories, developer tools, secrets, and runtime dependencies the task requires, giving agents a complete engineering workspace while providing DoorDash with a consistent execution, security, and observability model.

Controlling this layer lets us support real engineering workflows, including changes across multiple repositories and multiple pull requests from a single session. Flux has a 95th percentile service level objective of under five seconds for the full end-to-end setup — from starting the microVM to cloning the required repositories, installing build tools, and configuring the supported coding agent harnesses.

MCP gateway provides governed access

Agents need access to the systems that engineers use every day, including continuous integration (CI), observability platforms, issue trackers, deployment tools, code search, documentation, and service metadata. But granting broad, unrestricted access should not be the default.

Flux connects agents to internal systems through an in-house MCP gateway called Agent Gateway. Each playbook declares the tools it requires, and Flux grants only the scoped permissions needed for that task. Every action is logged, creating a clear audit trail.

This gateway architecture gives us a centralized control point for authentication, authorization, observability, usage tracking, and policy enforcement, all of which make agent access both safer and easier to operate at scale.

Playbooks define the work

A playbook is a reusable unit of agentic work — the equivalent of a Docker container for skills and agent-driven tasks on the Flux platform. Defined in a single markup YAML file, it packages the task, inputs, context, skills, tools, permissions, validation, expected outputs, and safety boundaries needed to execute work consistently.

Playbooks can combine agentic steps, which provide flexibility and judgment, with deterministic steps, which offer predictability, lower cost, and easier validation. This lets teams move logic between agent-driven execution and conventional code as requirements evolve, without redesigning the workflow.

Invocation surfaces meet developers where they are

The same playbook can be triggered from Slack, GitHub, cron, the CLI, or a conversational skill. That means teams can define a workflow once and invoke it from whichever surface best matches the moment:

  • Slack for collaborative delegation
  • GitHub for PR and CI automation
  • Cron for recurring maintenance
  • CLI for direct developer control, or invoked through a skill

This is what makes Flux easy to adopt.

Lessons learned

Building Flux taught us as much about product adoption as it did infrastructure, including:

  • Start narrow to earn trust. We began with automated code review instead of trying to automate the entire software development life cycle. Code review was frequent, measurable, and easy for engineers to evaluate. It gave us a production workflow where we could tune quality, latency, cost, and behavior before expanding into CI triage, on-call tasks, maintenance playbooks, and ticket-driven development.
  • Make the work visible. Our first Slack integration created private channels for each agent run. That made Flux useful for individuals, but it did not create team habits. Moving work into public threads changed the adoption pattern. Engineers could see what others delegated, watch Flux make progress, review the output, and build trust together.
  • Playbooks need enablement. Reusable workflows do not appear just because the platform exists. Workshops and hackathons helped teams translate repeated operational work into playbooks. The primitives made automation possible; enablement helped teams recognize which workflows were worth encoding.

What’s next

In future posts, we will go deeper into the platform primitives that make Flux work, and the developer experience for building new workflows. We’ll also discuss the applications we’ve built on top, including Flux Responder, our internal Slack agent.

Acknowledgements

Thanks to Adam Rogal, Adam Yarger, Andy Fang, Ashwin Kachhara, Fan Xia, Ivan Rudovol, Jason Prasad, Jialu Deng, Justin Block, Justin Deocampo, Justin Fan, Keith Lyall, Praneet Singh, Sean Chen, Tyler Berrett, and Volanda Zhu for their contributions to the platform and this post.

Stay Informed with Weekly Updates

Subscribe to our Engineering blog to get regular updates on all the coolest projects our team is working on

AI agents become useful when they can take action in real systems. At DoorDash, that means reaching internal APIs, engineering systems, observability platforms, ticketing systems, knowledge bases, and third-party SaaS products. The model context protocol (MCP) made those tools easier to expose by giving agents and servers a shared way to describe, discover, and invoke capabilities.

As agents moved from experiments into real workflows, we ran into a different problem. While MCP standardized the shape of a tool call, it did not answer key production questions around that call:

  • Which agent is allowed to call this tool?
  • Which user, team, or service is it acting for?
  • Which credential should be used?
  • Which subset of tools should the agent even see?
  • How do we revoke access?
  • How do we know what happened after the call?

These questions grow quickly. A coding agent might need GitHub, Jira, code search, continuous integration (CI), observability, and docs. A third-party MCP server might expose hundreds of tools when a workflow needs only five. A user-facing agent might need a user’s OAuth grant, while team automation should use a non-personal service principal. If each team solves these issues independently, every agent-tool pairing ends up with its own auth code, OAuth flow, secret handling, tool catalog, rate limits, and logs.

That’s why we built an Agent Gateway to make agent-tool access a platform capability. It is a single governed entry point where agents discover and invoke tools. The gateway authenticates the caller, checks authorization, exposes only the approved tool surface, injects the right credential, routes to the downstream MCP server, and records a structured usage event for every call.

The gateway is not just a proxy. It is the control plane for the agent ecosystem. It governs who can call tools, curates which tools agents can see, packages tools into task-oriented surfaces, and gives DoorDash one place to observe, rate-limit, revoke, and improve agent-tool access.

Problem: Tool access has three parts

While MCP helps agents invoke tools, the harder problem in production has three dimensions:

  1. The first is access. The platform needs to know who is calling, whether they are allowed, and which credential model applies: internal identity, gateway-held token, per-user OAuth, or service-principal access.
  2. The second is tool-surface curation. Agents should not receive a raw dump of every tool exposed by downstream MCP servers. Smaller, task-relevant catalogs improve safety, reduce model confusion, and make agents easier to operate.
  3. The third is operations. Tool calls need rate limits, traces, metrics, usage events, cost attribution, ownership metadata, and production visibility.

Our Agent Gateway exists because access, curation, and operations belong in one shared platform, not copied into every agent and every MCP server.

Gateway architecture

The gateway has two core pieces: a proxy and a registry, as shown in Figure 1 below. The proxy is the data plane; it receives each MCP request, authenticates the caller, authorizes the action, applies rate limits, injects credentials, forwards the request, and emits observability data. The registry is the control plane’s source of truth; it stores agents, MCP servers, owners, transport configurations, auth modes, policies, discovered tool catalogs, and tool-surface configurations.

Figure 1: High Level Architecture of Agent Gateway

This split gives the gateway a few properties that are hard to get from point-to-point integrations:

  • Agents use one consistent MCP endpoint pattern instead of learning every downstream server’s auth and routing model.
  • Tool owners register capabilities once, attach ownership and policy, and see production usage.
  • Security teams get one place to enforce access, revoke grants, and audit calls.
  • Platform teams can improve the shared path once and have every agent inherit the improvement.

We also keep internal and external trust boundaries separate. Internal DoorDash workflows and external-facing agentic use cases run through separate proxy planes with shared libraries and registry concepts. That contains the internet-facing blast radius and lets each auth model evolve independently.

Centralized identity, authorization, and secrets

Every gateway request resolves to a caller: a user, a service, or an agent acting with delegated user context. That identity is carried through authorization, credential injection, routing, and observability so every downstream action can be attributed.

The gateway checks authorization centrally. The policy questions do not stop with whether  this caller can access this server. They may also be:

  • Can this agent access this server?
  • Can this user call this tool through this agent?
  • Can this tool be exposed in this environment?
  • Can this caller use a write-capable variant, or only a read-only one?
  • Can this workflow use a team service principal, or does it require per-user OAuth?

Because policy lives in the gateway, DoorDash has a single place to change access and revoke it. Tool owners avoid rebuilding authorization for every server, and agent builders avoid encoding security decisions in prompts or application code.

Credential handling follows the same pattern, as shown in Table 1:

Auth modeWho holds the credentialWhat the gateway does
Internal service identityDoorDash infrastructureForwards verified caller context to internal services
Gateway-held tokenGateway secret storageInjects a vendor or service token without exposing it to the agent
Per-user OAuthEncrypted per-user grant storeInjects and refreshes the user’s token for user-scoped actions
Service principalGateway-managed team principalMints or brokers short-lived non-personal credentials

Agents should not hold raw credentials, such as vendor API keys, OAuth refresh tokens, or borrowed human grants for team automations. The gateway keeps those boundaries explicit and auditable.

Per-user OAuth without breaking the agent turn

User-scoped third-party tools need user authorization. Reading a user’s docs, filing a ticket as that user, or updating SaaS data on their behalf cannot safely use a shared key. Before we deployed the gateway, each team tended to build its own OAuth flow, token storage, and “go connect first” experience.

The gateway centralizes the OAuth grant. The unit of access is scoped to the agent, user, and server. On the first call that requires authorization, the gateway starts the provider’s OAuth flow, stores the resulting access and refresh encrypted tokens, and injects the user’s token on future calls. Agents never see the raw token.

For clients that support MCP elicitation, the gateway can pause the tool call, ask the client to show a connection prompt, and resume the original call after the user authorizes it. For clients without elicitation support, it returns a structured authorization-required response with a connecting URL, as shown in Figure 2. This turns connection from a failed turn into a recoverable part of the tool call.

Figure 2: Elicitation handshake

Curated tool surfaces: Bundles and filtering

Agents do not naturally think in MCP servers. They think in tasks. A coding agent does not want GitHub, Jira, observability, and docs as separate setup steps; it wants the tool surface needed to investigate an issue, modify code, open a pull request, inspect CI, and understand production behavior.

At the same time, most downstream MCP servers expose far more tools than any one agent should use. Third-party servers may publish hundreds of operations, including admin actions, destructive actions, billing APIs, and niche provider-specific features. Most DoorDash workflows need only a small approved subset of these.

The gateway solves both problems with curated tool surfaces, as shown in Figure 3. Bundles combine tools from multiple MCP servers into one logical MCP endpoint. Filters decide which tools from each server are exposed for a given bundle, agent, user group, environment, or audience.

For example, a developer-tools bundle can include:

  • selected GitHub tools for repository and pull-request workflows;
  • selected Jira tools for issue lookup and updates;
  • selected observability tools for logs, metrics, and traces;
  • selected code-search and documentation tools; and
  • selected deployment or feature-flag tools.

Figure 3: User experience with bundled MCP pack

The agent connects to one gateway URL, such as a developer-tools endpoint. Behind that URL, the gateway fans out tools/list across the servers in the bundle, applies authorization and tool filters, namespaces or aliases tool names where needed, and returns one coherent catalog. When the agent invokes tools/call, the gateway enforces policy again, routes the call to the correct downstream server, and applies that server’s credential model.

This gives agents a product-quality interface instead of a raw dump of downstream capabilities, and includes such things as approved tools, stable names, clearer descriptions, ownership metadata, and audience-specific bundles. Engineering, data analysis, support operations, and external bundles can all use the same gateway primitives while exposing different tool surfaces.

The benefits are practical:

  • Agent setup is simpler: One gateway endpoint instead of many server endpoints.
  • Tool discovery is safer: Agents only see tools inside the approved boundary.
  • Model behavior improves: Smaller catalogs reduce irrelevant choices and tool confusion.
  • Authorization stays centralized: Discovery and invocation enforce the same policy.

This is where the gateway becomes more than an access proxy. It shapes what agents can discover, what they can call, and how much irrelevant context they carry. A curated tool surface can be the difference between an agent that picks the right tool and one that wanders through an oversized API catalog.

Observability, cost attribution, and downstream protection

Because every call flows through the gateway, each request can emit a structured event with:

  • the server, tool, bundle, and owning team;
  • the user, agent, service, and platform involved in the call;
  • authorization result, status code, error source, and latency breakdown;
  • request and response size; and
  • downstream-reported cost metadata when available.

The gateway also emits metrics for request volume, per-tool latency, authorization decisions, OAuth refresh outcomes, rate-limit decisions, streaming connections, and upstream failures. Trace propagation lets teams follow a call from the agent through the gateway to the downstream server.

This has direct operational value; security can audit access, platform teams can find noisy agents, tool owners can see adoption and errors, and infrastructure teams can attribute cost.

The same point protects downstream systems. Rate limits can be scoped by server, tool, caller, user, bundle, or caller type. New limits can run in shadow mode before enforcement, showing what would have been rejected without breaking production traffic.

The gateway turns governance into data instead of relying on every team to log the right fields and enforce the right limits.

Self-serve onboarding

A gateway only works if teams prefer to use it rather than bypass it. Registration, discovery, filtering, and bundle management all must be self-serve.

Through the control-plane UI and API, teams can register MCP servers and agents, configure auth, discover tools, attach ownership, define filters, add tools to bundles, and inspect production usage.

The onboarding loop for the gateway follows these steps:

  1. Register the MCP server.
  2. Discover its raw tool catalog through tools/list.
  3. Select and approve the tools DoorDash wants to expose.
  4. Attach auth mode, ownership, and policy.
  5. Add the approved tools to one or more bundles.
  6. Watch traffic, latency, errors, authorization decisions, and cost.

Governance that requires tickets does not scale. The paved road has to be easier than copying a secret into an agent and connecting directly to a server.

What changed

The gateway gives each group a different benefit:

  • Agent builders get one integration, one curated catalog, and no downstream auth, OAuth, secret handling, or per-tool routing.
  • Tool owners get a managed distribution path with access control, approved tool exposure, and production usage data.
  • Security teams get centralized policy, secrets, OAuth grants, revocation, and audit trails.
  • Platform teams get leverage; identity, rate limiting, observability, tool quality, cost attribution, and builder experience all improve in one place.
  • Agents get smaller catalogs, clearer tool names, task-oriented bundles, fewer irrelevant choices, and recoverable connection flows.

Adoption

Adoption is the real test of a platform: teams only route through it if it’s easier than going direct. By that measure, the Agent Gateway has become the default path for agent–tool access across DoorDash engineering:

  • More than 200 MCP servers are registered behind the gateway, together exposing thousands of tools curated into approved, task-scoped subsets rather than raw catalogs.
  • More than 30 agents and services, used by thousands of employees, reach those tools through the gateway and none of them handle raw credentials. 
  • Millions of tool calls every week are routed through the Agent Gateway, with each one authenticated, authorized, and recorded as a structured usage event.
  • Onboarding is self-serve and fast. Registering a new MCP server and its tools takes minutes, and pointing an agent at an already-registered tool is even faster.

Lessons learned

Building and scaling the gateway reshaped how we think about agent–tool access. A few lessons stand out, not just at DoorDash:

  • MCP solves invocation, not governance. Identity, policy, secrets, curation, observability, and revocation become the hard parts.
  • The discovered tool catalog is an interface, which means that names, descriptions, filtering, grouping, and audience matter.
  • Bundles are the right unit for workflows. Agents need task-oriented toolkits, not server lists.
  • Tool filtering improves both security and agent quality by exposing a smaller, clearer set of actions.
  • Credentials belong in the gateway, where they can be rotated, audited, and revoked centrally.
  • Missing OAuth grants are normal states, not exceptions, handling connection inside the protocol.
  • The governed path must be self-serve or teams instead will go direct.

What’s next

The next major investment will be in stronger agent identity and user delegation. The target model gives every agent a cryptographic identity and lets the gateway mint short-lived delegated credentials scoped to the user, agent, task, and target tool. That gives the audit trail two real principals: the user and the agent.

We are also investing in builder tooling, including quality and security report cards, checks for risky tool descriptions, detection of secrets or personally identifiable information in errors, scaffolded server creation, automatic registration, dynamic tool discovery, and redacted tool-call event streams for analytics and compliance.

Dynamic discovery continues the same theme. Instead of giving an agent every tool in a bundle, the gateway can use task context, policy, and usage signals to surface only the tools likely to be useful for the current job.

Conclusion

The Agent Gateway turns agent-tool access from repeated integration work into a shared platform capability. MCP made it easier to describe and invoke tools; the gateway makes that access governed, curated, observable, and scalable.

Acknowledgements

Introduction

We built Ask DoorDash on a common platform that lets domain teams build and evolve their agents without rebuilding the systems beneath them. We judged the platform by two practical outcomes: how quickly teams could add features and domains, and how quickly they could evaluate and release improvements to cost, quality, and latency.

We launched Ask DoorDash with Restaurant and Grocery support in roughly two months, and it has since handled more than two million conversations. Adding Reservations, our third domain agent, took one week, roughly 10x faster than building the initial domain agents.

Our shared evaluation harness and rollout controls give teams clear quality signals as they move changes into production. Within a week of a new LLM’s release, we evaluated and deployed it, cutting p50 turn latency by 35% with no drop in quality scores. A subsequent model upgrade cut p50 turn latency by another 40%.

This post explains how we chose what belonged in the platform, how its components fit together, and the tradeoffs behind those decisions. It follows our engineering overview, our deep dive on intelligence, and our deep dive on evaluation. A deep dive on user experience will follow.

How we chose what to standardize

We centralized a capability only when multiple domains needed it and separate implementations would create reliability or operational problems. Orchestration, memory, model access, tracing, evaluation infrastructure, and rollout controls met that bar. Domain teams retained ownership of the instructions, skills, tools, evaluation criteria, and model choices that define their agents.

Where a suitable industry standard existed, we adopted it. Agent2Agent (A2A) defined how agents communicate, while Google Agent Development Kit (ADK) provided the framework for building and running them. These standards sometimes limited our design choices, but they gave teams shared contracts and saved us from developing proprietary equivalents.

For recurring problems without a suitable standard, we built the capability ourselves. A regression in shared code could affect several agents, so evaluation and rollout controls were mandatory. Every shared change had to pass common quality checks and release safeguards.

Ask DoorDash architecture

The central architectural decision was what every agent should share and what each domain should own. We built a shared execution path around specialized, domain-owned agents. A single general-purpose agent would grow harder to reason about as it took on more responsibilities, while fully independent agent stacks would duplicate infrastructure and produce inconsistent experiences.

As Figure 1 shows, each request enters through the Gateway. The Gateway authenticates the user, assembles entry-point context, and translates between the client’s HTTP and streaming interfaces and the platform’s A2A protocol. It then passes the request to the Orchestrator, which chooses the domain agent for each turn while preserving continuity as the conversation moves between domains. The selected agent loads the appropriate skills and uses Model Context Protocol (MCP) tools to interact with existing DoorDash services.

Below that request path, the platform provides session state, memory, artifacts, model access, tracing, evaluation infrastructure, and rollout controls. Domain agents use these common services while retaining control of their instructions, skills, tools, and evaluation criteria.

Figure 1: Ask DoorDash separates domain-specific agent behavior from the shared execution and production capabilities used across the system.

A Gateway for responsive, multimodal experiences

DoorDash client gateways were built for APIs that return a complete response quickly. Agents work differently. The first text may be ready while the agent is still calling tools, and the final response may combine prose with interactive store or item cards. We needed to stream that work without teaching every client how agents communicate.

The Gateway handles this translation. It authenticates the user, adds context about where the conversation started, and converts client HTTP requests into A2A requests. On the return path, it uses the Vercel AI SDK to send text updates and widget payloads to clients over Server-Sent Events (SSE). Clients render the updates as they arrive, so users can see progress while the agent completes the turn.

Building and operating the Gateway took work, but it concentrated work that would otherwise be duplicated across every DoorDash client. Agents can change behind one stable client contract without requiring coordinated client releases. A new domain agent reuses the existing authentication and streaming path instead of integrating separately with each app.

One assistant, specialized domain agents

A conversation can cross domains without warning. A user might start with “I want chicken pot pie delivered in under 30 minutes,” then decide, “actually, I want to make it at home.” The first request belongs to the Restaurant domain and the second to the Grocery domain, but the experience needs to feel like one conversation.

We tried a single-agent architecture, but Restaurant, Grocery, and Reservations rely on different tools, policies, and evaluation criteria. Combining them would make the agent harder to test and force every domain onto the same release cycle. Each domain therefore owns a specialized agent. The Orchestrator reads the conversation and routes each turn to the appropriate agent over A2A.

That routing adds latency and input tokens, even when the conversation remains in the same domain. To reduce this cost, the Orchestrator pins follow-up turns to the selected domain agent. Pinning lets that agent handle subsequent requests directly, without another routing step. If the conversation changes direction, the domain agent recognizes the out-of-scope message and returns control to the Orchestrator. The Orchestrator reroutes the request within the same turn, hidden from the user.

Figure 2: Pinning avoids an Orchestrator call on follow-up turns while still allowing the conversation to move between domains.

Skills keep context and cost bounded

Specialized agents kept Restaurant, Grocery, and Reservations from sharing one enormous context window. As each agent gained capabilities, however, the same context problem appeared within individual domains. New features added instructions and tools to every turn, even when the turn never used them. That increased input-token cost, and overlapping or conflicting instructions confused the agent.

We added skill support to our harness to modularize instructions and tools. On each turn, the agent loads only what is relevant. A user might begin with, “find me a highly rated Thai restaurant that delivers in under 30 minutes,” then say, “show me the menu at the first one and add pad see ew to my cart.” Both requests stay with the Restaurant agent, but the first loads the search-discovery skill. The second loads the cart-ordering skill, which handles menu browsing and cart actions.

Dynamic loading adds a selection problem. The selector must recognize when a capability is needed without loading unrelated context too often, so we evaluate skill selection as part of the agent’s behavior. The added complexity was worthwhile because teams could add capabilities without expanding the context of every turn.

To measure the effect, we counted the tokens in the agent’s base and loaded skill instructions. We excluded conversation history, consumer messages, tool schemas, and tool results. The median skill-scoped turn used fewer than half as many instruction tokens as the monolithic system prompt.

AgentBase and skill instructions with all skills loadedMedian skill-scoped instructions per turnReduction
Restaurant~42,000 tokens

~20,000 tokens50%+
Grocery~25,000 tokens~10,000 tokens60%

Safe, reusable service access through MCP

Ask DoorDash needs access to DoorDash services to search for stores and items, read menus, manage carts, and act on a user’s behalf. The APIs behind these operations assume deterministic application code. An LLM choosing operations at runtime needs a narrower interface. Exposing the APIs directly would force the model to interpret low-level interfaces. Encoding permissions and business rules only in the prompt would add context without guaranteeing enforcement.

We built a shared Model Context Protocol (MCP) layer between the agents and DoorDash APIs. Each MCP tool exposes a focused operation with the inputs and outputs the model needs. The model chooses which tool to call. Deterministic code validates each request and enforces permissions and business rules before the call reaches the underlying service.

Prompts help the model choose the right operation, but that guidance is advisory. Validation and enforcement run in tool code on every call, creating a stable safety boundary as prompts and models change.

Tool design requires balance. A tool that is too broad can present the model with too many choices, while one that is too narrow can turn a task into a long chain of calls. Our shared MCP server now provides more than 60 tools across public and internal agent workflows. A new agent selects the tools it needs from that library, and improvements to validation, telemetry, or service integration benefit every agent that uses them.

Figure 3: Prompts influence what the model requests; MCP tool code determines what can execute.

Production readiness by default

ADK gave us the basic pieces for building an agent, including instructions, tools, callbacks, sessions, and model wiring. DoorDash agents also needed distributed tracing, model access, durable state, and rollout controls. Without a shared layer, each domain team would have to make those decisions and operate the resulting infrastructure independently.

We built reusable modules on top of ADK for capabilities that should work the same way across agents. Domain teams still choose their instructions, tools, and model configuration. The platform connects those choices to the systems required to run agents reliably in production.

Tracing is one example. A team enables it through configuration, and the shared SDK propagates a trace ID through A2A calls, MCP tools, and downstream DoorDash services. Engineers can follow one request across the Orchestrator, domain agents, and tool calls instead of piecing together logs from separate systems. The same trace data supports debugging and evaluation. Providing tracing through the platform saves roughly a month of observability work for each new agent launch.

Model access follows the same principle. Domain teams choose which models their agents use, while the platform standardizes how those models are invoked, traced, and protected by fallback behavior. Teams can evaluate and adopt new models without rewriting provider-specific integrations. That common path enabled the rapid model upgrades and latency improvements described earlier.

Shared state for reliable conversations

Earlier attempts at DoorDash agent products in 2025 showed how quickly the experience breaks down when state is unreliable. An agent could respond well to one request, then forget or misuse information from an earlier turn. Solving that problem required more than adding conversation history to the prompt.

Ask DoorDash uses three forms of state. Session state tracks the active conversation and work already completed within it. Memory preserves information that may be useful in later conversations, such as a user’s preferences. Artifacts hold structured outputs that agents create and update, including shopping lists and interactive cards. Each has a different lifetime and access pattern.

Earlier DoorDash agent projects implemented these capabilities independently. That duplicated persistence work and made reliability depend on each team’s choices. We centralized them in Managed Agent Services, which provides ADK-compatible APIs for sessions, memory, and artifacts. Domain agents use the same interfaces without operating their own stateful systems.

Centralizing state still requires separate storage and lifecycle rules for session data, long-term memory, and artifacts. Managed Agent Services keeps those boundaries in one place. Each domain still decides what information its experience should save and retrieve.

Managed Agent Services and the memory architecture are covered in more detail in Part Two.

Stay Informed with Weekly Updates

Subscribe to our Engineering blog to get regular updates on all the coolest projects our team is working on

What the platform changed

Reservations was a concrete test of the platform. The team reused the production path already serving Restaurant and Grocery and launched Reservations support in one week, roughly 10x faster than building the initial domain agents.

Domain teams keep ownership of agent behavior and quality criteria. The platform provides the evaluation harness, MCP tools, tracing, model access, and rollout controls, so improvements to those components reach every agent that uses them.

Shared infrastructure increases the blast radius of mistakes. A defect can affect several agents, and a premature abstraction can force different products into the same shape. We standardize a capability only after multiple domains need it and separate implementations would create reliability or operational problems.

Join Us

If building agent platforms and user-facing AI at scale sounds interesting, see our open engineering roles at careersatdoordash.com.

Terms of Service Consumer Privacy Applicant Privacy Notice

© 2026 DoorDash