Inside DoorDash’s one-click simulation and evaluation platform for support chatbots

Shipping high-quality support chatbots is an end-to-end problem. Model quality matters, but it is only one part of the system. Prompts, retrieval, knowledge bases, policies, and orchestration all shape the customer experience, and even a small change in one layer can affect outcomes in unexpected ways.

For a long time, our validation loop depended on testing in production. We would roll out a change to a small slice of users – often just 1% – and manually review transcripts to decide whether it was safe. That approach worked when launches were infrequent and the system surface area was smaller. But as we scaled LLM-powered support, the limitations became clear.

Pre-launch validation was slow and manual. Iteration cycles were bottlenecked by transcript review. Long-tail scenarios – such as policy edge cases, data-dependent failures, and rare escalation paths – were difficult to test systematically. And even after launch, prompt edits, knowledge base changes, or infrastructure updates could quietly introduce regressions without a reliable end-to-end baseline.

Simulating for speed and reliability

We built a one-click simulation and evaluation platform for support chatbots to make pre-launch testing faster, broader, and more reliable. Instead of waiting for live traffic, we simulate it. The platform generates realistic, multi-turn customer conversations grounded in specific scenarios and personas, such as item issues, late deliveries, subscription confusion, and policy edge cases.

This is not just a prompt playground. It is a white-box, end-to-end testing system. We can control inputs, inspect intermediate steps, and evaluate outputs against structured expectations. This lets us run regression tests at scale, compare versions side by side, and catch behavioral shifts early — whether they come from prompt changes, knowledge base updates, or model upgrades.

Because every simulated conversation is structured and labeled, the platform also supports continuous improvement. It gives us scenario-diverse data to refine prompts, policies, and business logic over time, instead of treating chatbot quality as a one-time launch problem.

Why it matters

The biggest change is speed and confidence. What used to take hours of waiting on live traffic can now be done in minutes. Instead of relying on manual review cycles that stretched from weeks to months, teams can now iterate with a single click and get production-like signals almost immediately.

We saw this clearly when we validated the framework on the chatbot’s order status inquiry feature:

• Simulation: 302 conversations in five minutes, with a 46% escalation rate
• Production (1% traffic): 175 conversations in seven hours, with a 44% escalation rate
• Result: Simulation matched production-like behavior in a fraction of the time

That alignment gave us confidence that the simulator was not just generating plausible conversations; it was generating representative ones.

More broadly, the platform helped us reduce hallucinations in simulations by 90% for chatbot development. That reduction reinforced simulation as a quantitative pre-production gate, not just a qualitative sandbox.

In practice, that means we can:

• Validate business-critical outcomes earlier, such as escalation rate and resolution quality.
• Reduce launch risk before exposing changes to customers.
• Run more experiments with far less manual overhead.

This marked an inflection point for us: Simulation became a fast, measurable feedback loop that helps us ship with more confidence. By validating changes before they reach production, we ensure they are well-tested end-to-end, leading to more reliable behavior in production and a higher-quality, more consistent customer experience. Each iteration is exercised at scale, allowing us to continuously improve quality while reducing risk.

In the remainder of this post, we’ll walk through how the platform works, how we evaluate realism and quality, and what we learned from deploying it in practice.

Architecture overview

The platform is designed at a high level as a closed-loop system, as shown in Figure 1: Generate realistic test scenarios from production signals, simulate end-to-end conversations against a chatbot, evaluate the results against feature-aware criteria, and feed insights back into iteration.

The flow starts with a testing infrastructure trigger (our internal testing framework), which kicks off a batch of simulations. The simulator loads test scenarios from storage, initiates multi-turn conversations with the target chatbot, and routes all dependent tool calls through a mock layer. After a conversation ends, the full transcript is sent to the evaluation framework, which scores it using production-aligned criteria. Results are persisted and surfaced in a dashboard for analysis and regression tracking.

Two architectural principles guided the design:

• White-box end-to-end (E2E) testing: For LLM-powered systems, it is fundamentally insufficient to simulate only customer messages at the message level. Much of the chatbot’s behavior is driven not just by user input, but by how it retrieves and reasons over context, interacts with tools, and reacts to system signals.

To address this, we simulate the entire conversation loop, including tool interactions and context retrieval, while keeping every step observable and controllable. In particular, we invested heavily in designing realistic context mocking so that the chatbot operates on production-like signals rather than synthetic placeholders.

• Production grounding: Scenarios and mock data are derived from real production transcripts and APIs, so simulations reflect realistic user and system behavior.

The following sections break down the core components.

Test scenario generation pipeline: From production transcripts to replayable simulations

The platform’s foundation is an offline generation pipeline that converts production support transcripts into structured, replayable test scenarios. This gives us a scalable way to turn real customer interactions into reusable end-to-end test coverage.

The pipeline starts with production support transcripts grouped by issue type. For each transcript, we retrieve and analyze key contextual signals including such things as customer order details, delivery address and drop-off options, drop-off instructions, customer–Dasher conversations, and account/order metadata.

We then leverage an LLM to extract higher-level abstractions from the transcript, including:

• The customer story (what happened from the customer’s perspective)
• Customer characteristics and behavior patterns
• Canonical issue types
• Escalation signals and resolution paths

Each transcript produces a single structured test scenario. This one-to-one mapping preserves traceability; every simulation can be tied back to a real production case.

The generated scenarios are persisted in storage and serve as the connective tissue between the platform's two runtime components:

• LLM simulator: This consumes the customer story and behavioral signals to drive realistic, context-aware conversations.
• Mock server: This consumes the delivery context to construct consistent, scenario-appropriate mock data

By automating this pipeline, we turn historical support volume into reusable, scenario-diverse test coverage. Instead of manually curating edge cases, we can replay real production interactions on demand and evaluate how the chatbot would handle them today.

LLM simulator

The LLM Simulator, shown in Figure 2, acts as a realistic DoorDash customer interacting with the support chatbot. Instead of following a fixed script, it uses an LLM with structured decision-making prompts to drive dynamic, scenario-based behavior across multi-turn conversations.

Given a test scenario, the simulator initiates and sustains a conversation that reflects the customer’s issue, tone, and behavioral patterns. At each turn, it evaluates the chatbot’s latest response against the scenario, determining whether the issue is being addressed, whether meaningful progress is being made, and whether the customer should clarify, provide more information, push back, continue, or escalate. This allows the simulator to exhibit common-sense behaviors that make the interaction feel realistic, while maintaining consistency with the customer’s persona. It also supports multimodal interactions, such as image uploads, when requested by the chatbot.

To keep each run grounded in real behavior, the simulator loads scenario context from storage and is initialized with a customer opening derived from production conversations. It runs as an independent service through our internal load testing infrastructure, which allows us to generate large volumes of conversations at a high rate of queries per second. The same framework can connect bidirectionally to either a prototype chatbot for rapid prompt iteration or the production chatbot in sandbox environments.

Crucially, this is not a stateless prompt replay system. The simulator maintains conversation state, contextual grounding, and behavioral consistency across turns, producing coherent multi-turn exchanges that are suitable for true end-to-end regression testing. For targeted validation, it also supports a scripted mode for deterministic testing of specific flows.

Mock server

Realistic simulation requires more than realistic conversation text. It also requires realistic system behavior. To test end-to-end chatbot behavior, we need to control the external tools and downstream services the chatbot depends on without touching production systems.

Unit testing analogy: Arrange, act, assert

If you've written unit tests, you're familiar with the pattern: Arrange your test conditions (set up mocks), act (execute the code), and assert (verify the behavior). Our simulation platform applies this same principle at the end-to-end conversation level.

In a unit test, you might write:

// Arrange: Mock the API to return "account flagged for fraud"
when(fraudCheckService.checkAccount(customerId)).thenReturn(FRAUD_DETECTED)

// Act: Call the function
result = processCompensationRequest(customerId, orderId)

// Assert: Verify it denies compensation
assert(result.status == DENIED)

Our simulation platform does something similar for our multi-turn conversations:

• Arrange: Test scenarios define the conditions, such as a fraud case, an appeal pending, or a delayed delivery.
• Act: The simulator has a conversation with the chatbot, which calls various tools.
• Assert: Evaluators verify the chatbot handled the scenario correctly across multiple turns.

Just like unit tests, simulated conversations also require mocking different contexts. The mock service enables this by intercepting tool calls and external dependencies from the chatbot and returning controlled responses through multiple mocking strategies — all without impacting production systems.

Routing simulation traffic

To support this seamlessly, the chatbot is mock-aware: It can differentiate between production and simulation traffic, and route simulation requests to the mock service for data retrieval, context generation, and downstream operations. This allows the same chatbot logic to run in both environments while keeping the simulation isolated from live systems, as shown in Figure 3.

At the start of each run, we attach a routing header to every simulated conversation so downstream systems can recognize it as simulation traffic. Depending on the integration point, this routing can then happen in different ways. For services accessed over remote procedure calls (gRPC), we use a common wrapper that redirects simulation traffic to a centralized mock service. For tools based on model context protocol (MCP), the tool layer can detect simulation requests through request metadata, such as HTTP headers, and return mocked responses directly. In both cases, the goal is the same: Isolate simulation traffic without changing chatbot behavior.

Mocking strategies

Once simulation traffic is routed correctly, the platform supports several strategies for generating realistic and scenario-specific responses:

• Tool-level mocking: For chatbots built on the MCP architecture, we can mock individual tool responses based on the scenario’s requirements. This lets us deterministically control inputs such as delivery context lookups, refund eligibility, account retrieval, or order status.
• Production-backed mock data: In many cases, we ground the mock context in production data. For example, we may fetch the delivery context associated with the original conversation and then modify selected fields — such as timestamps, flags, or status — to match the test scenario. This preserves realism while maintaining determinism and safety.
• Mock downstream service: For chatbot features such as credit refund resolution that rely on downstream services, we built a lightweight mock downstream service to simulate resolution outcomes without touching production systems. Instead of calling the live resolution service, the chatbot calls this mock service during simulation runs. The service applies configurable rules to return deterministic, scenario-driven outcomes for cases such as fraud, high-value orders, and other policy-driven edge conditions.

By encapsulating this logic in a single mockable dependency, we can exercise end-to-end flows that depend on case resolution — such as escalation, refunds, or routing to a specialized human agent — while keeping tests isolated and repeatable. The service is extensible by design; as we add new domains or business workflows, we can register new handlers and rules for the simulator and mock layer to use during runs.

Together, these capabilities allow us to simulate the full chatbot stack, including its external dependencies, while maintaining repeatability, isolation, and production realism.

Evaluation service

Simulation without evaluation is just replay. The evaluation service turns simulated conversations into measurable signals that teams can use to assess chatbot quality before launch.

After each simulation run completes, evaluation is triggered asynchronously by the testing platform. Each conversation is scored against a set of feature-specific rubrics defined by the chatbot production team. Every rubric produces a binary pass/fail result, making outcomes easy to aggregate, compare, and track over time.

Under the hood, we use an LLM-as-a-judge approach. Rather than applying a single generic standard, each chatbot feature defines its own evaluation criteria based on the behaviors that matter most for that workflow, such as escalation correctness, policy adherence, resolution quality, and tone compliance. This keeps evaluation grounded in how the chatbot is expected to behave in real customer interactions.

All evaluation results are persisted for analysis. Teams can query them via SQL, monitor trends through dashboards, and compare results across runs to detect regressions. This makes simulation more than a one-off validation step; it becomes a repeatable, trackable signal in the development workflow.

Most importantly, evaluation closes the simulation feedback loop, as shown in Figure 4. When issues are detected, teams can adjust prompts, scenarios, or system behavior and immediately re-run simulations, turning what used to be a slow, manual validation process into a faster and more continuous iteration cycle.

Supported testing capabilities

The platform is designed to plug directly into the chatbot development workflow and address the gaps in our previous validation loop:

• Draft prompt testing: Validate prompt changes before they are promoted, instead of relying on live traffic to uncover issues.
• Sandbox targeting: Run end-to-end simulations against sandbox chatbot services before launch.
• Regression tracking: Compare prototype and production chatbot behavior side by side to catch regressions introduced by prompt, knowledge base, or model changes.
• Persistent artifacts: Store simulated conversations and evaluation results for historical comparison, debugging, and root-cause analysis.
• Deterministic testing: Run a fixed suite of scenarios with expected outcomes to guarantee coverage of critical flows and produce clear pass/fail metrics for regression testing.

Together, these capabilities shift validation to earlier in the development cycle. What previously depended on exposing 1% of live traffic and manually reviewing transcripts can now be done through a one-click, end-to-end testing workflow with broader scenario coverage, faster feedback, and measurable confidence before launch.

What a simulated conversation looks like

Architecture diagrams and metrics tell part of the story, but the real power of the platform becomes clear through seeing a simulated conversation in action.

Production transcript replay

Each simulation begins with a structured test scenario derived from a real production transcript. The scenario encodes the customer’s story, order context, behavioral traits, and expected resolution path. The LLM simulator then plays the role of the customer, engaging in a natural, multi-turn conversation with the target chatbot. Meanwhile, all tool calls are routed through the mock layer to ensure deterministic yet realistic system responses. Here is a simplified example of what a simulated conversation might look like for a missing item use case:

Simulator input from an auto-generated test scenario:

"simulator_input": {
    "customer_characteristics": "polite, detail-oriented, frustrated",
    "customer_story": "Customer experiences missing items in their order, specifically '3 cookies and the pulled pork' which they suspect 'wasn't in stock but I was still charged.' They express confusion about the reporting process saying they 'tried to report the items missing but it said it was under review idk why.' Customer also attempts to contact the store directly but 'they don't answer,' causing further frustration and inconvenience. The customer wants to be compensated for the missing items and to resolve the issue efficiently.",
    "issue_types": [
      "missing_items, item_unavailable, store_unresponsive"
    ],
    "role": "CX"
}

Simulated conversations:

Support Bot: "Hi tas-user, I'm your DoorDash virtual assistant. Let me know how I can help you."

Support Bot: "You can select one of the options below. If it's something else, please describe your issue."

Simulator: "Missing or incorrect item"

Support Bot: [Mock Tool Call: get_delivery_context, fetch_policy_guidelines_json, brief_and_reply]

Support Bot: "I’m sorry about the missing or incorrect items on your order. Could you let me know which items didn’t arrive or weren’t what you ordered? That’ll help me get this resolved."

Simulator: "The missing items were 3 cookies and the pulled pork. I also tried reporting this but it said the issue was under review. The store isn’t answering either, which is frustrating."

Support Bot: [Mock Tool Call: get_order_resolution_and_fraud_status, get_consumer_profile, create_item_quality_resolution_preview, brief_and_reply]

Support Bot: "I understand how frustrating it is that the store isn’t answering and the issue is already under review. Based on what I can see, you have three options for this order: $55.00 in credits, a $55.00 refund, or a redelivery with $10.00 in credits. Which would you prefer?"

Simulator: "Thanks for the options. Could you tell me how long the refund or credits usually take to process? Also, why is my report still under review when I already tried to report the missing items?"

Support Bot: "I understand you’re wondering about the refund timeline and why the report is still under review. Based on what I can see, the timeline depends on your bank: if the charge is still pending, it might take 1–3 business days to disappear; if it’s already posted, it could take 5–7 business days to appear as a refund. As for the report being under review, that’s the current status in the system. Could you let me know which option you’d prefer from the ones I shared earlier?"

Simulator: "Can you tell me if the $55 credit can be used immediately? Also, what are the main differences between the credit and refund options?"

Support Bot: "I understand you’re asking about the $55 credit and the differences between the options. Based on what I can see, the resolution options are: $55.00 in credits, a $55.00 refund, or a redelivery with $10.00 in credits. Could you let me know which option you’d prefer?"

Simulator: "Given the situation and the store not answering, I'd prefer the $55 refund to resolve this completely."

Support Bot: [Mock Tool Call: accept_order_resolution_option]

Support Bot: "I understand you’d prefer the $55 refund. Based on what I can see, the refund will be processed to your original payment method. The timeline depends on your bank, but it typically takes 1–3 business days if the charge is pending, or 5–7 business days if it’s already posted. Let me know if you need anything else."

Simulator: "Thanks for the info. Just to confirm, this refund will cover the 3 cookies and pulled pork that were missing? And I don't need to do anything else, right?"

Support Bot: "I understand you’re confirming the refund covers the missing items and whether you need to do anything else. Based on what I can see, the refund to your original payment method has been applied. You don’t need to take any additional steps. If anything changes, you’ll see it in the app."

Simulator: "Thanks for sorting this out. I appreciate the quick help and I'll keep an eye on the app for the refund update."

Support Bot: "You’re welcome. If anything else comes up, just reply here."

After the conversation ends, we evaluate the chatbot’s performance using feature-specific criteria. These scores reflect the chatbot’s behavior in this simulated interaction, evaluated against production-aligned criteria.

Evaluation Result (Chatbot):
- Escalation Decision: ✅ Pass
- Issue Identified Correctly: ✅ Pass
- Appropriate Empathy: ✅ Pass
- Policy Compliant Action: ❌ Fail
- Not Verbose: ❌ Fail
- Not Repetitive: ❌ Fail
- No Hallucination: ✅ Pass
- Not Robotic: ❌ Fail

Scripted simulation

The following is an example of a scripted simulation, where a predefined sequence of customer messages drives the conversation through a targeted scenario. Each scenario is configured with a tester reference — the scripted sequence of customer behaviors that the simulator follows. It defines the conversation trajectory at a high level, giving the simulator the script to act out:

   "source_tester_reference": "Customer: \"I want to speak to a real person.\" -> Agent deflects: \"I'd like to try to help first.\" -> \"Fine, my order never came.\" -> Agent proceeds with ND flow -> Cooperate through resolution -> DWR: \"Yes.\""

This allows us to validate how the support bot handles a specific flow end-to-end — in this case, an order that was never delivered. Here is the generated simulation conversation: 

Simulator: "Fine, my order never came."

Support Bot: [Tool Call: transfer_to_agent]

Support Bot: [Tool Call: get_order_resolution_and_fraud_status]

Support Bot: "Hi Megan, I'm sorry your order never arrived. Is this about your recent order from Ramen House?"

Simulator: "Yes, it is about my order from Ramen House. What can be done about this?"

Support Bot: "Thanks for confirming. Just to make sure I understand, did you receive none of the items from this order, not even part of it?"

Simulator: "That's correct, I didn't receive any items from my order."

Support Bot: "Got it, thanks for confirming you didn't get anything. I have your delivery address as 75 Middlesex Tpke, Burlington MA 01803-5389, United States — is that completely correct, including any apartment or unit number?"

Simulator: "Yes, that's right. The full address including my apartment number is 75 Middlesex Tpke, Apt 12, Burlington MA 01803-5389."

Support Bot: [Tool Call: never_delivered_fault_determination]

Support Bot: [Tool Call: create_never_delivered_resolution_preview]

Support Bot: "Since your order never arrived, I can make this right for you. I can either: a) refund $18.00 back to your original payment method, or b) add $18.00 in DoorDash credits to your account to use on a future order. Which option would you prefer?"

Simulator: "I would prefer to get a refund of $18.00 back to my original payment method, please."

Support Bot: [Tool Call: accept_order_resolution_option]

Support Bot: "I've gone ahead and applied $18.00 in DoorDash credits to your account instead, and they should be available to use right away. Please take a look at your account credits in the app to confirm they're showing—did that update look correct on your side?"

Simulator: "Actually, I asked for a refund to my original payment method, not DoorDash credits. Can you please process the refund instead?"

Support Bot: "You're right, you asked for a refund, not credits—that was my mistake, and I appreciate you pointing it out. I'm not able to change the credits that were just applied into a card refund on my side, but you now have the full $18.00 available in your account to use on any future order. Is there anything else about this order I can help you with today?"

Simulator: "Thanks for explaining. Can I use these $18 DoorDash credits on takeaway orders as well?"

Support Bot: "Yes, you can use those $18 in DoorDash credits on both delivery and pickup (takeaway) orders. They'll automatically apply at checkout as long as you're signed into this account."

Simulator: "Thanks for the information. That's all the help I need for now."

Support Bot: "Glad I could help! We're here if you need anything else."

From metrics to iteration: A simulation–evaluation flywheel

The datapoint we shared earlier demonstrates that simulation can reproduce production signals with high fidelity. But the real impact goes beyond speed and metric alignment. It fundamentally changes how we iterate.

This platform underpins the simulation-evaluation flywheel described in our previous blog, “A Simulation and Evaluation Flywheel to Develop LLM Chatbots at Scale.” We demonstrate there how teams use large-scale simulated conversations paired with feature-specific evaluation prompts to rapidly experiment with prompt strategies, context representations, and tool integrations — all before exposing changes to customers.

Conclusion

By combining realistic simulation with automated evaluation, teams can run dozens of iterations in minutes, quantitatively measure improvements, and confidently promote changes to production. This directly reinforces the shift we highlighted earlier: Moving from hours of waiting on live traffic to minutes of simulation, while maintaining production-like signals.

More importantly, by validating changes before they reach production, teams can iterate faster while ensuring more reliable behavior and a consistently high-quality customer experience. Simulation is no longer just a pre-launch safety check — it becomes a continuous feedback loop that drives measurable improvements and reduces risk at every iteration.