Using a Smart Clip Router to Benchmark Conversational Speech-To-Speech Agents 

At DoorDash, one challenge we face when deploying automated voice agents in support and food ordering roles is how to evaluate the overall quality of a model. Testing a model against pre-recorded conversations doesn’t tell you what happens when the conversation deviates from the recording, and having two agents simulate conversations doesn’t capture all the nuances of real human speech. To address these gaps, we built OrderUp-Bench (repository link coming soon), an evaluation benchmark of food-ordering scenarios with known target order carts. In our harness, a router agent simulating a customer is given access to a set of real human audio recordings that it can choose dynamically to interact with the order-taking agent and place an order.

The benchmark measures how accurately the agent captures the order within the scenario's clip sequence, how compliant it is with the order-taking policy given to it, and the total duration of each ordering session. In our experiments, we compared the performance of three commercial speech-to-speech models — GPT Realtime 2, Gemini Flash Live 3.1, and Grok Voice 1.0. We used Whisper with GPT-5.5 as a cascaded baseline. All settings used the same prompt, which instructed the agent to take the customer's order and follow the same rules for seeking options and follow-ups.

Our key finding: Turn-level accuracy and compliance are weak proxies for expected overall success rate. As shown in Figure 1, GPT Realtime 2 had the highest clean pass rate of any model we tested (66.4%, vs. 28.5–62.9% for the others), meaning it most often ended with exactly the right cart without the simulated customer having to step in and correct it. It was also the model most likely to fail silently (8.9% of tasks, vs. 2.0–4.0% for the others), giving the customer no signal that anything went wrong. Once we account for failures that a customer could plausibly catch and correct mid-conversation, the 37.9-point spread in overall success rate shrinks to under 5 points (79.6–84.4%). Mistakes that are verbalized and then corrected still carry a penalty, but their effect shows up by increasing the conversation's duration, with recovered runs taking about 35% longer than clean passes. This better reflects the trade-offs in real production settings where customers may take time to correct an agent, which produces a degraded customer experience rather than a failed result.

From our experience with voice agents, we have observed that improving metrics focused on individual conversation turns like accuracy or response latency doesn’t always move the needle on downstream business metrics such as success rate and task completion time. An agent that records the customer’s order perfectly and responds quickly can still fail to drive the conversation in the right way. Before rolling out a new agent version, we would often be unable to answer questions like:

• If we change the agent to provide more succinct responses, do conversations actually complete faster?
• How often are customers working harder to interact with the agent because they're unsure if they were heard correctly?
• Is the agent asking too many unnecessary clarifying questions? What happens if we direct it to make informed guesses more often?

To try and answer any of these questions in an offline setting, we needed to model how a customer might respond to the way a voice agent phrases its responses and questions. Informed by real customer interactions, we designed a smart customer clip router which can play clips extracted from human audio sessions to answer follow-up questions, offer corrections, and repeat or rephrase orders in response to the agent.

For each sample scenario, we record whether each agent trajectory was:

• Correct: The agent was fully compliant with its standard operating procedure (SOP) and recorded the expected order by the end of the session, or
• Recovered: The agent goes off-policy at some point — such as asking a superfluous question or recording the wrong order — but the customer corrects the agent and steers the trajectory back to the expected result, or 
• Recoverable: The agent recorded the wrong order at the end, but clearly conveyed what they were doing in a way that, in theory, a customer could have heard and corrected the order, even if the customer in the sample didn't, or
• Unrecoverable: The agent silently recorded a partially incorrect order. A customer would have no opportunity to correct it.

Effective success rate credits a fraction of the recoverable failures as if a human had caught and fixed them. Figure 2 reports two additional anchors beyond the 50% assumed correction rate, using a 100% and a 0% correction rate to show the extreme ranges between motivated and inattentive users. GPT Realtime has a significant clean pass margin over the other models, measuring both for a correct output and full policy compliance. But the results appear to be much closer when all recovered runs are counted as correct but assuming all recoverable runs failed. Many of the compliance failures which penalize the clean run rate can be harmless issues such as asking an unexpected question about non-required options. A real customer would be able to answer the question and the extra time would be captured by the duration metric; consequently, the effective success rate @X metrics would be more representative of a real deployment success rate.

Effective duration applies the same idea to timing; it's the average time cost per task if recoverable failures were corrected during the conversation, with recoverable runs scaled up by a time penalty relative to a clean pass on the same task. Averaging over a set of tasks where repeated evaluations produced at least one recovered trajectory and one correct one, we derived a 1.37X time scaling factor which we use to estimate how long a recovery could take.  A model is penalized more for an unrecoverable failure than a recoverable one, and is penalized in duration for producing a wrong run compared to a correct one. Figure 3 shows how the effective duration for each model compares against the average time spent by the model per turn. Driving down response latency doesn’t always translate to a faster interaction for the customer, as we can see by comparing GPT against Gemini. Any edge Gemini has in response latency is eaten up in extra turns answering superfluous questions or correcting mistakes, demonstrating how the lower clean pass rate from Figure 1 compared to GPT manifests itself as extra duration. See our technical report [TODO: add link] for a full breakdown of failure severity and the recovery-time-factor methodology behind effective duration.

Our evaluation harness

We built an evaluation harness to measure how well a voice agent might perform in a real production setting. This harness can reproduce problems that make the audio domain uniquely challenging. The evaluation framework includes such features as:

• An evaluator that streams audio to an examinee agent, which then calls a set of tools to record user intent in a structured form before streaming audio back to the evaluator to ask clarifying or required follow-up questions.
• The evaluator agent includes a smart clip router, which plays audio clips collected from real human conversations.
• The agent responds to the examinee by choosing an audio clip from a pre-defined pool of clips specific to the given scenario.
• We record the conversation’s length and compare its intents against an expected result.

Instead of just playing the clips in a pre-defined sequence, the clip router has a decision tree it can use when the agent doesn't follow the exact trajectory of the original human conversation. The router uses a goal state to try to choose a clip that best matches what a human would say to reach that goal in response to the agent's questions. Figure 4 demonstrates some sample flows through different conversation trajectories, with branches for common agent failure modes like missing a required question, asking a superfluous one, or duplicating an item.

This design allows us to:

• Minimize the variability introduced by letting the evaluator give free-form responses;
• Test the examinee agent’s response to known challenging speech patterns, such as accents, disfluencies, and mispronunciations;
• Stress the examinee’s ability to comply with its SOPs over multiple turns while extracting the correct information from the evaluator; and
• Most importantly, test the combined capability of multi-turn response chains to challenging speech. When the speech is genuinely hard to understand, we determine whether the agent can recover the conversation with a targeted clarifying question, extending it by an extra turn, or whether it instead silently makes its best guess, risking recording the wrong intent. Ultimately, we seek to determine the accuracy versus time tradeoff for different models and different prompting biases.

Scenario design

In this evaluation benchmark, scenarios consist of conversations in which the evaluator is ordering food from the examinee. The examinee has access to a menu with exact names for items and the options that apply to them. Additionally, the menu indicates which options are required, meaning that the order can’t be completed without collecting a response to a required option question.

In turn, the evaluator has access to a set of clips in which the simulated customer can:

• Refer to items with a name other than their canonical name, or which can refer to two or more possible menu items.
• Omit required options.
• Ask for multiple items in a single utterance.
• Correct or modify previous requests.

In response to each clip, the examinee has to make a judgement call about whether to ask a disambiguating or option question or whether to instead make a best guess and move the conversation forward. If the examinee asks another question, the evaluator is ready with a clip to answer it,  but it won’t offer the extra information on its own; an examinee that speeds through to the conversation’s end without follow-up questions risks missing a required option or recording an incorrect item name, which subjects it to penalties related to  accuracy.

Modeling customer trajectories

A good agent goes beyond understanding and processing a user’s intent to drive the conversation to its next defined step. It also communicates understanding and acknowledgement, asking for clarification when a genuine acknowledgement is not warranted. Often a user-agent conversation devolves into a failure when the user repeats their intent after an ambiguous acknowledgement. Silent duplication of recorded intents or confusing responses lead to increasing divergence between the user’s understanding of what the agent is saying and what the agent is actually doing, resulting in frustration and an inevitable request to speak to a human.

To model this, the evaluator has an additional task: to record whether the agent acknowledged every item it has been asked for during the conversation. When an intent isn’t acknowledged, the evaluator automatically repeats it before moving on to additional items or closing the conversation, as demonstrated in Figure 5. This design forces the evaluator to maintain a model of its partner’s mental state so that it can actively work to try to close any gaps between what it tried to say and what it thinks was understood.

Given that audio clips are collected from a single conversation, if a sample trajectory diverges too far from the original conversation, there may not be a clip that would correspond exactly to the examinee’s response. Because of our scenario design, an ideal agent would follow the expected trajectory in the exact sequence as the original conversation. In practice, however, we also care about whether the examinee/evaluator pair can return to the expected trajectory after a divergence and record the correct final customer intent. In general:

To support this hierarchy, we generated a few universal fallback clips via text-to-speech so that the clip router could address common failure modes. Fallback clips like “yes,” “no,” “just the defaults,” or “I only want one” can be used to address some agent responses which don’t have a close analog in the original recording. The examinee can use these to handle common agent failure modes like asking about a non-required option or adding additional quantity. We found that using these universal clips instead of allowing unconstrained speech generation strikes a good balance. It allows a common set of trajectory divergences to recover gracefully without risking generating speech when playing a recorded human clip is desired.

Dataset Creation

To ensure our dataset represented the kinds of speech our agent to which our agent would be exposed, we collected data from different devices with variations across accents, background noise, and disfluencies, including such things as pauses, umms and ahhs, and  mid-sentence corrections. Participants were not given an explicit script on how to order, leading to variation in order grouping, such as asking for an item with all options, offering options as a question response, and changing or rephrasing items after an initial ask.

What's next

While our harness allows us to reliably surface complex agent failures that we couldn't otherwise reproduce, we don't yet have a single numerical score that could confidently say Agent A performs better than Agent B. In order to construct a binary pass/fail decision that could promote a better agent, we still need:

• Weight calibration: The assumed 50% recovery rate in the effective success rate calculation was chosen for simplicity, not based on measurement. Additionally, different failure modes have different likelihoods of customer correction. A fully representative effective success rate would make use of calibrated recovery rates for different error types derived from data. Ultimately, we would want to calibrate our offline success and duration metrics against online success and acceptance metrics to understand how to optimize an agent for our product goals.
• Further variance reduction: Because our evaluation harness deals directly with streaming audio, its results are sensitive to differences in streaming latency from run to run. Our evaluation harness streams sampled background noise in between selected clips; any difference in the time taken to select a clip results in the examinee experiencing a different stretch of background noise. More fundamentally, the semantic meaning of a response can change based on when it’s perceived by the listener. If a speaker asks two questions with time between them, and their partner responds to the first question, the meaning can change based on whether that response is heard before or after the second question. Of the 248 samples evaluated across the four models, when the same model is run three times on the same sample, its correctness label stays consistent only 53.3% of the time. While variation averages out over three sweeps, it remains difficult to say whether any given scenario is improved when testing a new agent.

Driving model improvements

Our evaluation framework allows us to reproduce complex failure modes for voice agents. We hope the release of this benchmark and framework will drive further development of native speech-to-speech models to increase their compliance and robustness in challenging deployment domains.