How we learned to trust our AI code reviewer at DoorDash

How we built a measurement layer that tells us where, why, and how much to trust an agentic code reviewer, and why a single metric never could.

In a nutshell

DoorDash recently described how we built a production code review agent that engineers actually listen to. That post covered the product and architecture choices behind the agent: a lead scout, deep reviewers, focused context, a precision-over-recall posture, and a model-agnostic design. DashBench is our measurement layer behind it. DashBench replays historical PRs and evaluates whether systems surface real, human-actionable findings instead of merely producing plausible comments. That distinction matters because convenient signals like acceptance rate, thumbs-up feedback, or a single aggregate score can make a reviewer look useful while hiding where it fails: missing important issues, over-indexing on easy comments, or trading recall for precision in ways product metrics do not expose.

The headline result: on the 105-case report, DoorDash's production code reviewer (Claude Sonnet 4.6 high scout + Claude Opus 4.8 high reviewer) found 504 real findings with 53.6% weighted recall, compared with 164 real findings and 30.7% weighted recall for a no-scout GPT 5.5 high baseline. The broader model-mix view is more interesting than a single winner: Kimi K2.6 scout + Claude Fable 5 reviewer led weighted recall and F1, Composer 2.5 scout + GPT 5.5 medium reviewer led weighted precision, and the single-pass baselines stayed much cheaper.

Note: Weighted metrics give more weight to higher-severity issues (critical = 4, high = 2, medium = 1, low = 0.5).

Why the obvious signals lie

The tempting way to measure a code reviewer – and the main metric most major code review tools use today – is to watch what happens in production: do authors accept its comments, do they act on them? That signal is real. But it’s built on a foundation of shaky assumptions and incomplete information. In the language of confusion matrices, acceptance only ever populates two of the four cells: a comment the author accepts is booked as a true positive (TP), one they reject as a false positive (FP). Both bookings assume the human's call is ground truth – that engineers are infallible, or at least they’re wrong rarely and randomly enough that the errors don't matter in aggregate. As we'll see, that's a bad bet. 

The other two cells stay empty, and that's the real cost. Acceptance can't record false negatives (bugs the reviewer missed) or true negatives (clean code where silence was the right answer). It tells you something happened; it doesn't tell you whether the review was right, whether silence was justified, or what gets missed. DashBench handles that differently: after adjudication, missed real issue clusters count as false negatives for benchmark scoring, even though production acceptance telemetry cannot observe them.

The other problem alluded to above is that human acceptance is a useful signal, but not ground truth. Authors accept and reject comments for product and workflow reasons: timing, PR urgency, ownership context, how invasive the fix is, or whether the issue was already handled another way. That makes acceptance a valuable product telemetry, but a weak benchmark label by itself. In disagreement audits, human review and agentic verification both surfaced mistakes; the point was not to declare one side right, but to separate "was accepted" from "was real." The most available metric can still point at the wrong target.

That’s the reason we built DashBench. If we lean on the convenient metric to build and improve a system our engineers now depend on, we would optimize confidently towards a foundationally flawed objective. The fix isn't a better single metric. It's a benchmark that triangulates across several flawed signals and never relies on any one of them as the ground truth. 

From a production architecture to a clean place to measure

Our production reviewer uses a staged architecture that separates noticing from verifying. A lead scout reads the change and flags suspicious areas. Deep reviewers then investigate the strongest leads, confirm whether each concern holds up, and drop the claims that don't survive scrutiny. It mirrors how strong human reviewers work: form hunches, aim attention at the risky parts of the diff, then validate before asking an author to do anything.

That design happens to give us a clean evaluation surface. Because the workflow is model-agnostic, every component is a variable we can hold fixed or change on its own: the scout model, the reviewer model, the context policy, the tool policy, the runtime budget. We can replay the same frozen PRs through the production agent, through staged variants with different model assignments, and through plain single-pass reviewers, and the thing being measured stays constant.

So the useful question stops being "which model is best?" It becomes: for a given architecture, context policy, tool policy, runtime budget, and model mix, what tradeoff between coverage, precision, cost, and latency do we actually get, and where does it fail? "Best" is meaningless until you say best at what, on which cases, at what cost.

How we built the benchmark

The important part is the loop, not any single box. New cases and model changes keep entering the same replay-and-score path, while disagreement review keeps the benchmark honest when human feedback, production behavior, and agentic judgment do not line up. In that pass, we manually inspect the evidence, decide whether the finding is real, and feed the resolved cases back into judge calibration. The judge itself is LLM-based, but DashBench treats it as a calibrated signal, not ground truth.

The cases.

DashBench started from roughly 1,000 raw PR candidates and was curated toward cases that stress different review behaviors, such as tricky diffs, noisy review histories, and varied severity outcomes. The post uses the 105-case valid report to analyze staged-versus-single-pass systems, scout/reviewer model choices, and severity-weighted quality on one consistent eval slice.

• Historical PRs with real review findings and enough surrounding context to faithfully replay the review.
• Benign PRs with close to zero real findings, so the benchmark can measure restraint and catch false positives; a reviewer that's loud on clean code is its own failure mode.
• PRs that were later reverted or hotfixed, so the benchmark can test whether a system catches genuine code-level regressions before merge.

The labels. 

This is where most benchmarks cut a corner we refused to. Preparing labels took more than importing human feedback. We had the engineers who wrote the PRs annotate candidate findings, then compared three sources against each other: the human annotations, the original candidate findings, and an agentic judge. Human feedback was valuable but frequently wrong: reviewers missed valid issues, accepted weak claims, or read a historical PR's context differently than the next reviewer would. Where the three sources disagreed, we re-reviewed the evidence by hand and resolved it, and those resolved cases became calibration data for the judge.

The result is a benchmark whose ground truth doesn't rest on any single fallible source. Human judgment, production feedback, and agentic evaluation each contribute; none is treated as infallible. 

The execution environment. 

Within a comparison, every configuration runs against the same frozen case set, context track, prompt and skill-pack selection, normalized output contract, and grading pipeline. The harness is part of what we are measuring: profiles differ in runner kind, scout/reviewer models, tool surface, context mechanics, and provider limits. Some run from a prepared repo with read/grep-style tools, some receive bounded repo context in the prompt, and some disable search or judge steps in the review run. Cost, latency, timeouts, retries, and failures are recorded in the run artifacts and suite summaries rather than treated as controlled constants. The point is repeatability: when a number moves, we want to know whether it moved because the model or harness changed, not because the eval drifted.

The Report: What DashBench actually shows

We use DashBench to answer four questions that production telemetry alone cannot answer. First, does staging actually buy coverage? Second, which scout/reviewer model choices change the cost, precision, and recall frontier? Third, do the headline rankings survive when we split findings by severity? Fourth, when do staged scout/reviewer setups beat stronger single-pass reviewers, and what do they give up to do it?

Staging benefits

Staging buys coverage, and the price is visible. On the 105-case report, DoorDash's production code reviewer (Claude Sonnet 4.6 high scout + Claude Opus 4.8 high reviewer) found 504 real findings with 53.6% weighted recall, compared with 164 real findings and 30.7% weighted recall for the no-scout GPT 5.5 high baseline. Weighted precision stayed in the same neighborhood, 87.0% versus 84.1%, but the production reviewer cost more per PR and took longer. That is exactly the kind of result we want DashBench to surface: not a trophy, but a measured tradeoff.

Scout/reviewer model choice

The model-mix comparison is where the benchmark becomes most useful: we swap which model scouts and which model reviews while keeping the benchmark fixed. No configuration dominates every axis. Kimi K2.6 scout + Claude Fable 5 reviewer had the strongest weighted recall and F1 on the 105-case valid subset, at 65.2% weighted recall and 75.3% weighted F1. Composer 2.5 scout + GPT 5.5 medium reviewer had the strongest weighted precision at 92.2%, but with materially lower recall. The no-scout baselines were cheaper, while Kimi K2.6 scout + Claude Opus 4.8 high reviewer was a lower-cost staged alternative with materially lower recall.

Impact of severity of findings on evaluation

Severity changes the story again. Across the 105-case valid subset, the union of adjudicated real findings contains 40 critical, 136 high, 271 medium, and 385 low clusters. Kimi K2.6 scout + Claude Fable 5 reviewer was strongest on critical, high, and medium coverage, while Claude Sonnet 4.6 high scout + Claude Opus 4.8 high reviewer covered slightly more of the low-severity tail. The no-scout GPT 5.5 high row was cheap and still useful on high-severity issues, but weaker on overall coverage. This is why the weighted score exists: it gives critical and high-severity misses more pull than low-severity misses, while still letting us inspect the full severity split.

The severity view shows why one score is not enough: configurations that look close overall miss very different kinds of issues.

Influence of model quality and size on benchmark results

The model-mix data says the same thing in numbers: Kimi K2.6 scout + Claude Fable 5 reviewer leads weighted recall and F1, Composer 2.5 scout + GPT 5.5 medium reviewer leads weighted precision, and the single-pass no-scout baselines stay cheaper while giving up coverage. Purpose-built staging does not make a weaker model magically best at everything; it changes the shape of the tradeoff. Scouts improve breadth when the reviewer can verify aggressively. Stricter reviewer configurations improve precision when the business goal is to reduce noise. Single-pass no-scout baselines are a useful lower-cost floor, but they leave coverage on the table. The result is not that one model wins. The result is that no single configuration dominates, and being able to say exactly that is the point.

Lessons from benchmarking PR reviewing

Trust must be earned. No single signal is a reliable source of truth for real-world benchmarking. Human labels, production acceptance, and agentic judgment each contributed, and each was wrong often enough that treating any one as ground truth would have quietly poisoned the scores. The benchmark is only as good as its refusal to trust any fallible input alone.

Humans don't scale. Human attention for labeling degrades fast on complex tasks, and faster across many of them, even when the labeler is the same engineer who wrote or reviewed the original PR. AI judging is what makes labeling at this scale tractable at all, but only under the constraints in the next two lessons.

Variance is a feature, not a bug. LLMs are non-deterministic, so multiple runs of the same agent surface additional valid findings, meaning a single run understates an agent's real coverage, and you have to run repeatedly and aggregate to score it honestly. The same stochasticity applies to the judge: deterministic matching is stable but misses semantic equivalence, while LLM judges reason more richly but need calibration, audit sets, and stable rubrics to stay trustworthy. The whole measurement stack is stochastic; the work is designing for that instead of pretending it away.

When one metric can't work, look to many. A single score is misleading by construction. Weighted precision, weighted recall, weighted F1, not-real findings, high/critical recall, latency, and cost all move independently; a system can win one and lose another in the same run, so "better" is meaningless until you say better at what.

What's next

DashBench exists so we can keep improving the reviewer with evidence instead of anecdotes. The goal was never a static leaderboard; it's a feedback loop where every material change to model, prompt, context, tool use, workflow, or runtime budget gets tested the same real PR-review cases before it reaches an engineer.

The next stretch is continuous benchmarking:

• New models and harnesses enter the benchmark quickly as they ship, including additional agent harnesses we have planned for this stage.
• Stale PR cases retire from the dataset while new cases are added continuously, so the benchmark tracks the codebase instead of drifting away from it.
• We're moving from a single judge to an agentic jury, to mitigate bias between individual judge models.
• We're benchmarking against external code-review solutions, not just internal variants.

And we're starting to benchmark coding agents, not just reviewers, on a real enterprise codebase with real tasks and features. More on that later.

The honest summary is this: making an AI system useful is only half the work. The harder part is knowing where it fails, why it fails, and whether the next change made it better or just different. Most of the field is still measuring the wrong unit with the wrong number. DashBench is our attempt to measure the work that actually matters: real findings, on real PRs, with the tradeoffs left visible. If that is the kind of problem you want to work on, we'd like to talk.

Appendix

Parking lot for detail that backs the post without slowing the body down. The Report section keeps the reader on the tradeoffs; this section keeps the audit trail.

A. Execution environment (full spec). Within a comparison, each configuration runs on the same dataset cut and produces the same structured finding contract: severity, evidence, and file anchors. The harness itself is part of what DashBench measures, so profiles can differ in runner kind, scout/reviewer split, model choice, tool surface, context mechanics, and provider limits. Scoring then uses the same matching path for that comparison, with deterministic matching where possible and agentic judging where semantic matching is needed. The point is not to erase system differences; it is to make those differences explicit enough that cost, latency, and quality move for interpretable reasons.

B. Dataset and eval-set sizes. DashBench started from roughly 1,000 raw PR candidates, then narrowed to cases that could be replayed and adjudicated. The post uses the 105-case valid report for the main model-mix and staged-versus-single-pass analysis. The severity denominator in that 105-case cut is the union of real finding clusters: 40 critical, 136 high, 271 medium, and 385 low. Those are finding clusters, not PR counts; one PR can contribute more than one cluster.

C. Full configuration metrics. The body shows the narrow version because that is the readable version. The full metrics are below for auditability, split so they stay on the page.

D. Severity breakdown (full). The severity view is the clearest reason the weighted headline metric exists. Critical and high misses are not interchangeable with low-severity misses, and the configurations move differently by tier.

Recall by severity

Precision by severity

Task intents are multi-label, so counts sum to more than 105.

Task intents

PR change size

Verifiability

Product domains

Top 20 domains are shown.

Impact

Impact labels are multi-label, so counts sum to more than 105.