Building an Automated Improvement Harness for an AI Support Agent

Engineers using a machine learning (ML) platform need help debugging failed jobs, deploying models, and understanding platform behavior. Answering these questions often requires piecing together documentation, logs, and live system state, along with knowledge held by platform engineers.

DoorDash built an internal artificial intelligence (AI) support agent catering to engineers using the ML platform. From one conversational interface, it can retrieve guidance, inspect live state, run diagnostic tools, and suggest next steps.

For example, an engineer received a warning that a test model deployment would be scaled down because it had little traffic. Its calling service was still under development, so the engineer asked how to temporarily exempt it from cleanup. The agent explained the cleanup policy but could not provide the exemption procedure. A platform engineer supplied the missing guidance.

That gap exposes the harder engineering problem: building a system that repeatedly finds unresolved cases, classifies their causes, applies appropriate fixes, and measures whether support improves.

Creating the support agent marks only the beginning of the process. The systems the agent supports are constantly evolving, which requires the agent to evolve alongside them to remain viable. To ensure the support agent continues working as intended, we created an automated improvement harness to constantly evaluate the agent and keep it up to date so that it remains a helpful tool for our engineers. 

The resulting harness is benchmark-driven, automated, and aligned with the judgment of platform domain experts. Together, these properties turn usage into a controlled improvement signal without assuming that the agent learns on its own. In this post, we explain how we benchmark support outcomes, turn recurring gaps into targeted improvements, and align automated workflows with our team’s judgment.

Choose an outcome metric, then make failures actionable

An improvement harness needs an outcome metric that reflects the agent’s job. For our support agent, success means fully resolving engineers’ questions. Reactions and broad quality scores are useful signals, but they do not tell us whether a response contained the information that resolved an issue.

Because of this, we built a live benchmark around resolution. For each eligible support case, it identifies the material facts behind the eventual resolution and asks whether the agent supplied them:

• Full: The response supplied the material information required to resolve the question.
• Partial: The response advanced the investigation but omitted something material.
• Missed: The response did not materially advance the resolution.

Our top-line metric is the full resolution rate, the share of evaluated questions the agent fully resolves. Other agents may optimize for verified task completion, diagnostic accuracy, successful handoff, or time to resolution. The reusable principle is to choose a measurable outcome and connect observed gaps to fixes that can be tested against it. Figure 1 shows how the harness connects observed gaps, targeted fixes, and verification to the chosen outcome metric.

In this case, the benchmark rated the response Partial. Explaining why cleanup happens did not answer how to prevent it temporarily. The missing exemption procedure gave the harness a specific gap to address. A human-provided resolution can include temporary workarounds or incorrect assumptions. Before adding facts from a support thread to shared documentation, the harness checks that they apply beyond the incident and verifies them against authoritative sources.

Let usage reveal gaps, then classify each gap by its remedy

Greater usage exposes a broader, more current sample of the agent’s behavior, including such things as new question shapes, repeated failure modes, and human-confirmed resolutions that contain facts or diagnostic paths the agent missed. Usage supplies evidence; the harness turns selected evidence into improvements.

That conversion begins by classifying each unresolved response by the remedy it needs, as shown in Figure 2:

• Knowledge gap: A durable resolution fact is absent from authoritative documentation.
• Capability gap: The agent lacks the tool, data source, or workflow needed to obtain evidence.
• Utilization gap: The capability exists, but routing, retrieval, invocation, or synthesis failed to use it.
• Correctness gap: Documentation exists, but a relevant claim is stale, ambiguous, or contradicted by the underlying system.

This classification prevents attractive but ineffective fixes. For instance, documentation cannot repair a failed live lookup, a new tool is unnecessary when routing fails to select an existing one, and copying an old workaround could make the knowledge base worse. The category should identify the required system change.

A completeness loop adds missing facts to shared documentation, while a correctness loop repairs stale claims. Capability and utilization gaps require changes to tools or agent behavior

Keep one authoritative knowledge source through two loops

Domain teams keep designated upstream documentation as the source of truth, while the agent serves a synchronized snapshot. This prevents the agent from becoming an independent knowledge authority that drifts from the documentation that engineers use. Humans and the agent rely on the same durable explanations.

Figure 3 shows how two independent loops keep shared documentation complete and correct:

• Completeness loop: For partial or missed responses, the workflow identifies missing facts, checks them against existing documentation and authoritative evidence, and updates the owning document. In the cleanup case, the update documented how to add a deployment to the exemption list, record an owner and reason, and remove the exemption once real traffic starts. Some cases need no documentation change because the advice is incident-specific, unsupported, already documented, or requires a capability fix.
• Correctness loop: Support outcomes reveal only the topics engineers happen to ask about. An independent correctness loop starts from the documentation: it samples concrete claims and checks them against authoritative implementation or configuration evidence. When evidence contradicts a claim, the workflow applies a focused correction through the same source-of-truth path.

Completeness asks: “What did the human resolver know that the agent could not find?” Correctness asks: “Is what the agent found still true?” Retrieval quality cannot compensate for a corpus that is incomplete or wrong.

Automate recurrence with versioned workflows and deterministic scripts

Scheduled workflows run the recurring benchmarking, knowledge maintenance, synchronization, and replay work. Each combines versioned skills with deterministic scripts that define the process, validation, and expected outputs. At DoorDash, we automate these workflows through cron-triggered playbooks on Flux, our cloud-based agent platform. 

Deterministic scripts control execution through selecting eligible inputs, validating structured outputs and evidence, bounding retries and timeouts, detecting duplicates and no-op cases, and returning labeled outcomes. Language models handle bounded interpretation, such as extracting candidate facts, classifying a gap, or comparing a claim with supplied evidence. Free-form model prose never controls workflow state.

This boundary lets the harness run frequently without redefining its controls. It gathers evidence, applies targeted changes, and leaves replayable artifacts; domain experts tune the governing rules between runs.

Align the harness with team judgment, then verify behavior

The first version of a workflow rarely captures every distinction the team cares about. A fact checker may select a weak source, a completeness workflow may mistake an incident-specific workaround for durable knowledge, or a classifier may recommend a new capability when an existing one was simply underused.

Platform domain experts compare workflow outputs with team expectations, flagging weak evidence, unhelpful findings, and recurring false positives. When a pattern recurs, we encode the lesson into the skill, script, schema, example, fixture, or validation gate that governs future runs. Human feedback improves the workflow contract without becoming a permanent manual gate.

Alignment is not a prompt written once or an attempt to reproduce one person’s wording. It is the repeated conversion of experienced judgment into explicit, replayable rules.

A change matters only if it improves the agent’s behavior. Documentation may be correct but unretrievable. Retrieval may succeed while the response omits the fact. Or a capability may fail to activate for the original question shape.

The harness replays the original case after a change reaches the knowledge snapshot or deployed agent. It checks whether the agent can retrieve and use the required facts. In a later replay of the cleanup question, the agent supplied the exemption procedure and was rated Full. The live benchmark tracks whether accumulated changes improve the full resolution rate across cases.

Conclusion

Launching the support agent was only the starting point. The harness we created around it made up the larger engineering lift. It provides resolution-grounded measurement, remedy-oriented classification, team-owned knowledge, independent completeness and correctness loops, scheduled workflows, deterministic controls, and continued alignment with domain judgment.

Over the course of the first couple of months, the harness has increased the full resolution rate by over 200% while catching some 100 knowledge base corrections and patches.

A support agent only works if it does what its name suggests by providing support for users. Otherwise, it becomes a burden rather than a tool. 

Greater usage gives that harness more representative questions, failure modes, and human-confirmed resolutions. The agent improves only when the system turns those signals into targeted changes and verifies the resulting behavior. The support queue becomes more than work to clear; it becomes a controlled signal for improving the agent, its capabilities, and the knowledge shared by humans and AI.

Acknowledgements: Thanks to Claudia Sun, Bowen Dan, Petter Ivmark and Mert Sanver for contributing to this post and harness.

Thanks to Artem Petrov, Emma Dang, Emmanuel Khayat, Francesco Peri, Haifeng Geng,  Norman Rosner, Sertan Şentürk, Sowmya Yellapragada, Steffen Klempau, Steve Hoffman, Taoxin Jiang, Teer Ying, Yong Go, Zhe Liu for the contribution to the development of the agent. 

Thanks to Santosh Banda and Jeffrey Hwang for their help in building resilient flux automations.