DoorDash’s experimentation platform manages over 60,000 feature flags across roughly 623 repositories. We identified over a thousand stale feature flags; in other words, the feature they once gated is fully shipped, but the flag and its branching logic still sit in the codebase as dead weight. Cleaning just one up manually takes an engineer one to two hours. With about 2,300 new flags being created every month, the backlog only grows.
So we built a multi-agent LLM system that cleans up stale flags end-to-end from a Jira ticket to a merge-ready pull request (PR) with no human in the loop except an engineer confirming the target value and the PR landing. Of the 50 most recent stale flags it processed, the system produced usable PRs for 45 of them at an average cost of $4.79 and 13.8 minutes per flag, saving countless human engineering hours. The system we built not only had to be effective, but its reliability boundary needed to be completeness, not correctness; if the agent failed, it had to do so safely without making any breaking code changes.
This post covers why off-the-shelf tools couldn't accomplish this in our codebase, how we designed the system, and what we learned during its deployment. Note: This work was peer-reviewed and accepted for the ICSME 2026 industry track.
Why stale feature flags are worth automating away
Feature flags are a cornerstone of modern software delivery. They let us roll out features gradually, run A/B experiments, and flip kill switches when something goes wrong. But every flag has a lifecycle and once a feature is fully rolled out, its flag becomes stale. stale feature flags are often left as technical debt because an engineer’s time has more value building new features instead of cleaning them.
Its conditional logic no longer serves a purpose, but it doesn't just sit there harmlessly:
- Stale flags can remain in the codebase for years. Prior research found that roughly 25% of feature flags persist for more than eight years in open-source projects; practitioners consistently report that cleanup is burdensome and repeatedly deferred.
- A stale flag is technical debt. Every stale flag adds a branch, an edge case, and a bit more to read and reason about, which can be especially painful during an incident.
- Stale flags can be an operational risk. A stale flag preserves a dormant code path that can be reactivated by accident. In one well-documented case, a trading firm repurposed a stale flag during a deployment, triggering $460 million in unintended trades in just 45 minutes.
Multiply tens of thousands of flags and a four-figure monthly creation rate and it becomes clear that manual cleanup simply cannot keep pace. Clearing our stale backlog by hand would cost thousands of engineering hours that would be better spent building.
Dynamic values
Feature flags at DoorDash are called dynamic values (DVs). They are powered by our experimentation platform. Throughout this post, the term DV is synonymous with feature flag.
A DV’s lifecycle follows the pattern creation → experimentation → rollout → stale → removal. We classify a DV as stale when it meets all of the following criteria:
- It hasn't been modified in 90 days.
- It's still referenced in code.
- It's not already in an end-of-life status (archived or retired).
- It's not on an exclusion list.
A daily cron job files a STALEDV Jira ticket for each identified stale DV and assigns it to the DV's creator. This marks the front of the queue from which our cleanup system works.
Agentic workflow vs. AST-based cleanup tools
The best-known automated approach to flag cleanup is Piranha and its polyglot successor, which simplify flag-gated conditionals by matching patterns in the abstract syntax tree (AST). It works well when a flag is read through a direct API call that a rewrite rule can pattern-match.
But that's not how DVs are accessed at DoorDash. Our services use a three-layer, dependency-injected wrapper pattern:
| // Layer 1: RuntimeKeys.kt - the DV name as a string constant object RuntimeKeys { const val ENABLE_X = "enable_feature_x" } // Layer 2: FeatureFlags.kt - a DI wrapper that calls the DV client @Singleton class FeatureFlags @Inject constructor( private val dvClient: DynamicValuesClient ) { fun shouldEnableX(userId: String): Boolean = dvClient.getBoolean(ENABLE_X, withContext("user_id" to userId), false) } // Layer 3: business logic - uses the flag via dependency injection class MyService @Inject constructor( private val featureFlags: FeatureFlags ) { fun process(userId: String) { if (featureFlags.shouldEnableX(userId)) handleNewPath() else handleOldPath() } } |
This is a clean separation of concerns, but it means the DV name, the API call, and the conditional logic each live in a different file, connected only through method calls and dependency injection. These are semantic relationships, not syntactic ones; AST pattern-matchers can't follow them.
Target-value determination
There is an additional problem a source-code-only tool cannot solve: target-value determination and live rollout querying. A tool that only reads source code could confidently hardcode the wrong value and silently change behavior. Ensuring the tool performs the correct code cleanup requires knowing the flag's live rollout state, which lives in the experimentation platform, not the repo. For example, DoorDash’s experimentation platform supports the usage of integer, string, and JSON feature flags. A live readout of the target value ensures that the tool is modifying the code with the feature flag’s correct value that represents the system’s current state. Similarly, rollout information is also an important part of the equation. An experiment may have been rolled out to 60% and then abandoned. The tool would need to have the intelligence to determine that this is a partial rollout and ask the user for confirmation before performing any cleanup. This can either trigger the user to complete the rollout/rollback before using the tool to perform the cleanup, or prompt the user to skip the cleanup for now.
Scale and complexity
Finally, even a simple Boolean DV can touch five to 20 files after accounting for the constant, the wrapper, every call site, and updating or deleting relevant tests. This is a job that needs to understand code, not just match syntax. This is exactly why we opted to use LLM agents. But the job wasn’t as simple as writing a prompt and asking the agent to perform the cleanup; that would not achieve a cost- and time-efficient cleanup with guardrails at scale.
A two-phase, human-in-the-loop pipeline
The system is built on Google's Agent Development Kit (ADK) and runs in two phases with a single human checkpoint between them, as shown in Figure 1.
Phase 1: Analysis and report
An orchestrator agent, in this case Claude Sonnet, processes stale-DV requests. For each DV, it:
- Fetches the stale DV Jira tickets via the Atlassian command-line interface tool.
- Queries DoorDash’s experimentation platform over model context protocol (MCP) for the DV's metadata — its UUID, rollout percentage, and target value.
- Discovers the local repository and searches for every code reference to the DV.
- Generates a structured report proposing a target value and listing the affected files for each DV.
That MCP integration is the quiet hero of the design. By reading the live production rollout state, the agent determines the correct target value instead of guessing from the code. For example, a DV that was introduced for a feature that ended up being abandoned shows a 0% rollout on the platform, so the agent knows to remove it using the baseline value rather than assuming the feature is on.
Human checkpoint
Before any code changes, an engineer reviews the report and confirms each DV's target value. This is a deliberate safety valve for ambiguous cases, such as partial rollouts or abandoned experiments, where the right answer needs human judgment. For example, an experiment may have been rolled out to 60% and then abandoned. The human checkpoint allows the user to intervene by deciding, for instance, whether to complete the rollout or roll back the experiment before the cleanup proceeds.
Phase 2: Parallel cleanup
For every confirmed DV, the orchestrator spawns a dedicated removal agent — Claude Opus — with each running in its own isolated git worktree. Up to four agents run concurrently per repository, which is configurable. Each one drives a full cleanup and opens a pull request assigned to the DV's owner.
Inside a removal agent
Each removal agent runs an iterative workflow as follows:
- Search for all references to the DV, including the constant, the wrapper function, and every call site.
- Decide a cleanup strategy based on the DV's type — Boolean, integer, or string — and its usage.
- Apply the edits; replace wrapper calls with the target value and simplify the now-constant conditionals.
- Remove the wrapper function, the definition, and the constant.
- Update or remove affected tests, such as deleting "flag disabled" test branches.
- Confirm the build passes.
- Confirm the tests pass.
- Verify patch coverage is at least 95% on changed lines.
- Run the Detekt linter.
- If any validation fails, diagnose, fix, and retry.
Autonomy over a codebase only works if the guardrails are real. Four mechanisms keep the agents safe:
- Git worktree isolation: Each agent works in a dedicated worktree, so concurrent agents never step on each other, and a failed cleanup is discarded by simply removing the worktree.
- A one-hour hard timeout per agent: This ensures that a stuck session can't run away.
- JaCoCo patch-coverage validation: This enforces at least 95% test coverage on the lines the agent changed.
- Detekt static analysis: This enforces style and catches common issues.
We also run Gradle with --no-daemon to eliminate cross-worktree state pollution during parallel runs. Crucially, an agent is not allowed to open a PR until tests and static analysis pass locally.
Results
Building a system is one thing, but only one question matters when it runs at DoorDash's scale: How well does it work? In this case, very well indeed. Of the 50 most recent stale flags the system processed, it produced a merge-ready PR for 45 of them at an average cost of $4.79 and 13.8 minutes per cleanup instead of the one to two hours a manual cleanup would have taken. The system introduced no bugs or regressions along the way.
How we evaluated it
We evaluated the system on the 50 most recent stale DVs it processed across several Kotlin repositories. We labeled each by complexity:
- Simple — A single Boolean check in one to two files; n=6
- Medium — Multiple usages across three to five files with some test changes; n=18
- Complex — nested conditionals, cross-file dependencies, non-Boolean types, or significant test updates; n=26
By type, 41 were Boolean, six were string, and three were integer/long. Each PR was judged on three criteria: CI passes — including the 95% patch-coverage gate — code correctness, and developer review.
Outcomes by complexity
As shown in Figure 2, 31 of the 50 cleanups were merged in the first shot, 14 needed one minor revision, and five required an engineer to step in. The complexity breakdown maps the system's reliability boundary:
- Simple DVs: 100% cleaned up in a single pass.
- Medium DVs: 94% (17/18) 14 done in a single pass, three after one revision, and one needing engineer intervention.
- Complex DVs: 85% (22/26), even with cascading parameter removal through more than five method layers, nested conditionals, and heavy test refactoring.
The 14 revisions were minor and self-correcting; six were insufficient patch coverage requiring one more iteration to add tests, and eight were incomplete dead-code removal with a leftover variable or reference deep in a call chain. However, all were fixed in one re-prompt of the LLM.
The most important finding had to do with what kind of thing went wrong. Across all 50 generated PRs, we observed no bugs or regressions introduced by the changed code. Every one of the five intervention cases came down to call-chain depth with cross-interface parameter threading; the agent correctly removed the majority of the dead code, but missed a link or two in a long multi-file chain. In other words, the system's reliability boundary is completeness, not correctness, alleviating concerns that the system could create larger problems when it was unsuccessful.
Cost and time
Cost and time scale with complexity, but stay firmly in "run-it-at-scale" territory, as shown in Table 1 and Figure 3.
Table 1 breaks down the average and median performance metrics, specifically time and cost, for the 50 analyzed feature flags categorized by their complexity. This data highlights that while more complex flags naturally require more time and budget, the system remains highly efficient even for the most difficult cleanup tasks.
| Complexity | n | Avg. time | Median time | Avg. cost | Median cost |
| Simple | 6 | 7.5 min | 8.2 min | $2.69 | $2.50 |
| Medium | 18 | 10.4 min | 9.9 min | $3.46 | $3.42 |
| Complex | 26 | 17.7 min | 14.4 min | $6.20 | $4.50 |
Table 1: Performance metrics by complexity level across 50 evaluated feature flags.
As illustrated in Figure 3, simple and medium DVs cluster tightly in the cheap and fast corner. Complex DVs spread out, with a handful of outliers reaching roughly 40 minutes and around $19. Even at the top of that range, the agent represents a dramatic reduction in engineering effort and reduced costs against even a conservative one-hour estimate of what would be required to remove a single DV manually.
What we learned
- Specializing models by role pays off. We use Claude Sonnet for orchestration, including broad context, lightweight metadata gathering and planning, and Claude Opus for the removal agents, including deep reasoning about code semantics, call chains, and test dependencies. Reserving the more expensive model for the cognitively demanding editing phase cut our overall API cost without hurting cleanup quality.
- Worktree isolation is non-negotiable for parallel agents. Our early experiments without worktree isolation produced race conditions where concurrent agents edited the same files and generated corrupted diffs. A dedicated worktree per agent plus --no-daemon Gradle made parallel cleanup safe and made failure disposable.
- Live experimentation integration was the highest-leverage decision. Without the current rollout state, any automated approach — rule-based or LLM — risks hardcoding the wrong value and quietly preserving the wrong behavior. Reading it over MCP added negligible latency and eliminated a whole class of target-value errors.
Any team considering similar automation should note that our results point to three practical prerequisites:
- A reliable source of deployed flag state
- Worktree isolation for parallel edits, and
- Deterministic validation gates before a PR is ever opened.
What's next
We're extending the system along two lines:
- A phase 1 confidence score to auto-approve low-risk cleanups. For example, a Boolean DV at 100% rollout whose code default already matches production, which reserves the human checkpoint for genuinely ambiguous cases.
- A post-cleanup code-quality pass. This would catch semantic issues such as variable names that no longer reflect their purpose once a flag is gone.
Feature flag cleanup is the kind of work that's individually small, but collectively enormous, particularly at DoorDash’s scale, and perpetually deferred. By combining an orchestrator, parallel removal agents in isolated worktrees, live experimentation platform lookups, and hard validation gates, our feature flag cleanup system transforms a thousands-of-hours backlog into a cheap, reliable background process that produces merge-ready PRs for 90% of the flags it touches at a cost of about $4.79 and 14 minutes each, with no regressions. The remaining boundary is completeness on the deepest call chains, and that's exactly where we're focused next.
Interested in building systems like this? DoorDash Engineering is hiring – come help us build at scale.