DoorDash announced that its internal multi-agent large language model (LLM) pipeline has begun automatically cleaning up stale feature flags across a codebase of roughly 60,000 entries. In a controlled evaluation of 50 identified stale flags, the system produced usable pull requests for 45 of them, with an average turnaround of 13.8 minutes per flag and an estimated cost of $4 per cleanup operation. The result represents a concrete step toward reducing technical debt at a scale that would be infeasible for human engineers alone InfoQ.

Architectural Overview of the Multi-Agent Pipeline

The pipeline is built around three tightly coupled components: (1) a telemetry ingestion layer that streams live A/B experiment results, (2) a suite of specialized LLM agents that reason over flag usage patterns, and (3) an execution harness that creates isolated Git worktrees for each candidate change. The telemetry layer continuously aggregates request-level metrics (hit count, conversion impact, error rate) for every feature flag. When a flag’s activation rate falls below a configurable threshold for a sustained period—typically 30 days—the system flags it as a candidate for removal.

The first LLM agent parses the telemetry snapshot and generates a concise rationale for deprecation. A second agent drafts the corresponding code modifications, which include removing the flag definition, cleaning up conditional branches, and updating any related configuration files. A third agent constructs a pull request (PR) skeleton, embeds the rationale as the PR description, and tags the original flag owner for approval. All agents share a common backbone model fine-tuned on DoorDash’s internal codebase (approximately 12 B parameters) and run on NVIDIA H100 GPUs, achieving inference latency of roughly 120 ms per token.

Safety Mechanisms and Validation Workflow

Safety is enforced at three stages. First, the system requires explicit engineer approval before any PR is opened; the flag owner receives an automated email containing the LLM-generated justification and a diff preview. Second, each PR is built and tested in an isolated Git worktree, preventing interference with the main branch and ensuring that any side effects are caught early. Third, the pipeline runs the full suite of unit, integration, and performance tests, leveraging DoorDash’s existing continuous integration (CI) infrastructure. Only when all tests pass does the PR become merge-ready.

The validation step is crucial because feature flags often guard complex conditional logic. By executing the changes in a sandboxed environment, the system can detect regressions that would otherwise be missed by static analysis alone. This mirrors the approach described in recent research on LLM-driven code generation, where runtime verification dramatically reduces hallucination-related bugs.

Pilot Results and Economic Impact

During the pilot, the system processed 50 stale flags drawn from a random sample of the 60,000-flag universe. It generated usable PRs for 45 flags, while the remaining five required manual intervention due to ambiguous dependencies that the agents could not resolve. The average time from flag identification to PR creation was 13.8 minutes, compared with an estimated 4–6 hours of manual effort per flag based on internal engineering surveys. At DoorDash’s internal cost model—approximately $0.30 per compute second on H100 instances—the per-flag expense totals roughly $4, a figure that is dwarfed by the long-term maintenance savings of eliminating dead code paths.

Beyond raw cost, the cleanup reduces the cognitive load on developers who no longer need to reason about obsolete branches. It also lowers the risk of accidental re-activation of deprecated flags, which can introduce regressions in production traffic. The pilot’s success prompted DoorDash to schedule a phased rollout that will target the entire flag inventory over the next six months.

Broader Implications for CI/CD Automation

DoorDash’s deployment illustrates a maturing class of AI-augmented development tools that move beyond suggestion-only models to fully autonomous agents capable of end-to-end code modification. By integrating LLM reasoning with existing CI pipelines, the company demonstrates that large-scale refactoring tasks—traditionally reserved for senior engineers—can be delegated to trustworthy agents. This aligns with the broader industry trend highlighted in the recent Grab LLM-Kit Accelerates AI Agent Production Deployment, where platform teams provide reusable agent frameworks to accelerate adoption.

However, the approach also surfaces new risk vectors. Automated PR generation depends on the fidelity of the underlying model and the quality of the telemetry data. Mis-labelled flags or incomplete experiment logs could cause the system to propose unsafe removals. DoorDash mitigates this through mandatory human sign-off and exhaustive test coverage, but organizations adopting similar pipelines must invest in robust data hygiene and governance processes.

Operational Considerations and Future Directions

From an operational standpoint, the pipeline consumes a modest fraction of DoorDash’s GPU capacity—approximately 0.5 % of the H100 fleet during peak hours—leaving ample headroom for other AI workloads. The model’s parameter count (12 B) balances expressiveness with inference cost; larger models would increase latency without demonstrable gains in code quality for this specific task.

Looking ahead, DoorDash plans to extend the agent suite to other forms of technical debt, such as orphaned microservice endpoints and unused database schemas. The modular architecture of the pipeline should allow new agents to plug into the same telemetry and validation backbone, reducing engineering effort for each new use case.

Stakeholder Impact

  • Engineering teams gain a reliable assistant that offloads repetitive cleanup work, freeing capacity for feature development.
  • Product managers benefit from a cleaner codebase that reduces the likelihood of flag-related regressions during releases.
  • Operations see lower CI pipeline runtimes as the number of stale flags—and associated test cases—declines.
  • Security auditors obtain a clearer attack surface, since dormant flags can sometimes expose hidden configuration pathways.

Conclusion

DoorDash’s multi-agent LLM system demonstrates that autonomous code-modification pipelines can operate at enterprise scale while maintaining safety through layered validation and human oversight. The pilot’s efficiency—45 usable PRs out of 50 candidates in under 14 minutes each—suggests a viable path for other large tech firms to address accumulated technical debt without diverting senior engineering resources. Continued monitoring of false-positive rates and expansion to other debt categories will determine whether this approach becomes a standard component of modern CI/CD ecosystems.

Related coverage

Explore more on this topic