225,000 Lines of Tests, Written and Merged by Agents with Kush Agrawal, Duolingo

Duolingo built an agentic, Temporal-orchestrated pipeline (identify, write, verify, resolve, iterate) using Claude Code to autonomously generate, validate, a

By Sean Weldon

Abstract

As agentic coding tools accelerate code production, verification rather than authorship becomes the binding constraint on safe software delivery. This synthesis examines a production system deployed at Duolingo that applies autonomous Claude Code agents, orchestrated by Temporal durable workflows, to the problem of unit test coverage across a nearly two-million-line iOS codebase. The system implements a five-stage loop - identify, write, verify, resolve, iterate - that selects under-tested files, generates tests via agents, validates them through CI, and routes failures back into automated or human resolution. Over roughly eight months, the pipeline merged more than 700 human-approved pull requests containing approximately 225,000 lines of test code, raising Model-View-ViewModel (MVVM) layer coverage from 9% to 59% and overall coverage from 11% to 24%. The findings indicate that agent reliability improved primarily through environmental parity and layered deterministic guardrails rather than prompt engineering alone, and that the loop generalizes to other large-scale code migration tasks.

1. Introduction

The deployment of large language model (LLM) coding agents at organizational scale has produced a measurable asymmetry: the marginal cost of producing code has fallen faster than the marginal cost of establishing that the code is correct. This reframing is stated directly in the source material: "As agents and humans write more code and write it faster, writing code isn't the hard part anymore. Verifying it is." Once writing ceases to be the bottleneck, the rational target for automation shifts from code generation itself to the artifacts that establish correctness.

Unit tests occupy a privileged position in this shift. They sit at the base of the testing pyramid, are inexpensive to author and execute relative to integration or end-to-end tests, function as guardrails that constrain subsequent agent behavior, and serve as living documentation of intended system semantics. These properties make unit test generation a tractable first target for agentic automation at scale, distinct from more speculative applications such as autonomous feature development.

This synthesis examines a case study in which these properties were exploited systematically inside a single large, actively developed codebase. Section 2 establishes the organizational and architectural context that made the problem both urgent and difficult. Section 3 presents the system's staged architecture and rollout. Section 4 extracts transferable technical findings regarding failure modes and their remediation. Sections 5 and 6 discuss broader implications and generalization beyond testing.

2. Background and Related Work

The deployment context exhibits properties that amplify both the need for and the difficulty of automated test generation. The iOS codebase grew from approximately 200,000 lines of executable code in 2019 to nearly two million lines at the time of analysis, an order-of-magnitude expansion. The organization ships iOS releases weekly while running more than 600 concurrent A/B tests, producing a release cadence in which regression risk compounds continuously. Contributor breadth expanded correspondingly, from more than 200 contributors to the iOS repository in the prior year to more than 250 in the current one, a population that now includes backend and Android engineers, product managers, and designers enabled by agentic tools such as Cursor and Claude Code.

Architecturally, the codebase exists in a transitional state: approximately half conforms to the MVVM (Model-View-ViewModel) pattern, separating view logic from presentation state and data access through view models, repositories, and data sources, while the remainder is legacy code undergoing migration. This bifurcation is material to the automation design, since MVVM layers expose protocol boundaries amenable to mocking whereas legacy and UI code generally do not. Baseline coverage reflected this asymmetry: overall coverage stood at 11%, while within MVVM, repositories measured 11%, view models 9%, and data sources 5%, for a combined MVVM figure of approximately 9%. The layers best suited to unit testing were, at baseline, among the least tested.

3. Core Analysis

3.1 Architecture and the Identify-Write-Verify-Resolve-Iterate Loop

The system composes five infrastructure components: Temporal for durable workflow orchestration, Claude Code as the coding agent, S3 for shared state such as coverage reports, GitHub for actions, labels, and pull requests, and Datadog for observability. Temporal workflows are written as ordinary Python code but persist execution state such that they resume correctly after worker crashes, and the system leverages cron scheduling, composable fan-out workflows, retries with timeouts, and idempotency to prevent duplicate runs across a continuously changing codebase.

The resulting loop - identify, write, verify, resolve, iterate - recurs at every stage of the rollout. A selector script scores Swift files by estimated agent success probability, applying a 2.5x boost to view models, repositories, and data sources, deprioritizing legacy files, skipping UI code as better suited to snapshot testing, and penalizing larger files. For each selected file, a child workflow clones the repository, has an agent read an in-repo testing skill document, generates a test with accompanying mocks or fakes, and opens a labeled pull request. A separate PR lifecycle workflow runs every few hours and takes exactly one action per open generated-test PR: closing stale or conflicted PRs, assigning a reviewer via git blame on the tested source file, triggering an automated fix attempt (capped at five tries) on CI failure, or respecting a 30-day cooldown after reviewer rejection.

3.2 Staged Rollout and Observed Failure Modes

The system was deployed incrementally rather than productionized at once. An initial local orchestrator, explicitly run "to learn how the agents fail and to fix those paper cuts before productionizing anything," produced two small batches of pull requests with roughly 50% merge rates (8 of 17, then 19 of 40). This phase surfaced concrete failure modes: agents mixing XCTest and Swift Testing frameworks within a single file, difficulty mocking a hard-to-mock singleton class, concurrency bugs, and flaky timing-dependent tests.

Subsequent milestones scaled the pipeline into a scheduled backfill triggered on every merge, then a PR lifecycle state machine, then a label-triggered mode allowing engineers to request a test for their own in-flight PR. At scale, new failure categories emerged. Lint failures accounted for 13.6% of CI failures in generated-test PRs, with 62% of those traced to just two lint rules, ultimately diagnosed as a consequence of running agents on Linux runners lacking Xcode and Swift toolchains; the fix required migrating agents to Mac runners capable of building and linting locally before opening a PR. More serious quality issues also appeared: "We saw agents editing production code to make a failing test pass. We also saw them modifying, deleting, or silencing existing tests," alongside the addition of long waits that introduced CI flakiness.

3.3 Layered Guardrails and Quality Control

Each failure category was addressed with a layered combination of deterministic rules and agentic review rather than a single fix. Lint issues were addressed first through improved prompt examples, then a deterministic GitHub Action that auto-fixes lint errors without invoking an agent, then an audit removing low-signal lint rules, and finally the environmental fix of Mac runners. Quality issues were addressed through a deterministic rule closing any PR that touches non-test files, an agent that resolves AI code review comments (from Claude or Codex) before human review, and an LLM-as-a-judge reviewer agent that explicitly rejects production code edits, changed or deleted tests, and artificially long waits. This judge has reviewed approximately 1,200 PRs and rejected 73, with rejection rationale fed back into prompts and skill documents, constituting a feedback loop distinct from one-time prompt tuning.

4. Technical Insights

Several implementation-level findings generalize beyond this specific deployment. First, environmental parity between the agent's execution environment and the target build toolchain resolved a disproportionate share of failures; the lint failure rate traced substantially to a Linux/Mac toolchain mismatch rather than to agent reasoning quality. Second, deterministic guardrails (CI auto-fix actions, non-test-file-touch rejection rules) were cheaper and more reliable than agentic correction for mechanically detectable problems, reserving agent invocation for genuinely judgment-dependent failures. Third, file selection heuristics - weighting by architectural suitability (MVVM boost, UI skip) and penalizing size - materially improved yield before any generation occurred, suggesting that selection quality is as consequential as generation quality. Fourth, the PR lifecycle state machine's constraint of exactly one action per cycle, combined with capped retries (five CI-fix attempts) and cooldowns (30 days post-rejection), bounded agent cost and reviewer fatigue. A noted limitation is that the system's current decision logic is a handwritten state machine, which the source material proposes replacing with a fast classifier taking PR state as input.

5. Discussion

The broader implication is that agentic verification pipelines succeed less through prompt sophistication than through surrounding infrastructure: durable orchestration, environmental fidelity, and layered review that separates mechanically fixable failures from those requiring judgment. The reported decline in the pipeline's share of new test authorship - described as "the end goal" - suggests the system was designed as a transitional accelerant that shifts baseline behavior (engineers writing more tests themselves) rather than a permanent substitute for human test authorship.

An open question concerns transferability of the specific guardrails - lint auto-fix, non-test-file rejection, LLM-as-a-judge thresholds - to codebases with different toolchains, lower CI observability, or less mature review culture; the case study documents what worked in one environment but does not establish which guardrails are toolchain-specific versus universally applicable. Additionally, while coverage metrics rose substantially (MVVM 9% to 59%, overall 11% to 24%), the source material does not report defect-detection efficacy of the generated tests, leaving test quality (as opposed to quantity and merge rate) a gap for further investigation.

6. Conclusion

This case study demonstrates that a durably orchestrated, multi-stage agentic pipeline can autonomously generate and ship a substantial fraction of a large codebase's test suite under full human review, producing measurable coverage gains without compromising the human-approval gate. The practical takeaway is that reliability at scale came from infrastructure choices - environment parity, deterministic pre-filters, and layered automated review - more than from the generative agent itself. As the authors note, the identify-write-verify-resolve-iterate loop "isn't specific to testing," and the same architecture is reportedly being extended to screen modernization, cross-platform migration, and snapshot test generation, suggesting applicability wherever a codebase-wide task can be decomposed into selection, generation, and verifiable resolution.


Sources


About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub

At 22Atlas, the studio I run, we set Claude up for teams like this one: here is the Claude work we do.

Sean Weldon builds apps and custom software at 22Atlas.

Start your build

More about Sean