'ReviewDebt: a practical framework for scoring every pull request - Sachin Gupta, Ebay'

Coding agents are creating unmeasured 'review debt' - the accumulating gap between code produced and code humans have actually reviewed and understood - which co...

By Sean Weldon

Review Debt: A Deterministic Framework for Quantifying Code Review Burden in AI-Assisted Development

Abstract

The proliferation of AI-powered coding agents has generated measurable productivity gains while simultaneously introducing an unmeasured liability: review debt - the accumulating gap between code produced by agents and code that human engineers have actually reviewed, understood, and validated. Analysis of industry benchmarks reveals a 25% year-over-year increase in commits alongside a 27% decline in review comments, with 31% of pull requests now merged without review. This paper presents a deterministic scoring framework comprising five signal families: diff size and coupling, test evidence gaps, directory and ownership spread, AI authorship indicators, and evidence/rational gaps. Validation across 524 pull requests from three public repositories demonstrates that complexity and volume, rather than AI authorship per se, drive review burden. The framework provides engineering organizations with actionable metrics to balance velocity gains against code quality governance, establishing accountability mechanisms essential for sustainable AI-assisted development practices.

1. Introduction

The integration of AI coding agents into software development workflows has fundamentally altered the economics of code production. GitHub's 2025 report documented a 25% year-over-year increase in commits, suggesting unprecedented developer productivity. However, this apparent efficiency gain masks a critical structural problem: the capacity for human code review has not scaled proportionally with AI-generated output. The Faros AI 2026 benchmark revealed that median pull request (PR) review time increased 441.5%, with reviewed PRs now requiring 5.4 times longer to process than previously. Simultaneously, 31% of PRs are merged without any review whatsoever, while PRs per developer increased 16% and median PR size expanded 63% (from 44 to 72 lines).

This paper introduces the concept of review debt - defined as the accumulating gap between code that agents produce and code that humans have reviewed, trusted, and understood. Unlike technical debt, which represents deferred refactoring work, review debt compounds through three distinct feedback loops. First, agents learn from unreviewed code, grounding future pull requests in potentially flawed patterns. Second, reviewers contract their attention to syntax and obvious bugs, moving architectural decisions from review to never. Third, velocity expectations reset upward without corresponding increases in reviewer capacity, eliminating slack needed to pay down accumulated debt. The debt accrues interest paid not in monetary terms but in human attention - a fundamentally non-scalable resource.

The central question this analysis addresses is whether organizations can maintain trust in shipped code when the volume of changes systematically exceeds review capacity. The framework presented here provides deterministic, computable signals that quantify review burden without relying on language models as judges, ensuring defensible and stable scoring mechanisms.

2. Background and Related Work

2.1 Empirical Evidence of Review Capacity Crisis

Recent industry data establishes the empirical foundation for the review debt phenomenon. Beyond the aggregate statistics, specific patterns reveal the mechanisms of debt accumulation. The paradoxical decrease in cycle time despite increased review burden exemplifies what this framework identifies as vanity metrics - superficial indicators that mask underlying quality erosion. When a single logical change splits into seven separate PRs, PR count increases artificially. When median PR size grows from 44 to 72 lines, this represents bloat rather than benefit. When cycle time decreases, this may reflect reviewers ceasing pushback rather than increased trust in code quality.

2.2 Manifestations in Development Practice

Review debt manifests through observable patterns in development workflows. Reviewer fatigue emerges as engineers shoulder substantially greater review loads than in prior periods. Late-night merge patterns reveal PRs remaining unreviewed for 3-4 days before sudden approval ahead of Friday deadlines, indicating approval without substantive review. Test theater describes the phenomenon where tests assert what code currently does rather than specifying intended behavior, effectively locking in bugs rather than preventing them. Architectural drift occurs when the same problem is solved three different ways across three files because no architectural trade-offs were held during review. Incident lag manifests as bugs discovered weeks or months after merge, with no traceable connection to AI-authored changes that introduced them.

3. Core Analysis

3.1 The Compounding Mechanism of Review Debt

Review debt differs fundamentally from technical debt through its compounding nature. Three feedback loops operate simultaneously to accelerate debt accumulation. The first loop involves agents learning from unreviewed code: when AI systems train or retrieve context from repositories containing unvalidated changes, they ground tomorrow's pull requests in potentially flawed patterns. The second loop describes reviewer behavior contraction: as volume increases, reviewers necessarily focus on syntax and obvious bugs, permanently shifting architectural and design decisions outside the review process. The third loop concerns organizational expectations: as velocity metrics improve, stakeholders reset baseline expectations without corresponding increases in reviewer hiring, eliminating any slack that might be used to pay down accumulated debt.

Each loop is individually survivable, but their interaction creates what the framework characterizes as "a runway" - a finite period before code quality and team accountability become unmaintainable. The interest on this debt is paid in human attention, which cannot be scaled through technological means. This distinguishes review debt from technical debt, which can often be addressed through refactoring sprints or automated tooling.

3.2 Five Signal Families for Deterministic Measurement

The framework quantifies review debt through five signal families, all computed deterministically from PR and repository metadata without using language models as judges. This design choice ensures scoring stability and defensibility over time.

Signal 1: Diff Size and Coupling measures net lines changed, files touched, and whether changes cluster or sprawl. The analysis reveals that agents exhibit bias toward fixing issues at call sites rather than routing fixes to root causes - a pattern that increases coupling and sprawl. The review cost of sprawling diffs is not proportional to size but increases steeply, as reviewers must maintain multiple mental models simultaneously.

Signal 2: Test Evidence Gap calculates the ratio of test lines added to production lines added. AI-generated PRs consistently ship with lower test-to-code ratios. More critically, tests frequently assert what code is currently doing rather than what it should do, creating test theater that provides false confidence.

Signal 3: Directory and Ownership Spread counts distinct code owner teams affected by a diff. Sprawling PRs requiring multi-party approvals impose enormous coordination overhead. The framework emphasizes that staying within one owner territory enables one approval, one context, and one mental model.

Signal 4: AI Authorship Indicators detect AI involvement through co-authored footers (strongest signal), branch name patterns (codex, copilot, cursor prefixes), and PR body phrases ("generated by", "assisted by"). Critically, this signal is not used for blame assignment but rather to weight scores for appropriate reviewer attention.

Signal 5: Evidence and Rational Gaps assesses whether PRs explain why changes were made or merely what was changed. High gaps indicate minimal PR body or commit messages, while low gaps include title, symptom, diagnosis, change rationale, and benchmark links.

3.3 Scoring System and Calibration

The framework produces a single composite score from 0-100 with configurable weights. Calibration proceeds backwards over the last 200 merged PRs against team-specific reviewer experience. Four bands structure interpretation: 0-24 (very low review burden), 25-49 (normal, standard care), 50-74 (needs evidence from author before senior review), and 75+ (high, split or request more context).

Healthy PRs with scores below 50 generate minimal output - a zero-noise design principle. High debt PRs (e.g., score 60) generate structured reports including: a wire list of which signals fired, a reviewer focus list prioritizing attention, and an author next-action list. This design ensures the tool functions as a review burden scorecard rather than an anti-AI scorecard.

3.4 Validation Across 524 Pull Requests

The framework was validated against three public repositories scanned over 27-90 day windows, analyzing 524 total PRs representing 228 cumulative senior reviewer hours. One high-velocity repository sustained a 9 PR per day merge rate. A single PR in the dataset required an estimated 5,036 minutes (84 hours) of review effort, receiving a score of 73.

AI authorship indicators fired on 5-20% of PRs weekly, but critically, none appeared disproportionately in high burden bands. All four PRs scoring in the high burden band (75+) were structural changes: large migrations, SDK rewrites, and multi-team refactors. This finding validates the framework's core insight: complexity drives burden, not authorship. Volume is the actual variable driving review debt accumulation, not the percentage of AI-authored code.

4. Technical Insights

4.1 Implementation and Adoption Strategy

The framework proposes a five-step adoption pipeline. Backfill involves running the scorer over the last 200 merged PRs to identify the highest historical scores, establishing baseline norms. Threshold setting determines the justify line (e.g., 50+) above which additional author evidence is required. Surface posts scores as PR comments for immediate visibility. Aggregate computes weekly per-team slopes as leading indicators of debt accumulation trends. Talk about it brings numerical data into retrospectives, roadmap reviews, and planning meetings, moving discussion from subjective feeling to objective measurement.

4.2 Remediation Principles

The framework articulates four core remediation principles. First, enforce one logical change per PR - not small PRs in abstract, but cohesive units of change. Second, require tests to ship with changes, with human authors explicitly confirming tests assert intended behavior rather than current behavior. Third, maintain single owner territory; split cross-cutting work per-team to preserve one approval, one context, one mental model. Fourth, require human authors to write PR bodies, treating this as the moment the author commits to understanding the change rather than delegating explanation to the agent.

Critically, the framework advocates applying the same review standard to AI-generated and human-generated PRs. The AI indicator amplifier only contributes when other signals are weak; in the validation dataset, AI indicators contributed only 5 of 60 points (8%) in a typical high-burden PR.

4.3 Anti-Patterns and Governance Implications

The analysis identifies several anti-patterns that accelerate review debt accumulation: "approve with comment" merges that defer issues to future work, "we'll catch it in QA" deferrals that shift quality assurance downstream, claims that "PRs are smaller now" that ignore increased PR count and sprawl, and substantiveless "LGTM" comments that provide no evidence of actual review.

Looking toward 2027, the framework positions review debt as the bridge between throughput gains and governance conversations. The central governance question becomes: can the organization trust the code it ships? If not, who is accountable when AI-authored changes cause incidents, and where is the audit trail? Review debt metrics enable quantified discussion: X% throughput gain requires Y points of review debt over Z weeks, which translates to N senior engineer hours needed for remediation.

5. Discussion

The validation data across 524 PRs demonstrates that the review debt framework successfully identifies complexity-driven burden independent of authorship source. The finding that all high-burden PRs were structural changes (migrations, rewrites, refactors) rather than AI-generated routine changes suggests that current concerns about AI code quality may be misattributed. The actual risk lies not in AI authorship per se but in the volume amplification that AI enables, which systematically exceeds human review capacity.

This distinction has significant implications for organizational strategy. Rather than restricting AI tool adoption, which would sacrifice legitimate productivity gains, organizations should implement measurement and governance frameworks that quantify the true cost of increased velocity. The deterministic nature of the five signal families ensures that scoring remains stable and defensible over time, unlike approaches that rely on language models as judges.

The framework's emphasis on human attention as the scarce resource that pays interest on review debt highlights a fundamental asymmetry in AI-assisted development. While code production scales with computational resources, code comprehension and validation remain bounded by human cognitive capacity. This asymmetry suggests that sustainable AI-assisted development requires explicit investment in review capacity, either through headcount or through tooling that reduces per-PR review burden.

Future work should investigate the relationship between review debt accumulation rates and downstream incident rates, establishing empirical links between the leading indicator (debt score) and lagging indicator (production failures). Additionally, research into automated remediation strategies - such as AI agents that split sprawling PRs or generate comprehensive PR bodies - could help organizations manage debt accumulation without sacrificing velocity.

6. Conclusion

This analysis presents a deterministic framework for quantifying review debt - the accumulating gap between AI-produced code and human-reviewed code - through five computable signal families. Validation across 524 pull requests demonstrates that complexity and volume, rather than AI authorship alone, drive review burden. The framework enables engineering organizations to move from subjective concerns about AI code quality to objective measurement of review capacity constraints.

The practical contributions include: a 0-100 scoring system with four interpretive bands, a five-step adoption pipeline from backfill through organizational discussion, and remediation principles emphasizing logical cohesion, test evidence, ownership boundaries, and human-authored rationale. The framework positions review debt as the essential bridge between productivity metrics and governance accountability, enabling evidence-based decisions about the true cost of AI-assisted velocity gains.

Organizations adopting AI coding agents should implement review debt measurement before capacity constraints compromise code quality and team accountability. The deterministic nature of the framework ensures scoring stability while avoiding the moving target problem inherent in language-model-based evaluation. As the industry moves from tool adoption in 2026 toward governance models in 2027, review debt metrics will become essential infrastructure for sustainable AI-assisted development practices.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub