60% of Our Data PRs Now Merge With No Human Review with Chetas Joshi, Robinhood

A hybrid PR review system -- deterministic scripts for provable facts, an AI agent for contextual judgment, and humans reserved for protected decisions -- le

60% of Our Data PRs Now Merge With No Human Review

Abstract

Pull requests against production data pipelines now arrive faster than human reviewers can safely evaluate them, as AI coding assistants have driven the marginal cost of opening a PR toward zero. This synthesis examines a hybrid reviewer architecture built by a data platform team at Robinhood to close that gap without removing human accountability from decisions that require it. The approach decomposes review into three duties with three distinct owners: deterministic scripts verify claims against systems of record, a large language model (LLM) agent supplies contextual judgment the scripts cannot provide, and human reviewers retain exclusive authority over a fixed set of protected decisions. As of August 31, 2026, approximately 60% of data pipeline PRs merged without human review, manual review effort fell roughly 75% (about 132 engineering hours per month), and related incidents dropped 30%, at a measured miss rate near 2%. The analysis details the architecture, a representative case study, measurement methodology, and structural limitations that remain unresolved.

1. Introduction

Data platform teams increasingly confront a structural mismatch: the rate at which pull requests can be authored has decoupled from the rate at which they can be safely reviewed. AI coding assistants make it cheap to propose schema changes, pipeline code, configuration changes, and access grants, but the team responsible for merging these changes is bounded by the number of people who understand what is actually safe to ship. As the source material states, "Pull requests to our production data pipelines now arrive at machine speed. But reviewing them safely still runs at human speed."

Conventional continuous integration (CI) combined with mandatory human sign-off catches genuine defects, but it scales only with headcount and with whatever context an individual reviewer happens to carry at the time. The team frames this as a persistent gap between CI passed and this change was actually safe - a gap that must be closed deliberately rather than assumed away.

This paper's central claim is that review quality improves when work is decomposed by epistemic type rather than delegated wholesale to either automation or people. Facts that can be checked against a source of truth (a schema registry, a catalog, a lineage graph) belong to deterministic scripts. Inferences that require reading code and inferring relationships belong to an LLM agent. Decisions that carry organizational accountability - who can access sensitive data, whether a migration changes meaning, what to do when confidence is low - belong to people. Section 2 situates this decomposition conceptually; Section 3 presents the architecture and a case study; Section 4 extracts technical insights; Sections 5 and 6 discuss implications and conclude.

2. Background and Related Work

The baseline practice in most data engineering organizations pairs automated CI gates with a mandatory human approval step. This combination fails not through false positives but through incompleteness: CI validates syntax and test coverage, while the properties that actually determine pipeline safety - contract conformance, catalog metadata, lineage topology, cluster capacity, and access policy - sit outside what a typical test suite exercises. Human reviewers compensate for this gap, but only to the extent of their individual knowledge of the system's current state, which does not scale with PR volume.

Three frameworks organize the approach examined here. The three-duties decomposition assigns scripts the job of proving what can be proven, an agent the job of judging what needs context, and people the job of owning calls only they should make. A flag-never-clear policy governs three categories - sensitive access grants, migrations that change data meaning, and low-confidence agent verdicts - in which the agent is permitted only to surface risk, never to approve. A miss versus false-escalation matrix treats both failure directions (bad PRs approved, and fine PRs needlessly escalated) as costs to be measured, rather than optimizing solely for catching defects.

3. Core Analysis

3.1 Architecture and the Merge Gate

The system operates in two layers plus a protected human tier. Layer 1 consists of deterministic scripts that query systems of record; any failed check turns CI red and blocks merge with no AI involvement. Layer 2 is an LLM agent that reviews the PR's diff and surrounding repository context, assigns a risk score, and either approves or escalates to a human. Merge requires conjunctive satisfaction: all Layer 1 checks must pass and an approving review must exist, whether from the agent or an escalated human.

A key design choice is that the agent's rules are configured relative to what Layer 1 already verifies, so inference is spent only on risks the scripts cannot establish. Restricting the model to judgment rather than fact-rediscovery is reported to keep AI inference costs down, since the agent is not re-deriving facts a script can check deterministically and cheaply.

3.2 Layer 1: Deterministic Verification

Four check families run against systems of record. The data contracts check compares schema and keys against the schema registry. The required tags check verifies the catalog records an owner and PII labeling for the affected table. The lineage blast radius check walks every downstream job registered in the lineage tool for a changed column to detect breakage. The performance check runs the change in staging at production data volume with mock data to validate cluster sizing and service-level agreement (SLA) adherence.

These checks are only as complete as the systems they query. The lineage blast radius check's notion of "downstream" is bounded by what is recorded in the lineage tool, not by the full set of actual consumers - a limitation that becomes consequential in Section 3.4.

3.3 Layer 2: The Agent's Code Walk

The agent is equipped with file tools to read, search, and list files across the repository; it has no live access to the catalog, lineage system, or production state. It reads the diff alongside pipeline definitions and access grants to surface relationships the author may have missed, with rules targeting query anti-patterns (e.g., an unpartitioned cross join that runs correctly but still overloads a cluster), sensitive data requests, and consumers outside registered lineage.

The agent is designed to be explicit about the limits of its evidence: as the source material notes, "If it does not find anything, it does not get to say no downstream consumers exist. It says it found none in the places it searched. None found is not the same as none exist." Output consists of line-level comments with reasoning, a risk score, and a call to approve or escalate.

3.4 Case Study: The Schema Widening PR

A representative PR added a new column mirroring an upstream source and widened an existing column from integer to big integer. All Layer 1 checks passed: the contract check confirmed the new column existed upstream, the lineage blast radius check found no breaks among registered consumers, and tags and performance checks cleared. Despite this, the agent's code walk identified an unregistered job - absent from the lineage graph - reading the widened column and writing it into another table where the column remained a plain integer. The first value exceeding integer range would have caused that downstream write to fail.

This gap existed because workloads run across different query engines, each of which must separately publish lineage; a missing publish step leaves the relationship invisible to automated checks. The agent escalated with its risk score and full reasoning chain; a human reviewer confirmed the unregistered job was live in production, and the downstream table was updated to handle the wider type before the original PR merged. No incident occurred. As the source material summarizes the broader pattern: "As far as the scripts can see, this PR is safe... It is not the whole story."

4. Technical Insights

Several implementation details generalize beyond this specific system. First, the performance check's use of staging environments at production data volume with mock data is a practical way to validate cluster sizing and SLA adherence before merge, without touching production data directly. Second, confining the agent's tool access to repository-level read/search/list operations - excluding live catalog, lineage, or production queries - is a deliberate cost and risk control, though it also bounds what the agent can verify independently.

Third, the flag-never-clear policy is enforced structurally: grants touching sensitive data, migrations changing data meaning, and low-confidence verdicts always escalate, meaning the agent's authority to approve is scoped narrower than its authority to review. Fourth, the measured miss rate of approximately 2% (as of August 31, 2026) was attributed primarily to rule gaps rather than agent judgment failures, suggesting that failure modes in this architecture are more tractable to address through rule maintenance than through model retraining. Fifth, the feedback loop is explicitly manual: humans reply to agent comments with agreement or disagreement and explanations, feedback is collected daily, and an on-call engineer updates rules by hand weekly - there is no automatic retraining, though the team has an unbuilt idea to use a separate incident-investigation agent to assist.

5. Discussion

The results reported - roughly 60% of PRs merging without human review, a 75% reduction in manual review effort (about 132 hours per month), and a 30% reduction in related incidents - suggest that decomposing review by epistemic type, rather than treating "AI review" as a monolithic replacement for human judgment, is a viable strategy for data platform teams facing PR volume that outpaces reviewer headcount. The schema-widening case study illustrates why neither layer alone would have sufficed: Layer 1's lineage check was blind to an unpublished relationship, while a judgment-capable agent reading code directly found what structured metadata missed.

A notable gap in the current design is that Layer 1 and Layer 2 run as separate systems triggered in parallel, with neither aware of the other's pass/fail state or outage status, and no automatic fallback when one layer cannot finish - the process simply waits for a human. The proposed future direction, a classifier that decides which changes warrant separate versus combined review and an agent granted callable access to live Layer 1 checks, remains unbuilt, indicating that tighter integration between deterministic and judgment-based layers is an open area for this class of system.

The emphasis on measuring false escalations alongside misses is also notable against prevailing industry tendencies to optimize automated review systems solely for catch rate. As the source material frames it, a reviewer that escalates everything is not reducing review burden, merely redistributing it - a distinction with direct relevance to any team evaluating the real cost savings of an AI review layer.

6. Conclusion

This analysis documents a hybrid reviewer architecture that assigns review duties by epistemic type: scripts for provable facts, an agent for contextual judgment, and people for protected decisions. The design achieved meaningful reductions in review burden and incidents while preserving human accountability over the decisions where accountability matters by policy, not merely by convention. The case study demonstrates the practical value of combining metadata-driven checks with code-reading judgment, since each catches failure modes invisible to the other.

For practitioners considering similar systems, the transferable lessons are to let deterministic checks absorb anything provable against a source of truth, to restrict automated approval authority structurally rather than relying on the agent's own confidence, and to measure false escalations as rigorously as misses when assessing whether an automated review layer is actually saving time.

Sources

• 60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub