Your PR Says the Tests Passed. Prove It. with Stephen Chin & Cassandra Chin

AI-generated pull requests often fail review because they contain inaccurate or unsupported claims; combining the Tracetune PR-writing framework with a light

Your PR Says the Tests Passed. Prove It.

Abstract

This synthesis examines a two-part approach to improving the accuracy of AI-generated pull request (PR) descriptions. The first component, Tracetune, is an open-source framework that iteratively assembles agent session traces, code, and contextual metadata to draft PR descriptions. The second, Relate, is a Rust-based embedded in-memory graph database that precomputes a connected evidence packet - termed a neighborhood brief - prior to invoking the writing agent. Evaluation on PR Bench, a benchmark combining audit traces, code, and large language model (LLM) judges with human cross-checking, shows quality improvements of roughly 74→76 and 82→86 on a 1-100 scale, alongside an approximate 50% reduction in token usage. The analysis further identifies a counterintuitive model-allocation strategy: higher-quality models for writing, cheaper models for review. These findings suggest that precomputed relationship graphs, rather than repeated LLM-driven re-research, offer a more efficient and accurate foundation for agent-authored documentation.

1. Introduction

As agentic coding systems move from experimental use to production workflows, the operational bottleneck increasingly shifts from code generation to code communication. A pull request is fundamentally a claim: it asserts what changed, why it changed, and what evidence supports its correctness. When those assertions cannot be substantiated, reviewers reject the submission irrespective of the quality of the underlying diff.

"A lot of pull requests are getting rejected. And it's not because the code is bad, it's because the PRs often contain incorrect information."

This failure mode is architectural rather than incidental. An agent tasked with writing a PR description typically reads only its current session log. That log is not a stable record of events: as the agent continues working, its context window is compressed, meaning the representation of earlier actions degrades progressively. Consequently, the agent writes about work it can no longer accurately recall, and factual drift accumulates over the course of a session.

The central question addressed here is whether the evidence required for an accurate PR description can be precomputed as structured relationships rather than re-derived through repeated LLM inference. The analysis proceeds through the framing problem (§2), the Tracetune loop and its residual failure modes (§3.1), the Relate graph-layer intervention (§3.2), the writer/reviewer model asymmetry (§3.3), benchmark results (§4), and generalization beyond PR authoring (§5).

2. Background and Related Work

Tracetune derives from an academic precursor known as the Show and Tell project, which addressed a measurement problem in education: students submitting AI-assisted programming work typically submitted only the final program. The prompting history and iterative effort - arguably the most meaningful evidence of learning - was discarded entirely. Trace capture was developed to reconstruct that process record for research purposes, and this same architecture was later repurposed for professional PR generation.

The same evidentiary gap exists in professional software engineering. A merged commit preserves the artifact but not the reasoning, dead ends, or verification steps that produced it. Reviewers are left to take the description on faith. Prior approaches to grounding agent output in relational data typically require heavyweight infrastructure such as Neo4j, imposing deployment and operational overhead inconsistent with a lightweight, per-branch developer tool. The alternative - having the agent re-research facts on every revision - relocates the cost from infrastructure to inference tokens, and repeated revision cycles consume substantial budget without guaranteeing convergence on correctness.

3. Core Analysis

3.1 The Tracetune Research Loop and Its Limits

Tracetune operates through an iterative loop: locate all traces associated with a given branch, research those traces, review the drafted description against them, add further context where gaps appear, and repeat until sufficient evidence exists to justify a quality PR. This loop measurably improves description quality relative to naive single-pass generation.

However, iteration alone does not eliminate factual error. In one documented case, a Tracetune-generated PR claimed that "no leader passing run is recorded" and that all checks had passed, while the underlying trace showed bun test and bun x tsc running concurrently - a configuration under which both commands cannot validly have completed successfully, since they require sequential execution. This illustrates that iterative self-review within a single session is insufficient when the agent's own evidence base is itself incomplete or misread. Notably, attempting to control cost by substituting cheaper writer models did not resolve the underlying problem; it degraded PR quality instead, indicating that the bottleneck was not model capability applied redundantly but the structure of the evidence being reasoned over.

3.2 Relate: A Precomputed Evidence Layer

Relate is an open-source, Rust-based, in-memory embedded graph database designed specifically for agent use cases, supporting multi-hop queries, explainable traces, and relationship/provenance tracking without the operational burden of systems like Neo4j. Its defining property is that graph construction and querying occur entirely through local compute and storage, incurring zero LLM or API cost, in direct contrast to the token expenditure of repeated agent-driven research loops.

Relate's output is the neighborhood brief: a connected evidence packet linking the "why," the code, and the checks, together with their sources and attributions. This includes structured evidence such as working-directory logs captured before and after cd commands, exceptions arising from directory-target mismatches, and specific check results (for example, type-checking results paired with a count of passing tests) tied directly to the submitted content. By precomputing this evidence graph once, the system eliminates the need for the writing agent to re-research and re-verify the same facts from scratch on every revision cycle.

3.3 Writer/Reviewer Model Asymmetry

A counterintuitive finding concerns model allocation across the pipeline. Conventional practice assumes that planning, reviewing, and checking roles should use the highest-quality available model, while generation can use something cheaper. The evidence here suggests the inverse for this task: the PR writer should use a higher-quality model (e.g., Sonnet-class or comparable top-tier models), while reviewers can use substantially cheaper models (e.g., Haiku-class).

"It's common that people always want to have the planner, the reviewer, the checker to be the highest quality model, but if you're doing high quality writing, a lower model, no matter how many iterations and how much feedback you give it, can't do it."

The underlying mechanism is that high-quality, human-readable writing is a generative task sensitive to model capability in ways that iteration cannot compensate for; weaker models tend to reduce writing to a mechanical checklist regardless of feedback. Reviewing - cross-checking factual claims against a structured evidence graph - is a comparatively mechanical, pattern-matching task well suited to cheaper models. The resulting pipeline uses a cheap reviewer to check draft claims against the Relate-constructed evidence, trigger rewrites as needed, and then hands off to the high-quality writer to produce the final brief for human approval.

4. Technical Insights

• The Tracetune loop (find traces → research → review → add context → iterate) improves baseline PR quality but does not, by itself, guarantee factual consistency, as demonstrated by the concurrent-test-execution error.
• PR Bench, a custom benchmark combining audit traces, code, and LLM judges cross-checked by human review, scores PR quality on a 1-100 scale and was used to evaluate Tracetune with and without the Relate graph layer.
• Side-by-side testing on Claude and Codex showed quality improvements from approximately 74→76 and 82→86 when adding the Relate layer to Tracetune.
• Token usage for Tracetune plus Relate was roughly half that of Tracetune alone, attributable to precomputed evidence removing redundant re-research across revision cycles.
• Running PR Bench evaluations in parallel across many PRs and agents, rather than sequentially, substantially increased compute demand, to the point of saturating the thread capacity of a high-end consumer workstation - an operational consideration for teams adopting similar parallel evaluation pipelines.
• The approach's zero-LLM-cost graph construction is a key trade-off lever: shifting work from token-metered inference to local, in-memory compute reduces variable cost even as it adds a fixed engineering investment in graph schema and construction logic.

5. Discussion

These findings point to a broader pattern in agentic system design: tasks that require relational reasoning across multiple pieces of evidence benefit from precomputed structure rather than repeated LLM-driven retrieval and synthesis. The Tracetune/Relate combination demonstrates that factual grounding and cost efficiency are not necessarily in tension - in this case, the better-grounded approach was also the cheaper one, because grounding was achieved by moving work out of the token-metered inference loop entirely.

The model-allocation asymmetry identified here - expensive writer, cheap reviewer - also has implications beyond PR generation. It suggests that the common heuristic of allocating the most capable model to "oversight" roles may be misapplied when oversight is reducible to mechanical fact-checking against structured evidence, while the generative role is where model capability is irreplaceable.

A notable gap in the available evidence is the extent to which Relate's graph construction generalizes across codebases with different tooling, test runners, or trace formats; the examples discussed are drawn from a specific toolchain (bun-based testing). Further investigation would be needed to establish how much schema design must be customized per project versus how much can be standardized.

6. Conclusion

This analysis demonstrates that AI-generated PR rejection is frequently an evidentiary problem rather than a code-quality problem, and that this problem can be substantially mitigated by precomputing a relationship graph of session traces, code, and check results before invoking a writing agent. The Tracetune and Relate combination achieved measurable quality gains on PR Bench (74→76, 82→86) while halving token consumption relative to Tracetune alone.

The practical takeaway for teams building agentic documentation or evaluation pipelines is twofold: first, consider whether repeated LLM-driven re-research can be replaced by a precomputed, queryable evidence structure; second, reconsider default model-allocation assumptions, particularly where a generative task (writing) and a verification task (reviewing) are separable, since the stronger model may belong on the generative side rather than the oversight side. The authors note this precomputed-graph pattern is not specific to PR writing and may generalize to other data analysis, evaluation, or curation tasks where agents currently rely on costly, inaccurate re-research.

Sources

• Your PR Says the Tests Passed. Prove It. - Stephen Chin & Cassandra Chin - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub