All Code Is Slop (Except Mine) with Lauren Tan, SpaceXSI

Agentic coding at scale requires software engineers to shift from writing code themselves to curating a trustworthy environment - deterministic codebase constr

By Sean Weldon

Abstract

As coding agents increasingly author production code, the binding constraint on software delivery shifts from writing throughput to verifiable trust. This synthesis examines a practitioner account of agentic development on Grokbot, a production Electron client at SpaceX/xAI, in which a single engineer shipped 3,000 pull requests (PRs) in one month and the broader team shipped 11,000, while the PR revert rate held near 0.5%. The central finding is that quality at this velocity derives from environment curation rather than prompting technique: an inverse pyramid of trust that pushes correctness into deterministic constraints, layers runtime and formal verification above them, and reserves prose rules for genuinely judgment-dependent cases. The analysis shows agents amplify existing codebase patterns, that unenforceable comments function as rationalization escape hatches, and that automation should follow verification maturity rather than precede it. Practical implications include sequencing recommendations for teams building agent-driven development pipelines.

1. Introduction

Prior to the widespread adoption of coding agents, PR volume was a metric of interest mainly for statistical outliers - engineers credited with feats such as rewriting a major mapping product over a weekend. The account under analysis argues this distinction has collapsed: with agents, high output is broadly accessible, and the differentiating skill of software engineering becomes maintaining or improving code quality while productivity increases by orders of magnitude.

Two terms anchor the discussion. A software factory denotes an automated outer loop in which many agents are orchestrated to produce, verify, and merge changes with limited human intervention. Invested trust denotes trust grounded in the agent's environment - checks, constraints, and verification artifacts - as distinguished from unconditional trust in model output. The trajectory to thousands of monthly PRs, some merged while the author slept, reportedly took approximately six months of such investment, though the account suggests this interval is compressible for teams that front-load the right infrastructure.

This synthesis proceeds as follows. Section 2 situates the techniques invoked within established software-engineering practice. Section 3 develops the trust model and its concrete instantiation in the Grokbot codebase. Section 4 extracts implementation-level findings and metrics. Sections 5 and 6 discuss broader implications and recommended sequencing for teams adopting similar practices.

2. Background and Related Work

The techniques invoked are not novel in isolation. Static analysis, type systems, lint enforcement, property-based testing, model checking, and theorem proving are mature disciplines. TLA+ has long served for specifying and model-checking concurrent and distributed protocols; Lean is an established interactive theorem prover; property-based testing descends from generative testing frameworks such as QuickCheck.

The reported shift is economic rather than methodological. These methods historically carried authoring costs that were prohibitive relative to their defect-detection yield, confining them to high-assurance domains such as aerospace or distributed systems infrastructure. Agents reduce the marginal cost of writing specifications, generators, and verification harnesses, which the account characterizes as the onset of a "golden age of software testing" - rigorous verification performed at a scale previously uneconomical for ordinary product engineering. The intellectual contribution is therefore less a new technique than a reallocation: specialist verification methods become default product-engineering practice once an agent, rather than a human, bears the authoring cost.

3. Core Analysis

3.1 The Inverse Pyramid of Trust

The organizing framework is a pyramid inverted relative to the volume of reliance placed on each layer:

The operating principle is that every correction made to an agent should, where possible, be converted into a deterministic check rather than appended as another prose rule, since code-based checks are reproducible and cannot be rationalized away, whereas prose can be reinterpreted or ignored under pressure.

3.2 Codebase as a Garden

A second mechanism concerns the codebase itself as an input. The account reports that pre-agent code review volume was roughly three PRs per colleague; post-agent, that figure rose to approximately 300, converting each engineer into the effective reviewer of a team of agents. Because agents learn local conventions from surrounding code, existing anti-patterns, workarounds, and deprecated idioms are reproduced and propagated - described as spreading "like weeds" - while clean, well-constrained code is copied faithfully. This motivates aggressive codebase hardening prior to automation: useEffect was banned entirely in Grokbot product code except within an internal framework's own APIs, on the grounds that its correct use requires contextual nuance agents handle poorly. Similarly, a dependency-graph analyzer and lint rules enforce a hard boundary between electron-renderer and electron-main directories, preventing main-thread code from leaking into the renderer process - a defect class the account notes has also appeared in other Electron-based editors.

3.3 Verification as the New Code Review

As PR volume scales, human code review shifts from reading diffs to reviewing verification artifacts: screenshots, videos, state-machine diagrams, and formal verification outputs. Supporting infrastructure includes a "feature map" (a site map of application features from the user's perspective, giving agents and reviewers shared vocabulary) and custom CLIs reused across skills to reduce token consumption while remaining independently testable. Formal methods are applied selectively: TLA+ model-checks multi-device notification flows, and Lean is used to prove correctness properties of a custom UI virtualization library susceptible to flickering. A property-based testing library (Bombadil) has reportedly caught multiple UI defects. A cited external account corroborates the pattern independently: a colleague reports that roughly 70% of her daily token usage now goes toward verification rather than generation, a ratio the primary account says matches its own experience.

4. Technical Insights

Several concrete, reproducible patterns emerge from the account:

A limitation acknowledged implicitly: these practices presuppose substantial upfront investment (approximately six months) in constraints and verification tooling before automation becomes net-positive; applying automation to an unprepared, "slop-producing" codebase is reported to generate more defects and token expenditure rather than less.

5. Discussion

The findings suggest a broader reframing of software engineering practice under agentic development: the engineer's locus of leverage moves from individual code authorship to the design of constraint systems that shape agent behavior at scale. This has structural implications for team organization - code review capacity, traditionally a per-PR human bottleneck, is redirected toward verification-artifact review and constraint maintenance, implying different skill requirements (formal methods literacy, static analysis tooling design) than conventional code review.

A notable gap is the absence of detail on how constraint and verification systems themselves are built, maintained, and kept from becoming their own source of technical debt, as well as limited discussion of failure modes beyond reverts (e.g., latent defects not caught by existing verification layers). The account's metrics are self-reported from a single organization and product, and it remains an open question how transferable the specific thresholds (0.5% revert rate, 70% verification token share) are across different codebases, languages, or domains with less mature static analysis ecosystems than a TypeScript/Electron stack.

This account aligns with broader industry discourse on agent-driven development emphasizing "guardrails over prompts," and its application of previously specialist formal methods (TLA+, Lean) to mainstream product engineering is consistent with the general thesis that agents lower the marginal cost of rigor rather than invent new forms of it.

6. Conclusion

The central contribution of this account is a sequencing argument: codebase hardening and verification infrastructure are prerequisites for, not consequences of, high-velocity agentic automation. The inverse pyramid of trust - deterministic constraints at the base, verification in the middle, prose at the apex - offers a practical heuristic for allocating engineering effort when scaling agent-authored code. The reported outcome, 11,000 team PRs in a month with a steady 0.5% revert rate, indicates that volume and quality are not inherently in tension once the right layer of the pyramid absorbs the correction. For teams considering similar adoption, the practical takeaway is to fix the codebase and establish verification maturity first, and treat outer-loop automation as the final step rather than the starting point.


Sources


About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub

Sean Weldon builds apps and custom software at 22Atlas.

Start your build

More about Sean