The Agents Were Fine. Everything Around Them Broke. with Joey Wang, Harvey

As AI coding agents proliferate, the real engineering challenge and competitive differentiator is not the model or agent itself but the surrounding harness -

By Sean Weldon

Abstract

As coding agents built on large language models become capable enough to generate production pull requests (PRs) at scale, the operational bottleneck shifts from model quality to the infrastructure that runs, verifies, and merges agent output. This synthesis examines Spectre, an internal cloud agent harness deployed across a software engineering organization, using operational data spanning approximately 10,000 agent-authored PRs and 460 monthly active users. Three narratives are analyzed: completion of the full delivery loop beyond code generation; command-line interface (CLI) parity between human and agent actors; and failure modes observed under concurrency load, including two saturation incidents and a 60% company-wide surge in PR volume. Findings indicate that reliability, fairness, and merge throughput - not model capability - constitute the dominant engineering risks. A fair launch queue reduced task start latency from roughly 2 seconds to 70 milliseconds under a 300-task burst. The harness, not the underlying agent, emerges as the durable competitive asset.

1. Introduction

The rapid adoption of autonomous coding agents - systems built on large language models (LLMs) capable of reading, modifying, and committing source code without step-by-step human direction - has outpaced the infrastructure typically used to supervise them. Tools such as Claude Code, Cursor, and Codex now operate competently on individual developer workstations. However, this mode of deployment is bounded: empirical experience suggests a practical ceiling of five to ten concurrently running agents per machine, constrained by credential management, compute availability, and the attention required to supervise output.

Beyond that ceiling, a distinct architectural layer becomes necessary - what this analysis terms the agent harness: infrastructure responsible for secret custody, provisioning of realistic staging environments, ingestion of alerts and tickets, enforcement of a uniform policy set across human and machine actors, and sufficient scheduling capacity that no participant is starved of resources. The central claim examined here is that this harness, rather than the agent or the underlying model, constitutes the primary engineering challenge and the locus of durable differentiation.

"We don't compete with them. We run them."

This paper first situates the harness architecturally (Section 2), then analyzes three operational narratives - completion loops, interface parity, and load behavior - drawn from a deployment at scale (Section 3). Section 4 distills technical findings with implementation implications, and Section 5 discusses open problems, including unresolved merge-throughput constraints under rising automated PR volume.

2. Background and Related Work

Spectre is deliberately positioned beneath the coding agent layer rather than in competition with it. Engineers select whichever agent they prefer and push the resulting session into cloud execution, where the harness supplies environment provisioning, policy enforcement, and lifecycle management. This separation allows agent-level heterogeneity while centralizing governance and auditability - a design pattern increasingly relevant as organizations adopt multiple commercial and in-house agents simultaneously.

Several recurring constructs frame the deployment. The follow-up loop is an automated remediation cycle that reads continuous integration (CI) failures and issues corrective commits without human prompting. The fair launch queue is a scheduling mechanism that enforces per-actor resource fairness under contention. The CLI-parity principle - "If a person can click it, an agent can call it" - governs interface design by requiring every user-facing action to be exposed programmatically. Skill projects serve as structured knowledge-capture artifacts that encode tacit operational knowledge (e.g., how to test a fragile service, which configuration flags not to touch) for reuse by both agents and humans, producing an auditable record of authorship. Notably, usage of the harness extended beyond software engineering during the observed period, encompassing Slack-based data queries and a marketing team's generation of a product video storybook - suggesting the harness abstraction generalizes beyond code production.

3. Core Analysis

3.1 Finishing the Job: The Completion Loop

The most consequential design decision observed is the extension of agent responsibility past code authorship into verification and remediation. In the canonical loop, an alert fires in Slack at 2 a.m.; an agent claims it without human initiation, authors a fix, provisions a live replica of the affected stack, and posts validation evidence directly on the PR. A secondary loop then reads CI failures and pushes additional corrective commits automatically while human engineers are offline. By morning, the resulting PR enters the same review queue, under the same rules, as human-authored work, and - once approved - can be automerged.

This follow-up loop was enabled company-wide in early October. In the following month, four out of five agent PRs merged into the backend repository had passed through it, indicating rapid and near-total adoption once the mechanism was available to all engineers rather than a subset. This pattern supports the broader thesis that the harness's value lies not in generating a fix but in closing the loop between failure detection, remediation, and safe integration.

3.2 Any Interface, Any Agent, Same Rules

A policy established in May - "If a person can click it, an agent can call it" - required every platform action (starting a task, toggling the follow-up loop, enabling automerge, publishing a knowledge page) to be exposed through a CLI. This parity rule enabled laptop-based agents, sandboxed cloud agents, and CI pipelines to invoke identical actions through a common interface, removing the need for agent-specific integration work.

The effect was substantial: CLI-originated tasks grew to approximately 6,500 per month by October without displacing web application usage, indicating the emergence of a new class of terminal-only power users who never interact with the graphical interface. One documented usage pattern involved a coding session functioning as a "coordinator," decomposing a large task into roughly ten milestones, delegating most to Spectre as hundreds of independently sandboxed tasks, and later using a separate agent to monitor progress, draft responses to clarifying questions, and route them back for human approval. This illustrates a shift from single-agent task execution toward multi-agent orchestration patterns in which the harness functions as shared infrastructure rather than a single-agent execution environment.

3.3 Reliability and Fairness Under Load

Two saturation incidents expose the gap between agent capability and system-level reliability. In mid-August, a buggy release generated a per-workspace alert, producing hundreds of alerts in a single Slack channel; an automated workflow tagged an agent on each one. Normal daily volume was approximately 50 tasks; during the incident, nearly 900 tasks started within roughly four hours, all addressing the same underlying bug and crowding out unrelated work. The remediation was architectural rather than punitive: a repeat alert now joins an already-open investigation instead of spawning a new task.

Two months later, a human engineer - acting within encouraged usage patterns - submitted approximately 950 tasks via the CLI in roughly three hours, saturating the cluster and blocking all other users. This incident was resolved within four days by introducing a fair launch queue, which assigns each task a queue position, dispatches tasks as capacity allows, prioritizes active human users, and allocates shared capacity equitably with visible queue status. Under a controlled load test simulating a 300-task burst, this mechanism reduced task start latency from nearly 2 seconds to approximately 70 milliseconds. Comparing the busiest observed days in August and November, task non-completion fell from roughly 1 in 10 to a markedly lower rate despite more than double the task volume, though the organization notes that simultaneous fixes prevent attribution of improvement to any single change.

4. Technical Insights

Several implementation-level findings carry direct relevance for teams building similar infrastructure:

5. Discussion

These findings collectively indicate that as coding agents approach commodity status, competitive and operational risk migrates toward the systems that govern their execution. Model selection becomes a configurable input rather than a differentiator, while harness-level concerns - provisioning latency, fairness under contention, audit trails, and merge integration - determine whether increased agent throughput translates into increased delivered value or into system-wide contention.

A significant open problem is merge throughput itself: as agents generate PRs at increasing volume, the prevailing CI-rebase-and-merge model couples every pending PR's success to the stability of the main branch, creating a scaling bottleneck that none of the reported fixes directly address. Related unresolved questions include whether every communication channel should host an always-on, proactively acting agent, and whether PRs should be required to carry verification evidence before human review - a practice currently applied to only 1 in 20 PRs. These gaps suggest that verification infrastructure, not generation infrastructure, may be the next area requiring investment.

For practitioners, three recommendations generalize beyond this specific deployment: ownership of post-generation processes (testing, remediation, merging) should be treated as core engineering scope rather than an afterthought; every user-facing action should be programmatically callable to support heterogeneous agent and human usage; and caps on initiation rate - whether by human, agent, or automated alert - should be designed in before scaling usage, rather than retrofitted after a saturation incident.

6. Conclusion

This analysis supports the thesis that, in an environment of increasingly capable coding agents, the engineering challenge of consequence is not agent output quality but the surrounding harness that executes, verifies, and integrates that output at scale. The Spectre deployment demonstrates that straightforward infrastructure interventions - repository caching, fair scheduling queues, automation guardrails, and CLI-parity design - yielded measurable reliability and latency improvements, while merge-throughput constraints under rising agent-driven PR volume remain unresolved. For organizations adopting coding agents at scale, the practical takeaway is to invest commensurately in the completion, verification, and fairness layers of the pipeline, since these - rather than the choice of agent - determine whether agent-generated work is delivered safely and sustainably.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub

Sean Weldon builds apps and custom software at 22Atlas.

Start your build

More about Sean