How to Build a Software Factory That Improves Itself with Zach Lloyd, Warp

Companies moving beyond interactive AI coding assistants should build "cloud software factories" - infrastructure-grade, code-defined, measurable systems whe

Abstract

This synthesis examines a proposed architectural transition in agentic software development: the movement from interactive prompting, in which engineers invoke coding agents locally, toward automated development mediated by a cloud software factory. The central thesis holds that organizations should treat agentic coding capacity as infrastructure - open, adaptable, and defined in code - rather than as a product or "AI teammate." The analysis covers the security and governance deficits of local agent execution, the properties required of a code-defined factory, a measurement-first evaluation methodology substituting organization-specific benchmarks for public leaderboards, and the emergence of factory engineering as a discipline distinct from product engineering. Evidence includes cost-versus-quality routing data and a self-improvement loop in which scoring agents detect recurrent failure patterns and propose corrective changes to the factory definition itself. Practical implications concern team sizing, cost accounting, and verification methodology.

1. Introduction

The deployment of large language model (LLM) coding agents has, in most organizations, settled into a pattern described as interactive prompting: individual engineers run tools such as Claude Code or Codex directly on laptops, prompting agents conversationally within a local environment. This arrangement produces measurable individual productivity gains but resists organizational scaling, because every invocation is gated on a human initiating and supervising it.

The research question addressed is structural rather than model-centric: what infrastructure must exist for an organization to maximize the business value of agentic development at the lowest cost, in a governed and secure manner? The proposed answer is the cloud software factory - a cloud-hosted loop in which humans and agents collaborate across the full software development lifecycle, governed by explicit rules for human-agent interaction and continuously measured against its own performance data.

Section 2 establishes the limitations of local agent execution that motivate this shift. Section 3 analyzes the factory concept across four themes: infrastructure framing, code-defined specification, measurement methodology, and organizational division of labor. Section 4 presents implementation-level findings from a reference platform (Warp Factories). Sections 5 and 6 discuss implications and open questions.

2. Background and Related Work

Three failure modes of local, interactive agent execution motivate the factory architecture. First, a throughput bottleneck: because a human must run the agent, capacity is bounded by engineer attention, and cost control and governance become difficult as spend accrues across uncoordinated individual sessions. Second, a security surface: a local agent operates with the ambient authority of its signed-in user, so a compromised agent can take over whatever that user has access to. Third, governance opacity: each engineer independently configures MCP servers, skills, and tools, producing heterogeneity with no audit visibility into agent behavior or data access.

"The solution to both these problems is to sandbox your agents and move them to the cloud."

The relevant prior art for measurement is twofold. Public coding benchmarks such as SWE-bench and Terminal-bench, along with informal model claims circulated on social media, supply comparative signal but are argued to be unreliable proxies for performance on any specific codebase. Separately, DORA metrics supply an established vocabulary - notably cycle time - for assessing delivery performance, repurposed here as factory-level telemetry alongside automation rate and spend.

3. Core Analysis

3.1 Factories as Infrastructure Rather Than Product

The central framing argument is that a software factory should be built and consumed as infrastructure rather than purchased as a packaged "AI teammate" product. Three properties are specified. Open means support for multiple models and harnesses, open-weight or frontier, since the best available model or harness changes frequently and lock-in to one vendor's choice is a liability. Adaptable means the factory integrates with a company's existing infrastructure rather than requiring wholesale replacement. Defined in code means the factory's workflows, skills, MCPs, and execution environments are captured in a strict, version-controlled definition, analogous to infrastructure-as-code practices elsewhere in the stack.

This code-defined property has a compounding consequence: because the factory definition is itself code, it can be tested across versions, and - critically - it can be modified by coding agents. The factory becomes an artifact that improves itself through the same mechanism it uses to ship product code. This is tied to a broader claim of AI sovereignty: owning data in and out of the system, running agents on owned infrastructure and compute, controlling inference, and preserving optionality across models and harnesses rather than being bound to a single vendor's roadmap.

3.2 Measurement-First Evaluation Over Public Benchmarks

The analysis argues that public benchmarks and informal social-media claims about model performance are poor guides to model selection for a specific codebase, since task distributions, languages, and conventions differ across organizations. The proposed alternative is a measurement-first approach: score every coding-agent run, feed those scores into agents tasked with improving the factory, and construct custom benchmarks from an organization's own historical tasks.

A concrete illustration is given in the cost-versus-quality comparison: GLM 5.3 Flash scored 93% correct at a cost of $2 per task set, while GPT 5.6 scored 95% correct at approximately three times the cost. This two-point accuracy gain at a threefold cost premium is presented as the type of organization-specific signal that enables a model routing strategy - assigning cheaper models to tasks where the marginal accuracy loss is acceptable, and reserving costlier models for higher-stakes work. Such routing decisions are argued to be impossible to derive from public leaderboards alone, since they depend on the organization's actual task mix and cost tolerance.

3.3 Factory Engineering as a Distinct Discipline

The framework distinguishes two categories of engineering work once a factory exists. Product engineering continues largely unchanged in purpose - building features and fixing bugs for customers - but is routed through the factory rather than through ad hoc local agent sessions. Factory engineering is a new and distinct discipline in which platform teams or designated engineers improve the factory itself: its skills, context provisioning, model mix, and workflow routing. This division implies an organizational commitment: factory engineering requires dedicated ownership, not an assumption that the factory will self-maintain without investment, even though the self-improvement loop (Section 4) automates part of that maintenance.

3.4 Implementation Findings from the Warp Factories Platform

The reference platform exposes a manager- or factory-engineer-facing dashboard reporting productivity, automation level, DORA-style cycle time, and spend, with all agent traces and artifacts from every engineer and automation stored centrally. This centralized storage serves dual purposes: audit visibility, addressing the governance opacity problem identified in Section 2, and raw material for subsequent factory improvement.

Work enters the factory through engineers' existing tools - Slack, Linear or Jira, GitHub PR comments - routing into a centralized orchestrator agent that triages work to spec, implementation, or code review steps depending on context. The entire factory definition (agent definitions, secrets, MCPs, prompts, models, automations, benchmarks, skills, webhooks) lives in a version-controlled repository, operationalizing the "defined in code" property from Section 3.1.

Online evaluation is performed via LLM-as-judge scoring of agent runs, which was reported to detect a common coding-agent failure mode: writing redundant tests. In one cited instance, scoring agents aggregated failure patterns across runs and identified a skill-matching failure occurring 37 times, then proposed a pull request to correct the factory definition - demonstrating the self-improvement loop described in Section 3.1 operating in practice. Past runs can additionally be curated into benchmark suites and replayed against alternative model configurations to regenerate cost-versus-quality comparisons as models and prices change.

4. Technical Insights

The infrastructure stack is described in layers, from the bottom up: a code-defined data layer governing data in and out; a sandboxed execution environment; a flexible inference layer supporting multiple models; a self-improvement and scoring layer; an orchestration layer for multi-step workflows; and an access layer integrating existing engineer-facing tools. This layering is presented as a vendor-agnostic requirement rather than a feature specific to any single product.

Verification methodology receives specific attention. Recommended practice combines spec-driven development, in which changes are checked against an explicit specification, with computer-use verification, in which the agent produces direct evidence - such as a browser-automated demonstration - that a feature behaves as intended. This is argued to reduce the burden on human code review by shifting correctness evidence earlier in the pipeline.

Budgeting guidance centers on measuring cost per pull request and the proportion of spend allocated to scoring and benchmarking, then extrapolating from that baseline rather than from vendor pricing sheets. Benchmark curation can proceed either manually, by selecting representative small, medium, and hard tasks, or by directing the factory itself to surface interesting past sessions from its stored history. A noted limitation is that correctness in current benchmark charts is driven entirely by LLM-as-judge scoring; human grading was acknowledged as a reasonable but not-yet-implemented addition, leaving an unresolved question about judge reliability. Team-size guidance suggests the approach is accessible to small teams but is characterized as overkill for individual developers and most valuable for teams of approximately ten or more engineers.

5. Discussion

The factory framing reframes agentic coding tooling as an infrastructure investment with ongoing engineering cost, rather than a one-time procurement decision. This has organizational implications: it requires dedicated factory engineering capacity, a version-controlled artifact subject to the same rigor as application code, and a measurement system before meaningful optimization is possible. The claim that "the world you want to get to as you automate more development is actually being measurement-first on your own data" positions internal evaluation infrastructure as a prerequisite for, rather than a byproduct of, automation scaling.

A notable gap is the reliance on LLM-as-judge scoring without human-in-the-loop validation of judge accuracy, which introduces a dependency on the correctness of the evaluating model that is not independently verified in the material presented. Similarly, the reported self-improvement example (37 recurrences of a skill-matching failure) demonstrates pattern detection but does not establish the long-run reliability of agent-authored changes to the factory definition itself, a question with direct bearing on governance.

The emphasis on model and harness optionality reflects a broader industry trend toward decoupling agentic workflows from any single vendor's model, consistent with rapid model iteration cycles observed across the field.

6. Conclusion

This analysis has characterized the cloud software factory as an infrastructure-grade alternative to local, interactive agent use, distinguished by openness to multiple models and harnesses, adaptability to existing company systems, and a strict code-defined specification enabling testing and agent-driven self-modification. Measurement-first evaluation, grounded in organization-specific cost-versus-quality data rather than public benchmarks, is presented as the mechanism enabling informed model routing decisions.

The practical takeaway for organizations of roughly ten or more engineers is to prioritize building internal scoring and benchmark infrastructure before scaling automated agent usage, and to treat factory engineering as a distinct, resourced discipline rather than an incidental byproduct of adopting agentic tools.

Sources

• How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub