How We Solved Agent Building - Andrew Qu, Vercel

Verscell's journey building an internal data science agent (D0) evolved from a simple mega-prompt into a multi-agent pipeline, then a single stateful agent, ...

By Sean Weldon

How We Solved Agent Building: An Architectural Case Study of Vercel's D0 Agent

Abstract

This paper synthesizes an internal engineering account of D0, a data science agent developed at Vercel, tracing its evolution through four architectural generations: a static mega-prompt, a role-decomposed multi-agent pipeline, a single stateful agent with an extended step budget, and a sandboxed file-system agent modeled on Claude Code/Opus 4.5. Evaluation data across these iterations show that minimal, generically-trained tool sets outperform prescriptive, task-specific scaffolding - the file-system rebuild doubled evaluation scores relative to the single-agent design, which itself scored only approximately 30%. These findings were generalized into Eve, a convention-driven agent framework analogous to Next.js. The analysis further examines a "skills" system that distills roughly 100 reusable query patterns from production traffic. Findings suggest that company-specific domain knowledge, operationalized through such distilled artifacts, outperforms generic vertical agent products, with implications for enterprise agent strategy across data, legal, and marketing functions.

1. Introduction

The proliferation of Large Language Model (LLM) agents has produced a wide design space with limited consensus on architecture. Practitioners must choose among multi-agent orchestration, where a workflow is decomposed into specialized sub-agents with scoped tools; single-agent designs that manage internal state across an extended step budget; and file-system agents, which expose a minimal, general-purpose tool surface (file listing, file reading, shell execution) over a sandboxed environment. These architectural choices carry measurable consequences for task success rate, error recovery, and long-term maintainability.

This paper synthesizes a longitudinal account of one organization's attempt to answer a concrete operational question: can an agent reliably replace the ad hoc analytical labor of a human data science team? The motivating framing was explicitly generational, echoing Bill Gates' 1980 vision of "a computer on every desk" and asking instead whether there could be "an agent on every desk." Following an internal survey of marketing, sales, finance, and legal functions, the data science team was identified as the highest-value target, since analysts were continuously context-switching to author SQL queries on behalf of other teams.

The central thesis advanced here is that agent capability is maximized not by increasing architectural prescription but by reducing it, while simultaneously increasing the density of organization-specific context available to the model. Section 2 establishes infrastructural background; Section 3 analyzes the four architectural generations and the skills system; Section 4 distills actionable technical findings; Sections 5 and 6 discuss implications and conclusions.

2. Background and Related Work

Vercel's platform lineage moved from web and page hosting toward agent-building primitives, most visibly through the AI SDK, a unified model interface that abstracts provider-specific integration logic. This abstraction is a precondition for rapid architectural iteration, since swapping frontier models becomes a configuration change rather than a rewrite.

The decisive external influence was the release of Claude Code and Opus 4.5. These systems demonstrated that agents trained extensively on a small set of canonical developer tools exhibit strong emergent exploratory behavior: rather than following a designer-specified control flow, they navigate an environment, form hypotheses, and self-correct. This observation directly contradicted the prevailing assumption that finer-grained, semantically precise tools yield better performance, reframing agent design as an exercise in environment construction rather than workflow specification.

3. Core Analysis

3.1 From Mega-Prompt to Multi-Agent Pipeline

The first version of D0 consisted of a raw system prompt containing a full Snowflake schema dump alongside a natural-language question, with the resulting SQL manually copy-pasted and executed. This version established a baseline capability - models could generate valid SQL given sufficient schema context - but lacked guardrails, automation, and any mechanism for iterative correction.

The second version decomposed the data scientist's workflow into discrete roles: query, planning, execution, and reporting agents, each with a dedicated system prompt and scoped toolset (for example, the planning agent was restricted to "read entity YAML" and "search schemas" tools). This improved end-to-end automation from question to answer but introduced an architectural constraint: only a summary or snippet was passed between agents, which limited the context available for downstream reflection and made error recovery difficult when an upstream agent's abbreviated output omitted information a later agent required.

3.2 The Single Mega-Agent and Its Limits

The third version consolidated the pipeline into a single large agent operating with a step budget of approximately 100, self-managing its own internal state across planning, building, execution, and reporting phases. This design improved on the multi-agent pipeline by allowing the agent to reflect on its own prior steps and recover from errors through further exploration or reading, rather than being bottlenecked by inter-agent summaries.

Despite this improvement, initial rollout to trusted users yielded a evaluation score of approximately 30%, described internally as "awful" once users submitted question types the design had not anticipated. This result is significant because it demonstrates that increasing an agent's autonomy within a fixed, prescriptive tool architecture does not by itself resolve generalization failures; the agent could manage state, but the underlying tool surface still constrained what it could discover.

3.3 The File-System Agent Breakthrough

The fourth version was motivated directly by the release of Claude Code and Opus 4.5, which demonstrated that minimal, well-trained tool sets - list file, read file, run bash - produced stronger emergent behavior than prescriptive, semantically-scoped tools. D0 was rebuilt in a sandboxed, file-system paradigm: the semantic layer was dumped into the sandbox, and the agent was given bash, read, and write tools alongside a small number of Vercel-specific tools.

This architectural shift doubled the evaluation score relative to the single mega-agent version. The mechanism cited is that the agent could "use the tools that agents are well trained on" and explore or write work as needed, rather than being confined to a fixed set of task-specific operations. This finding generalizes the core thesis of the paper: reducing tool prescription while increasing environmental context (via the sandboxed semantic layer) outperformed increasing agent autonomy within a constrained tool architecture. A blog post documenting this transition reportedly drove 70% of Vercel.com's traffic during its publication week, indicating substantial external interest in the finding.

3.4 Skills as Distilled Organizational Knowledge

Analysis of thousands of daily queries revealed recurring query shapes - aggregation, lookup, billing information, and similar patterns. Vercel built a recurring job to distill recent queries into reusable "skills," producing approximately 100 such artifacts. Each skill provides new agent runs with pre-established contextual knowledge, obviating the need to reconstruct common query logic from first principles on every invocation. A companion tool, Skillsh, was built as a registry for discovering and executing these skills. This mechanism represents a distinct axis of improvement from the tool-minimization finding in Section 3.3: rather than changing what the agent can do, it changes what the agent already knows, front-loading domain-specific context that would otherwise require exploration to rediscover.

4. Technical Insights

Several implementation-level findings emerge from this account. First, the transition from prescriptive to minimal tool sets is not merely a simplification - it requires constructing an adequately rich environment (the sandbox plus semantic layer dump) so that a general-purpose agent has sufficient material to explore. Removing prescriptive tools without providing this environment would likely degrade performance rather than improve it. Second, the multi-agent pipeline's core failure mode - passing only summaries between agents - suggests that inter-agent context loss is a structural risk in any decomposed architecture, independent of how well each individual agent is prompted. Third, the step-budget increase in the single mega-agent (approximately 100 steps) improved error recovery but was insufficient on its own to resolve generalization gaps, indicating that autonomy and tool surface are separable variables with independent effects on evaluation outcomes. Fourth, the skills system illustrates a maintenance trade-off: recurring extraction jobs require ongoing curation, but they amortize the cost of context discovery across many future agent runs. Finally, the Eve framework's convention-based structure (skills folder, tools folder, channels folder) suggests that once an effective architecture is identified empirically, its patterns can be codified into reusable scaffolding, analogous to how Next.js codified web application conventions after ad hoc React patterns matured.

5. Discussion

These findings carry implications beyond the specific case of D0. The repeated failure of prescriptive tool design - first in the multi-agent pipeline, then in the mega-agent's initial tool architecture - suggests that agent designers may be systematically over-engineering tool interfaces based on assumptions about what a task "should" require, rather than what a well-trained model can already do with general-purpose primitives. This has direct relevance to the current industry trend toward file-system and shell-based agent frameworks.

The skills system additionally suggests that competitive differentiation for enterprise agents may derive less from architecture and more from the density and specificity of domain knowledge embedded in the deployment. This supports the paper's claim that company-specific agents can outperform generic off-the-shelf vertical products, since the latter cannot access an organization's accumulated query patterns, schema idiosyncrasies, or historical corrections.

A limitation of this account is its reliance on a single organization's evaluation metrics, which are not fully specified in terms of methodology, making cross-organizational generalization uncertain. Future investigation could examine whether the tool-minimization finding holds across domains beyond structured data querying, such as legal redlining or marketing analysis, both of which are cited as existing use cases at Vercel.

6. Conclusion

This synthesis documents four architectural generations of an internal agent and identifies a consistent pattern: reducing tool prescription while increasing contextual richness outperforms increasing prescriptive complexity or agent autonomy alone. The file-system paradigm, combined with a distilled skills system, doubled evaluation performance relative to a stateful mega-agent and was subsequently generalized into Eve, a reusable framework for building organization-specific agents. Practically, teams building internal agents should prioritize minimal, well-trained tool surfaces and sandboxed environments over bespoke tool taxonomies, while investing in mechanisms to distill recurring organizational knowledge into reusable artifacts.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub