We Kept Rebuilding Our AI Agent. Here's What Survived with Fuad Ali, Arize AI

Over two years of building and rebuilding Alex, Arize's in-product AI agent, nearly all meaningful improvements came not from newer models but from deliberat

We Kept Rebuilding Our AI Agent. Here's What Survived: A Case Study in Agent Harness Engineering

Abstract

This synthesis examines two years of iterative development on Alex, the in-product agent at Arize AI, as a case study in production agent engineering. Drawing on a documented sequence of rebuilds between late 2023 and mid-2026, it argues that nearly all durable capability gains originated in harness engineering - the custom code surrounding the model that governs planning, turn termination, context management, and tool exposure - rather than in model upgrades. Evidence is organized around five recurring design problems: incremental migration of planning state out of the model, explicit turn-ending control flow, tiered memory anchored by a capped per-user markdown store, consolidation of specialist sub-agents into a single skill-loading agent, and the use of production traces as regression data. Findings indicate that simplification repeatedly outperformed elaboration, and that human judgment remains indispensable for evaluation baselines and latency/quality tradeoffs. Practical implications extend to any team building model-agnostic, long-running production agents.

1. Introduction

The prevailing assumption that agent quality tracks frontier model capability is incomplete. Agent harness denotes the deterministic code surrounding a large language model (LLM): the control loop, planning representation, stopping rule, context budget, memory store, and tool interface. The central claim examined here is that the harness, not the model, carried most of the improvement across repeated rebuilds of a production agent, even as underlying models advanced substantially on their own schedule.

Alex provides an unusually legible case because it was rebuilt from first principles several times while serving real users, and because its developers instrumented every interaction with their own observability product - a practice sometimes called dogfooding. This self-observation extended to meta-tracing: Alex's own debugging sessions were themselves traced and analyzed by a separate internal agent, Signal.

The thesis under examination is twofold. First, that engineering problems in agent design - planning representation, turn-ending logic, memory scoping, and tool exposure - remain open and consequential even as frontier models improve, such that "it's not all up to the model and you can't just run a frontier model and expect it to do everything for you." Second, that human judgment is not fully displaceable by metrics: evaluators and judges require human-labeled baselines, and tradeoffs between speed and quality remain decisions rather than computations. Section 2 establishes the development chronology and theoretical context. Section 3 analyzes four core design problems. Section 4 consolidates implementation-level findings. Sections 5 and 6 discuss implications and limits.

2. Background and Related Work

No third-party agent framework was used at any stage of Alex's development. The control loop, planning representation, stopping rule, and memory subsystem were all custom-built, a choice that made the harness itself the primary unit of iteration and measurement rather than a fixed substrate beneath model swaps. This contrasts with approaches that treat agent frameworks as largely interchangeable scaffolding around an interchangeable model.

The relevant intellectual context is the broader industry observation that harness engineering - rather than prompt engineering or model selection alone - increasingly determines production agent reliability. This is reflected in the system's long-term memory experiments, which tested vector retrieval, episodic and semantic stores, and temporal knowledge graphs before settling on a far simpler mechanism, and in the eventual rollback from ten specialist sub-agents to a single agent with on-demand skill loading. Both trajectories suggest a recurring pattern in which added architectural complexity is tried, measured against production traces, and then partially or fully reversed.

3. Core Analysis

3.1 Planning as a Migration from Model to Harness

Planning illustrates the general pattern most clearly. The design space can be framed by analogy to delegating chores to a child: handing over a complete list, checking in continuously, or retaining the list and revealing one item at a time.

Granting the model a to-do tool produced a characteristic failure: the agent expended most of its LLM calls maintaining and checking off list items rather than performing substantive work, since "models can be very eager with lists." Focus also degraded measurably beyond roughly twenty tool-call iterations.

The remedy was incremental relocation of plan state from the model into the harness rather than a single redesign. An "in progress" marker was first added to the rendered plan; subsequently, the plan was rendered as a message alongside the data on every model call, keeping state externally visible rather than internally tracked. The "in progress" marker was later removed once remaining items were stack-ranked by default, since the signal had become redundant. At the time of analysis, the team was testing removal of the to-do tool entirely, keeping plan state fully in the harness and message history - a trajectory that moves monotonically toward less model-managed state, not more.

3.2 Turn-Ending as Explicit Control Flow

A second recurring problem was determining when the agent had finished a turn. The initial heuristic - treating "text without tool calls" as a completion signal - broke under models that announce a step in one turn and execute the corresponding tool call in a later turn. Because users can select their own model provider, the harness cannot assume a single vendor's calling conventions and must guard against inconsistent behaviors across models.

The solution was an explicit finish call that bounces back to the agent if plan items remain open, retried up to three times, paired with a warning issued one step before the iteration limit followed by a forced finish. This converts an implicit, model-inferred signal into explicit, harness-enforced control flow - a pattern consistent with the planning migration in 3.1.

3.3 Memory as Tiered, Capped, and Auditable

Memory management addressed two distinct problems: managing within-session context and managing long-term, cross-session user history. The within-session problem is described by analogy to a desk: an oversized tool result is a book too long to read; multiple such results are multiple books filling the available desk space. The adopted solution stores large tool results externally and retains only a "sticky note" - a pointer or summary - in context. For large JSON traces specifically, content is truncated and the agent is given code tools (e.g., JQ) to query specifics on demand rather than receiving the full blob. For long trajectories, the harness forces the model to write a checkpoint and drops replaced history under a fixed token budget.

The long-term, cross-session memory problem took a more notable trajectory. After testing vector retrieval, episodic/semantic stores, and temporal knowledge graphs, the team converged on a single markdown file per user, capped at 8,000 characters, loaded on every request and updated by a background job after each session. As summarized in the source material, "a small file actually beat an all graph." Beyond raw accuracy, the capped-file approach has an organizational benefit absent from the graph-based alternatives: its small, auditable size allows human labelers to review and directly correct stored content, and to adjust the background job's writing instructions accordingly - an evaluation affordance that more opaque structured stores do not offer as readily.

3.4 Architectural Consolidation: From Specialists Back to a Single Agent

The system progressed from a single trace-analysis agent in 2025 to ten page-specific specialist sub-agents, then rolled back to a single agent that loads skill instructions and tools on demand (One Alex, June 2026). This reversal is attributed jointly to improved underlying model capability and a deliberate drive toward simplification, with the explicit acknowledgment that "as engineers, we love to build things... sometimes we get in our own way and we over-engineer." The consolidation parallels the memory finding in 3.3: a structurally simpler design outperformed a more elaborate one once model capability crossed a sufficient threshold.

4. Technical Insights

Several implementation-level findings generalize beyond this specific system. Mathematical and numeric computation was kept out of the model and placed in code from the earliest prototype onward, with the model reasoning and explaining over precomputed results rather than performing raw arithmetic; this decision, made under early model limitations, survived every subsequent rebuild. Relatedly, capabilities are exposed to the model as discrete tools rather than as plain-text questions, which constrains how the model can access data and computation.

On model selection, the newest available model is explicitly not treated as the default best choice; "best" is use-case dependent rather than a fixed benchmark ranking, and the stated default strategy is to choose fast, good-enough models and invest engineering effort in the harness instead. The team was exploring auto-routing (selecting models per task) and training small, post-trained SLMs for narrow surfaces - for example, AI-based filter generation now works well with very small models, sometimes without an agentic loop at all, using a single model call. This suggests a trade-off consideration for practitioners: agentic loops and large models are not free defaults but costs to be justified per task.

On debugging, agent failures rarely surface as explicit errors. Observed signals instead include a nominally "ok" status accompanying 43 identical tool calls in a single turn, or a 227-second latency - with "too long" treated as use-case dependent rather than a fixed software metric. Full production tracing, feeding into regression data sets replayable turn-by-turn, and a dedicated trace-reading agent (Signal) were used to reduce the manual trace-review burden on staff engineers over time.

5. Discussion

These findings collectively suggest that durable agent quality is produced by narrowing, not expanding, the model's discretion over control flow and state - plans rendered externally, turns ended by explicit harness logic, memory capped and externalized, and specialist decomposition rolled back into a single on-demand architecture. This runs counter to an intuition that more capable models warrant delegating more responsibility to them; instead, capability gains appear to have been redirected into supporting simpler, more constrained harness designs (e.g., a single agent instead of ten specialists) rather than into granting the model more autonomy.

A persistent limitation is that evaluation itself resists full automation. Golden data sets, gold labels for judges, and the human baseline required to align any automated judge with engineer-level correctness all originate from human labeling. Likewise, speed-versus-quality tradeoffs and cross-cutting architectural decisions (such as standardizing on a single model like Astra everywhere) remain human judgment calls even when metrics quantify the tradeoff space - metrics inform but do not resolve these decisions.

An open question, not resolved within the source material, is how auto-routing and task-specific SLMs will interact with the harness-centric philosophy once they mature: whether model selection itself will eventually be absorbed into harness logic, or whether it will remain a separate, judgment-driven layer.

6. Conclusion

The two-year history of Alex indicates that, in this production setting, harness engineering - not model upgrades - accounted for the overwhelming majority of durable capability improvements, manifesting as incremental migrations of state and control flow out of the model and into deterministic code. The practices that survived every rebuild - code-based computation, tool-based capability exposure, golden data sets in CI, streaming and persistent sessions, and skills as a concept - point toward a stable core of harness design principles independent of any particular model generation. For practitioners building production agents, the practical takeaway is to treat planning, turn-termination, and memory as harness responsibilities first, to default to fast, adequate models while reserving engineering effort for the surrounding system, and to preserve human-auditable, human-correctable structures (such as a small capped memory file) over more elaborate but opaque alternatives.

Sources

• We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub