How to Test Your Agents on Real Tasks With Harbor with Xander Song, Arize AI

Harbor is an open-source framework for evaluating AI agents against realistic, sandboxed tasks with implementation-agnostic verifiers, enabling teams to iter

Shipping Agents on Evidence: An Analysis of Harbor's Sandboxed Evaluation Architecture

Abstract

Agent development frequently proceeds without a reliable mechanism for distinguishing improvement from regression when prompts, tools, or harnesses change. This synthesis examines Harbor, an open-source framework for evaluating AI agents against realistic tasks executed in isolated, Dockerized sandboxes seeded with real data. Harbor's design holds task instructions and verifiers constant while permitting agent implementation to vary, producing directional signal on each change - a discipline its creators describe as "shipping based on evidence rather than vibes." This analysis examines Harbor's pipeline architecture, implementation-agnostic verification semantics, horizontal scaling model, and integration with the Phoenix observability platform via ATIF and Open Inference. A production case study on SQL tool adoption demonstrates that a single prompt revision reduced average tool calls and conversational turns while holding pass rate constant. Practical implications include scalability to 200 concurrent sandbox workers, interoperability with OpenTelemetry-based tracing standards, and the identification of realistic data acquisition as the dominant engineering cost in building evaluation suites.

1. Introduction

As agentic systems absorb more of the software surface - coding assistants, server-side task runners, tool-augmented retrieval layers - the question of how to verify that a change actually helped has become acute. Unlike conventional software, where unit tests provide deterministic feedback, agent behavior is stochastic, environment-dependent, and mediated by large language models whose responses to prompt perturbations are difficult to predict in advance. The practical consequence is that many teams iterate on agents through manual spot-checking rather than systematic measurement.

Harbor is positioned as a corrective to this practice. It is an open-source framework for evaluating agents on real tasks with real data inside isolated sandboxed environments. Its pipeline is straightforward in outline: natural language task instructions are given to an agent operating in a fresh sandboxed environment seeded with real data; after the agent runs, task-specific verifiers inspect the resulting side effects and produce a reward - typically a binary pass/fail signal. The framework's leverage comes from a controlled-experiment structure: task and verifier are held fixed while the agent implementation (model, prompt, harness, skill set, or tool surface) is varied, isolating the effect of that change.

This analysis is organized as follows. Section 2 situates Harbor relative to integration testing and AI observability practice. Section 3 examines Harbor's architecture, verification semantics, scaling model, observability integration, and a production case study on SQL tool adoption. Section 4 distills actionable technical findings. Sections 5 and 6 discuss broader implications and conclude.

2. Background and Related Work

Harbor draws on two established traditions. The first is integration testing, in which a system under test is exercised through its public interface and graded on observable outcomes rather than internal mechanism - mirroring, as the source material notes, tests that are "agnostic of the implementation behind an API." The second is AI observability, the instrumentation of LLM applications to capture traces of model and tool invocations for debugging and analysis.

Two specifications anchor Harbor's observability layer. ATIF (Agent Trajectory Interchange Format) is a JSON-based tracing specification for recording agent steps - LLM calls and tool calls - intended for debugging and logging. Open Inference is a semantic-conventions specification built atop OpenTelemetry (OTLP) for AI application tracing, recently donated to the CNCF/OTel working group. Harbor-emitted ATIF trajectories are convertible into Open Inference format, enabling ingestion by downstream platforms. Phoenix, Arize AI's open-source observability, experimentation, and evaluation platform, is one such consumer: it maps Harbor's primitives - data sets, experiments, trials, rewards (termed evaluations) - onto its own vocabulary and supplies versioning, experiment tracking, and metric visualization via UI, CLI, skills, and an MCP server. Harbor also ships its own UI and hosted web service, making Phoenix integration optional rather than required.

3. Core Analysis

3.1 Architecture and the Verification Contract

Each Harbor task is specified by three artifacts: a markdown file containing natural language instructions, an environment (typically a Dockerfile) seeded with relevant data - a SQLite file, a git repository pinned to a commit - and verifiers located in a test directory. Tasks are grouped into data sets, and each task can be run across a configurable number of trials to account for the stochasticity inherent to LLM-driven agents.

The execution sequence per trial is deliberate: a fresh Docker container is built and seeded with data, the agent is dropped in and allowed to run, and only after the agent finishes does Harbor mount the verifiers. This sequencing is a structural safeguard against reward contamination - the agent cannot inspect or game the grading criteria because the criteria are not present in the environment during execution. Verifiers write their result to a reward.txt file, described as "the most important artifact in the pipeline." A full run of a data set - tasks × trials × verifiers × trajectories - constitutes a job.

Harbor also supports an oracle pattern: a known-correct solution script can be dropped into the environment in place of an agent. This validates that the task-and-verifier pipeline itself is correctly wired before any tokens are spent on real agent runs, separating pipeline-correctness bugs from agent-performance questions.

3.2 Implementation-Agnostic Verification

Canonical Harbor verifiers treat the agent as a black box, grading solely on side effects and output state in the environment rather than the method used to produce them. This property enables what the source material calls refactoring "under green" - changing agent implementation (model, prompt, tool set) while verified functionality is held constant, analogous to refactoring code under a passing test suite. It is explicitly framed as a departure from mocking-based testing strategies, which couple tests to implementation detail.

Verifiers can optionally inspect the ATIF trajectory directly - for instance, to confirm a specific skill or tool fired - but this is acknowledged as a form of "cheating" that increases coupling between test and implementation. The trade-off is accepted selectively: trajectory inspection is useful when the test's explicit purpose is to confirm skill-firing behavior rather than general task success, trading test purity for specificity.

3.3 Scale, Parallelism, and Metrics

Because each task/trial executes in its own independent Docker container, the evaluation pipeline is horizontally scalable by construction. Harbor is compatible with multiple sandbox providers, including Daytona, Modal, and E2B; the team reports using Daytona with 200 concurrent workers to achieve fast execution across large data sets. Aggregate metrics, including pass@k, can be computed per task and across an entire data set, giving a rapid read on whether a change produced net improvement or regression across the full capability surface being tested.

3.4 Case Study: SQL Tool Adoption

A production example illustrates the evidence-based iteration loop in practice. The team's architecture - a single-node Phoenix server with a SQLite backend exposed via REST and GraphQL - was tested across three agent surfaces: a coding agent via CLI, a coding agent via MCP server, and a server-side agent ("Pixie") equipped with a bash emulator. The initial problem was that users asked aggregate questions (e.g., busiest day, cost per model) poorly served by REST/GraphQL, causing agents to paginate and over-call tools to assemble answers.

The remediation combined a tool change and a prompt change: a read-only SQL query tool was added to the MCP server, and prompts were updated to direct agents toward SQL for aggregate, group, and count queries rather than GraphQL traversal. Claude was used to brainstorm representative user queries across domains (traces, spans, data sets, experiments) and to write oracle solutions, accelerating construction of a synthetic evaluation data set.

Comparing two Opus 5 jobs run before and after the prompt change in Phoenix showed: SQL tool usage increased, GraphQL tool calls decreased, average tool calls per trajectory dropped, and the number of conversational turns decreased - while the reward (pass/fail) rate remained stable and high. This demonstrates the core value proposition: a behavioral efficiency gain was confirmed quantitatively, with individual task traces available for drill-down verification rather than relying on spot-checking.

4. Technical Insights

Several implementation details carry direct engineering weight. First, the post-execution verifier-mounting pattern is a simple but effective anti-cheating mechanism that generalizes beyond Harbor to any agent-grading pipeline. Second, the oracle pattern - substituting a known-correct script for the agent - is a low-cost way to isolate pipeline bugs from agent-performance variance before committing token budget to full agent runs. Third, Harbor's "contract" with the host environment is narrow: scripts for task start, verification, and oracle solutions must be placed in conventional locations within the Docker image, which keeps the orchestrator itself lightweight (it runs on the host, not inside the sandbox).

A notable limitation is the difficulty of testing distributed systems - the source material cites S3-backed components as harder to evaluate than single-node systems, with modularization behind a stable interface suggested as a mitigation, though not yet solved by the team. Realistic data acquisition is repeatedly identified as the dominant cost: "getting the data is the hard part for any of these tasks," since more realistic data surfaces more realistic failure modes. The team also reports being in early stages of regression testing, currently scoped to "happy path" tests organized around skills and workflows across models and providers, rather than exhaustive adversarial coverage.

5. Discussion

Harbor's design reflects a broader convergence between software testing discipline and agent evaluation practice. The implementation-agnostic verifier pattern - grading side effects rather than method - directly imports the integration-testing philosophy into a domain where behavior is nondeterministic, suggesting that classical software engineering discipline is adaptable to LLM-driven systems with modest structural changes (fresh containers per trial, trial repetition for statistical confidence, reward aggregation via pass@k).

The interoperability layer, built on ATIF and Open Inference/OpenTelemetry, is notable for linking an agent-specific evaluation framework to the general-purpose observability ecosystem rather than requiring a bespoke visualization stack. The recent donation of Open Inference to CNCF/OTel suggests movement toward standardized AI tracing conventions, which could reduce future integration friction between evaluation frameworks and observability platforms.

An open question, acknowledged directly by the source material, is how this evaluation discipline extends to distributed, multi-service systems where seeding realistic state is substantially harder than for single-node SQLite-backed applications. The SQL-tool case study succeeded in part because the underlying system was simple enough to Dockerize wholesale; more complex production systems may require architectural concessions - stable interfaces, service virtualization - to remain testable in this paradigm.

6. Conclusion

Harbor operationalizes a discipline of evidence-based agent iteration by holding task instructions and verifiers fixed while varying agent implementation, converting agent development into a measurable optimization loop rather than a trial of intuition. Its architectural commitments - per-trial container isolation, post-execution verifier mounting, oracle-based pipeline validation, and implementation-agnostic grading - collectively enable confident refactoring and tool/prompt iteration at scale, as demonstrated by the SQL-tool case study's quantified efficiency gain at constant pass rate.

The practical takeaway for engineering teams building agentic products is that the primary investment is not in the orchestration layer, which is comparatively lightweight, but in constructing realistic task data sets and verifiers that faithfully represent production conditions. Teams considering adoption should expect data acquisition and verifier design to dominate implementation effort, with distributed-system testing remaining an open engineering challenge.

Sources

• How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub