What if the harness mattered more than the model? - Aditya Bhargava, Etsy

Building excellent harnesses (the infrastructure around AI models) matters more than the models themselves, and language-level support is essential for creat...

By Sean Weldon

Abstract

This synthesis examines the comparative impact of agent harnesses versus model capability on AI agent performance. Analysis of the Harness Bench benchmark reveals that infrastructure quality produces performance variations exceeding 23 percentage points (52.4% to 76.2%) using identical models, with amplified effects observed on weaker models. The research presents Agency, a domain-specific language addressing fundamental limitations in existing agent frameworks through language-level primitives. Key architectural contributions include interrupt-based safety mechanisms enabling pause-and-resume execution, partial function application for capability constraint without explicit permissions, ReAct-pattern feedback loops for iterative verification, and integrated optimization frameworks for systematic improvement. Findings demonstrate that superior harness design enables open-source models to achieve performance parity with proprietary alternatives, reducing dependency on commercial systems while maintaining safety guarantees and measurability. These results challenge prevailing assumptions regarding the primacy of model capability in agent performance.

1. Introduction

Contemporary discourse in AI agent development operates under the assumption that model capability constitutes the primary determinant of agent performance. This paradigm has driven industry focus toward increasingly powerful proprietary models while treating infrastructure as a secondary concern requiring minimal sophistication. The prevailing wisdom suggests that simple harnesses suffice when paired with sufficiently capable models, creating structural dependency on expensive commercial systems and overlooking systematic approaches to agent architecture.

This analysis challenges the model-centric assumption through empirical examination of harness impact. An agent is formally defined as the composition of a language model and a harness - the infrastructure encompassing tool definitions, execution patterns, safety mechanisms, and optimization frameworks. While model capability receives extensive research attention, the quantitative contribution of harness quality to overall agent performance remains underexplored. Evidence from standardized benchmarks suggests harness design exerts greater influence on agent capability than model selection, particularly for resource-constrained deployments requiring local or open-source models.

The central thesis posits that harness quality matters more than model capability for practical agent performance. This proposition carries significant implications: if harness architecture determines performance, organizations can achieve superior results through infrastructure investment rather than model procurement, enabling independence from proprietary systems. This synthesis examines quantitative evidence for harness impact, identifies fundamental requirements for agent infrastructure inadequately addressed by existing frameworks, and presents language-level solutions through Agency, a domain-specific language designed specifically for agent development.

2. Background and Related Work

2.1 Agent Architecture and Execution Patterns

Traditional agent frameworks implement harnesses as minimal wrappers around model APIs, providing basic tool-calling capabilities without sophisticated execution control or safety mechanisms. This architectural approach assumes that sufficient model capability compensates for infrastructure limitations - an assumption that recent empirical evidence challenges. The ReAct pattern (Reason-Act-Observe) establishes a foundational framework for agent feedback loops, enabling iterative problem-solving through systematic observation of action outcomes. This pattern transforms agents from single-pass generation systems to architectures capable of self-verification and iterative correction.

Partial function application (PFA), derived from functional programming paradigms, enables argument binding at function definition time rather than invocation time. When applied to agent systems, PFA provides capability constraint mechanisms that prevent modification of specified parameters without requiring explicit permission systems for every operation. This approach balances autonomous operation with safety constraints by restricting operational scope rather than mandating approval workflows.

2.2 Benchmark Methodology

Harness Bench provides standardized evaluation across 106 tasks designed to measure agent performance while controlling for model capability. Unlike benchmarks that evaluate models directly, Harness Bench isolates harness impact by testing identical models with different infrastructure implementations. This methodology enables direct measurement of infrastructure contribution to agent capability, separating harness effects from model effects through controlled comparison.

3. Core Analysis

3.1 Quantitative Impact of Harness Quality

Harness Bench results demonstrate performance variation from 52.4% to 76.2% using identical models with different harnesses - a 23.8 percentage point spread attributable solely to infrastructure differences. This magnitude exceeds typical performance gains from model upgrades and establishes harness quality as a primary performance determinant rather than a secondary factor. Critically, the data reveals that harness impact increases for weaker models, suggesting that superior infrastructure design can enable resource-constrained open-source models to achieve performance parity with more capable proprietary alternatives.

This inverse relationship between model capability and harness impact carries practical implications for deployment strategies. Organizations constrained by cost, latency, or data sovereignty requirements can compensate for weaker models through superior harness design, achieving comparable results to proprietary model deployments while maintaining local control. The finding challenges the assumption that simple harnesses suffice with powerful models, as even capable models benefit from sophisticated infrastructure.

3.2 Safety Mechanisms Through Interrupts

Existing agent frameworks lack systematic mechanisms for safe autonomous operation, forcing developers to choose between unrestricted autonomy and manual approval for every action. Agency addresses this limitation through an interrupt system that pauses execution and requests human approval for sensitive operations. Functions that mutate data or access sensitive information throw interrupts by default, creating a safety-first architecture that requires explicit opt-in for autonomous operation.

The interrupt mechanism supports handlers - custom approval logic that evaluates actions without requiring manual intervention. This architecture enables conditional autonomy: agents can operate independently when actions fall within approved parameters while automatically escalating exceptional cases for human review. Interrupt state persists across execution sessions, allowing agents to pause, await approval, and resume without architectural modifications or state management complexity.

3.3 Capability Constraint Through Partial Function Application

Safety mechanisms based solely on approval workflows create operational bottlenecks that limit agent utility. Agency implements partial function application to constrain agent capabilities at the architectural level rather than through permission systems. By locking specific function arguments at definition time, PFA prevents agents from modifying constrained parameters while maintaining full autonomy within specified bounds.

For example, applying PFA to file system operations by locking the directory parameter enables agents to read and write files autonomously within a specified directory while preventing access to other filesystem locations. This approach eliminates the need for approval workflows while maintaining strong safety guarantees through architectural constraint. The mechanism balances capability with safety by restricting operational scope rather than requiring human oversight for every action.

3.4 Feedback Loops and Iterative Verification

The ReAct pattern implementation in Agency creates systematic feedback loops that enable agents to verify their own work and iterate until success. Rather than generating single-pass solutions, agents execute a continuous cycle: read relevant files, execute tests, observe failures, modify code based on failure observations, and re-test. This architecture transforms agents from suggestion generators to autonomous problem-solvers capable of verification and correction.

Empirical results demonstrate that feedback loops significantly improve task completion rates, particularly for complex multi-step problems where initial attempts frequently fail. The ability to observe action outcomes and adjust subsequent actions based on observations enables agents to navigate solution spaces that would be inaccessible through single-pass generation. This capability proves especially valuable for weaker models, where initial generation quality may be insufficient but iterative refinement achieves satisfactory results.

3.5 Sub-Agent Architecture for Context Management

As agent complexity increases, context management becomes a limiting factor. Providing all available tools in a single context creates several problems: reduced tool selection accuracy due to semantic interference between unrelated concepts, context length limitations that prevent inclusion of all necessary tools, and sequential execution bottlenecks when parallel execution would be possible. Agency addresses these challenges through a sub-agent architecture where specialized agents function as callable tools.

Sub-agents maintain consistency with standard tool usage patterns while enabling sophisticated orchestration. Each sub-agent receives only the tools relevant to its specialized domain, improving selection accuracy by reducing semantic noise. Independent sub-agents execute in parallel when their operations do not have dependencies, reducing sequential execution overhead. This architecture provides a natural decomposition mechanism for complex tasks while maintaining the simplicity of the standard tool-calling interface.

4. Technical Insights

4.1 Language-Level Requirements and Agency Design

Existing frameworks and libraries prove insufficient for sophisticated agent development because they lack necessary primitives at the language level. Agency addresses this gap through a domain-specific language designed specifically for agent development, incorporating TypeScript-inspired syntax with Python elements. The design principle "simple things should be simple, complex things should be possible" guides architectural decisions, ensuring that common patterns require minimal code while enabling sophisticated customization when needed.

Automatic tool generation from function definitions eliminates boilerplate: every function in Agency automatically becomes available as a tool through JSON schema generation. This approach reduces the implementation burden while maintaining type safety and documentation consistency. The Agency standard library provides built-in tools for common operations including file system access (read, write), information retrieval (Wikipedia search), and verification (test execution), establishing a foundation for agent development without requiring external dependencies.

4.2 Optimization and Measurement

Agency incorporates built-in optimization frameworks, specifically the Japa optimizer, for systematic prompt improvement. Rather than relying on trial-and-error refinement, the optimizer enables mathematical, measurable improvement methodologies. Variables marked with the optimize modifier become optimization targets, and the system runs multiple iterations against defined objective functions to establish baseline metrics and iteratively improve performance.

This integrated optimization capability addresses a fundamental limitation in current agent development: the absence of systematic improvement methodologies. Developers typically iterate through manual prompt refinement without quantitative measurement of improvement or systematic exploration of the optimization space. The Japa optimizer transforms this ad-hoc process into a principled approach with measurable outcomes, enabling continuous improvement against defined objectives.

4.3 Implementation Considerations and Trade-offs

The language-level approach introduces learning overhead: developers must acquire familiarity with Agency syntax and primitives rather than working within existing programming languages. However, this cost is offset by the elimination of boilerplate code, automatic safety mechanism integration, and built-in optimization capabilities that would require significant implementation effort in general-purpose languages.

The interrupt system introduces execution complexity through pause-and-resume semantics that persist across sessions. This architecture requires careful state management and introduces potential failure modes if interrupt handlers are not properly configured. The trade-off favors safety and flexibility over simplicity, accepting implementation complexity in exchange for robust safety guarantees and flexible approval workflows.

5. Discussion

The findings presented challenge fundamental assumptions in AI agent development and suggest a reorientation of research and engineering priorities. If harness quality exerts greater influence on agent performance than model capability, investment in infrastructure development yields higher returns than model procurement for many deployment scenarios. This insight proves particularly relevant for organizations constrained by cost, latency, or data sovereignty requirements that preclude reliance on proprietary models.

The inverse relationship between model capability and harness impact suggests a democratization opportunity: superior harness design can enable resource-constrained deployments using open-source models to achieve performance comparable to systems using proprietary alternatives. This finding has implications for accessibility and independence in AI deployment, reducing barriers to entry and enabling local deployment scenarios that would be impractical with model-centric architectures.

Several areas warrant further investigation. The Harness Bench benchmark provides valuable quantitative evidence for harness impact, but additional research examining the mechanisms through which specific harness features contribute to performance would strengthen causal understanding. The generalization of findings across different task domains and model architectures requires validation through expanded benchmarking. The long-term maintenance and evolution characteristics of agent systems built with language-level support versus framework-based approaches remain unexplored.

6. Conclusion

This analysis demonstrates that harness quality constitutes a primary determinant of agent performance, with quantitative impact exceeding 23 percentage points on standardized benchmarks using identical models. The findings challenge prevailing assumptions regarding the primacy of model capability and suggest that infrastructure investment yields greater returns than model procurement for many deployment scenarios. Language-level support through domain-specific languages like Agency addresses fundamental limitations in existing frameworks, providing primitives for safety, capability constraint, feedback loops, and optimization that prove difficult to implement in general-purpose languages.

Practical implications include the viability of open-source model deployments achieving performance parity with proprietary alternatives through superior harness design, reduced dependency on expensive commercial systems, and improved safety guarantees through architectural mechanisms rather than procedural controls. Organizations developing AI agent systems should prioritize harness architecture and infrastructure investment alongside model selection, recognizing that infrastructure quality may ultimately determine deployment success more than model capability. Future work should focus on expanding benchmark coverage, identifying specific harness features that contribute most significantly to performance, and developing systematic methodologies for harness design and optimization.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub