Don't Trust Your AI Agents. Test Them Like a Scientist - Stephanie Jarmak, Omni Analytics

Evaluations of AI agents are easy to misinterpret and manipulate; trustworthy evaluation requires scientific rigor, matching harness/configuration to real wo...

Don't Trust Your AI Agents: Test Them Like a Scientist

Abstract

Evaluation practice for autonomous AI agents has not kept pace with deployment velocity, producing measurements that are simultaneously easy to misread and easy to manipulate. This synthesis argues that trustworthy agent evaluation requires scientific rigor in experimental design, alignment between evaluation harness and production workload, deep observability into execution traces, and principled skepticism toward both public benchmarks and agent self-reports. Evidence drawn from agent-benchmarking practice demonstrates the scale of the problem: harness substitution alone shifted relative model rankings by more than 30 percentage points on Terminal Bench, a 28% cost reduction masked a persistent tool-usage failure, and injected security defects survived hash-based tamper checks. A production-grounded evaluation framework is proposed, encompassing internal benchmark construction, multi-trial non-determinism control, detection-enforcement decoupling, and explicit autonomy boundaries calibrated to human review capacity.

1. Introduction

The deployment of AI agents - large language model (LLM) systems that invoke tools, maintain state across multiple turns, and act with partial autonomy - has outpaced the methodological maturity of their evaluation. Organizations routinely select models and configurations on the basis of leaderboard positions, yet the measurement apparatus producing those positions is sensitive to design choices that are rarely held constant and seldom reported.

A terminological distinction anchors the analysis. A benchmark is a standardized reference set constructed for broad landscape understanding and candidate shortlisting. An evaluation is a measurement activity undertaken to drive a specific product or process decision within a specific organizational context. The two are frequently conflated, with the consequence that benchmark results are treated as transferable evidence about deployment behavior when they constitute, at best, a prior to be tested, not a conclusion to be inherited.

The central thesis advanced here is deliberately inverted relative to common framing: the practical question is less how to trust agents than how to withhold trust systematically. As one practitioner framing puts it, "It says how to trust your agents, but I think largely this talk is mainly about how to not trust your agents." Section 2 situates the problem relative to existing benchmarking infrastructure. Section 3 analyzes metric fragility, harness sensitivity, evaluation design, and debugging methodology. Section 4 extracts implementable technical guidance. Section 5 discusses implications for trust calibration and human oversight, and Section 6 concludes.

2. Background and Related Work

Contemporary agent benchmarking rests on a small number of shared artifacts. Terminal Bench 4 provides a task suite - approximately 63 tasks in the referenced configuration - for evaluating agents operating in shell environments. Open Hands is one of several agent harnesses: the scaffolding layer that mediates between a model and its environment, determining prompt construction, tool registration, context management, and control flow. Code Scale Bench represents an effort to broaden coverage beyond the bug-fixing tasks that dominate existing coding benchmarks toward the full software development lifecycle.

Complementary methodology includes LLM-as-judge auditing, in which a model evaluates the outputs or traces of another model, and the precision-recall / F1 framework imported from information retrieval for assessing tool behavior. An emerging organizational pattern - the software factory approval queue - formalizes escalation from autonomous agent action to human review. These artifacts are necessary but insufficient. None of them, by construction, guarantees that measured performance generalizes to an organization's configuration, workload distribution, or risk tolerance.

3. Core Analysis

3.1 Metric and Harness Sensitivity

A recurring finding is that single-number metrics obscure the behaviors they purport to summarize. In one documented case, a prompt modification reduced an agent's run cost by 28% while the agent continued to fail to invoke its intended tool - a result that would read as unambiguous improvement under cost-only monitoring but represents no functional gain. This illustrates a general hazard: optimizing a proxy metric (cost, latency, token count) without joint observation of task correctness can produce regressions disguised as progress.

Harness choice exerts comparably large and underappreciated effects. Swapping Open Hands for an alternative harness changed which model "won" on Terminal Bench tasks by more than 30 percentage points between Opus and Astra-class models. Because the harness governs prompt construction, tool registration, and context management, it functions as a confound on par with the model itself; reporting benchmark scores without specifying harness configuration is analogous to reporting an experimental result without specifying the apparatus.

3.2 Benchmarks versus Evaluations as Decision Instruments

The distinction between benchmarks and evaluations is not merely semantic but methodological. Public benchmarks serve landscape orientation and shortlisting; they are constructed for breadth and comparability across vendors, not fidelity to any single deployment context. Evaluations, by contrast, are measurements taken to drive a specific decision and must therefore reflect the organization's actual workload, tooling, and risk profile. A law firm, for instance, derives limited decision-relevant information from terminal or coding benchmarks unless its workflows specifically involve such tasks.

The recommended practice is construction of internal benchmarks that mirror production task distributions, iterated within a flywheel of hill-climbing optimization - repeated cycles of measurement, change, and re-measurement against the organization's own reference set rather than an external leaderboard. Code Scale Bench exemplifies this logic at a broader scale: its motivation is explicit dissatisfaction with existing coding benchmarks' concentration on bug-fixing tasks to the exclusion of other phases of the software development lifecycle, prompting construction of task families that better cover real workload composition.

3.3 Designing for Non-Determinism and Interaction Effects

Rigorous agent evaluation design must account for two statistical properties often neglected in practice. First, agent outputs are non-deterministic; single-trial comparisons between configurations risk mistaking sampling variance for genuine effect. Multiple trials per configuration are necessary to establish that observed differences exceed the noise floor. Second, changes interact: a model substitution combined with a harness substitution can produce effects not predictable from either change in isolation, as demonstrated by the divergent harness-dependent rankings in Section 3.1. Evaluation designs that vary one factor at a time while holding others fixed will systematically miss such interaction effects.

3.4 Debugging via Full-Pipeline Observability

Diagnosing agent behavior requires tracing the complete execution pipeline: whether a tool was available, whether it was called, what values it returned, and whether the returned information actually altered the outcome. Each stage can fail independently. A documented example involves Claude using grep and returning 295 files when only six were relevant to the task - a case of high recall achieved at the cost of precision so low that the output's practical utility is doubtful. Critically, improving a tool's retrieval capability does not guarantee improved task outcomes if the agent does not leverage the additional information correctly; observability must therefore extend past tool output to the agent's downstream use of that output. The appropriate evaluation metric - precision, recall, or F1 - must be selected according to actual business cost structures rather than defaulting to whatever a benchmark's automated validator assumes.

4. Technical Insights

Several implementation-relevant findings emerge. First, cost and efficiency metrics should never be evaluated independent of task-correctness metrics; the 28% cost-reduction case demonstrates that efficiency gains can coexist with unchanged functional failure. Second, harness configuration should be treated as a reported, controlled experimental variable, given the 30-plus percentage point ranking reversal observed between harnesses on Terminal Bench. Third, multi-trial designs are a minimum requirement for any comparison intended to support a deployment decision, given inherent agent non-determinism. Fourth, trace-level observability - covering tool availability, invocation, return values, and downstream use - is necessary to localize failure points that aggregate metrics cannot reveal. Fifth, agent self-reported traces require independent verification; security researchers found that injected security issues passed hash-based tamper checks and were caught only through independent comparison of all model traffic records, indicating that an agent's own attestations of correctness are an insufficient trust anchor. A limitation across all these points is cost: multi-trial, full-trace evaluation is computationally and operationally more expensive than single-pass benchmark scoring, requiring organizations to budget evaluation infrastructure commensurate with deployment stakes.

5. Discussion

These findings collectively suggest that agent evaluation is less a solved measurement problem than an open methodological one, structurally similar to early-stage experimental science requiring explicit control of confounds. The implication for human oversight is substantial: automated flagging mechanisms (e.g., LLM-as-judge auditors) must be decoupled from enforcement, since agents may detect but cannot necessarily halt problematic actions. This decoupling matters at scale - a 1% false alarm rate in a judge system can generate approximately 10,000 human reviews, a volume capable of overwhelming review capacity and incentivizing alert fatigue or rubber-stamping, which would undermine the safeguard's purpose.

A further implication concerns the limits of self-reported agent telemetry. The documented case in which injected security issues passed hash-based checks but were detected only via independent traffic comparison indicates that verification must sometimes occur outside the agent's own reporting channel entirely. This points toward a broader industry trend: production deployment, with shadow deployments for higher-risk changes, functions as the ultimate arbiter of agent behavior, since "production is truth." Open questions remain regarding how organizations should formally define autonomy boundaries - specifying permissible next actions, required rationale, and operating scope - in a way that is both auditable and scalable to human review capacity.

6. Conclusion

This synthesis has argued that trustworthy agent evaluation depends on treating measurement as an experimental discipline rather than a leaderboard-reading exercise. Metric fragility, harness sensitivity, non-determinism, and unreliable self-reporting each independently threaten the validity of agent performance claims, and their combination compounds the risk of deploying systems whose benchmarked behavior does not transfer to production. The practical takeaway is a production-grounded evaluation framework: internal benchmarks matched to actual workload composition, multi-trial designs sensitive to interaction effects, full-pipeline observability, decoupled detection and enforcement, and explicit, capacity-calibrated autonomy boundaries. Organizations adopting this framework are better positioned to distinguish genuine agent capability from artifacts of evaluation design.

Sources

• Don't Trust Your AI Agents. Test Them Like a Scientist - Stephanie Jarmak, Omni Analytics - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub