'How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude - Laurie Voss, Arize AI'

Building reliable AI agents requires a systematic evaluation workflow - tracing, reading traces manually, writing layered evals (code, LLM judge, human), valid...

How to Know If Your AI Agent Is Any Good: A Systematic Evaluation Workflow

Abstract

Artificial intelligence (AI) agents built on large language models (LLMs) are routinely shipped on the basis of informal manual inspection, a practice that provides no coverage guarantees and no mechanism for detecting regressions. This synthesis presents a systematic evaluation workflow that replaces subjective assessment with measurable evidence: instrumenting agents to emit traces, manually reading traces to discover unknown failure modes, constructing layered evals (code-based, LLM-as-judge, and human), validating judges against human-labeled golden data, running controlled experiments, and locking the resulting criteria into continuous integration (CI). Empirical observations from a financial-research agent built with the Claude Agent SDK and instrumented via Phoenix demonstrate the workflow in practice, including an actionability improvement from 5/16 to 48/48 passing, a faithfulness metric that remained stubbornly low despite prompt and model upgrades, and a correctness evaluator that failed all cases due to judge knowledge-cutoff artifacts. The practical implication is that evaluation design - particularly selecting the right question - dominates evaluator tuning.

1. Introduction

The dominant release practice for AI features is to execute a prompt several times, inspect the outputs informally, and deploy. As noted in the source material, "Looks good to me is not a test suite, but it's how most AI features get shipped." Conventional unit testing does not transfer to this setting because LLM outputs are non-deterministic: identical prompts can yield divergent yet equally valid responses, defeating exact-match assertions.

Two complementary artifacts underpin the alternative. A trace is a structured record of what an agent did, composed of spans capturing inputs, outputs, timing, token counts, and cost. An eval is a programmatic or human judgment of whether that behavior was acceptable. The governing mental model is concise: traces describe what happened; evals determine whether it was any good.

The central thesis is that reliable agent development requires an end-to-end measurement loop rather than ad hoc inspection, because "a prompt change that you haven't measured is just a vibe." Prompt modifications affect the entire input distribution, so fixing tone may silently degrade factual accuracy, producing a whack-a-mole dynamic. Each new model release creates upgrade pressure that cannot be rationally evaluated without a measurement baseline. This paper proceeds through the intellectual background (§2), the core workflow (§3), implementation-level insights (§4), broader implications (§5), and conclusions (§6).

2. Background and Related Work

The methodology draws on established frameworks from qualitative research and classification theory. Open coding and axial coding, borrowed from social science, structure the initial exploration of traces: annotations are first written in plain language, then grouped into emergent categories, rather than being forced into categories hypothesized in advance. The Swiss cheese model of layered defense motivates stacking heterogeneous evaluators, since each type has distinct blind spots; it argues against constructing a single monolithic "god evaluator."

Evaluator validation borrows from classification methodology: Cohen's Kappa quantifies inter-rater agreement, and the precision/recall tradeoff governs threshold selection. Reliability metrics distinguish pass@K (success on at least one of K attempts) from pass^K (success on every attempt). Two public benchmarks illustrate the fragility of rigid evaluation criteria: in Anthropic's Tau2-bench, Claude upgraded and then downgraded a passenger's seat to satisfy a user - reasonable behavior scored as failure under inflexible rules - and in CoreBench, Opus 4.5 scored 42% before flawed test conditions were corrected, after which the score rose to 95%.

3. Core Analysis

3.1 Layered Evaluation Design

Three eval types operate with complementary strengths. Code evals are fast, free, and deterministic, suited to checks such as JSON parsing, length constraints, or keyword presence; they cannot detect semantic failures. LLM-as-judge evals use a second model to grade outputs against a rubric, capturing meaning but introducing cost, unpredictability, and the risk of being "confidently wrong." Human evals remain the gold standard for quality and for surfacing unknown failure modes, but are slow, expensive, and subject to fatigue. The reported workflow treats these not as competing options but as layers in a Swiss-cheese defense, each catching what the others miss.

A minimal code eval checking whether a report mentions the correct ticker caught 2 of 16 failures at zero marginal cost, demonstrating that simple checks deliver outsized value before any LLM judge is invoked. Critically, code evals should grade outcomes rather than the path taken, since agents may reach valid solutions through unexpected trajectories.

3.2 Reading Traces Before Writing Evals

Before any eval is written, a dozen or more traces must be read end-to-end to expose unknown unknowns. This manual step uses open coding (free-text notes) prior to axial coding (categorization), avoiding premature assumptions about failure taxonomy. At scale, coding agents can apply Phoenix's error-analysis skill to annotate traces automatically with failure categories. Prioritization of fixes should follow frequency multiplied by severity, not frequency alone, since a rare but catastrophic failure may outweigh a common but minor one.

Agents compound this complexity because early mistakes cascade through subsequent steps, and "confidently wrong is worse than obviously wrong" - a well-formatted report about the wrong subject is more dangerous than an obviously malformed one, since it is more likely to be trusted.

3.3 Judge Construction and Validation

Phoenix ships built-in evals for correctness, faithfulness, completeness, hallucination, refusal, PII, toxicity, and tool-handling. In the reported financial-research agent, the correctness eval failed all 16 reports because the judge's knowledge cutoff (approximately January) predated the agent's October data, flagging accurate information as hallucinated. The faithfulness eval - checking grounding in provided context - proved more useful, uncovering a genuine reasoning error in which Tesla analyst percentages summed to 116% (50% + 52.6% + 13.1%) due to munging unrelated search results. This asymmetry supports the claim that "choosing the right eval beats tuning it."

Custom rubrics follow a five-part structure: assigning the judge a role, defining explicit criteria derived from real error analysis, wrapping data in XML tags (which fixed a bug causing unlabeled examples to be misjudged as non-actionable), cautiously including labeled examples to guard against overfitting, and constraining outputs via tool-call-defined choices rather than freeform text.

Judges must then be validated as classifiers against human-labeled golden data, split roughly 75/25 into dev and test sets to prevent tuning on held-out data. Because human inter-rater agreement is often low (Cohen's Kappa approximately 0.2-0.3), judges need only exceed human-level consistency rather than achieve perfection. Recall is typically prioritized over precision for agent evals to minimize undetected failures reaching customers, and any failure flagged by a judge should "seem fair" to a human reviewer - a criterion violated in CoreBench's initial 42% score before benchmark correction.

4. Technical Insights

Phoenix relies on OpenTelemetry and the OpenInference standard, enabling automatic instrumentation across frameworks including the Claude Agent SDK, Vercel AI SDK, Mastra, LangChain, and the OpenAI Agent SDK. A TypeScript-specific gotcha arises because the Claude Agent SDK is an ES module with read-only exports, requiring use of Phoenix's manually instrumented copy rather than the raw import.

The demonstrated agent architecture uses a two-turn design - a research turn followed by a writing turn - linked via session resume and wrapped in a single parent trace span. Structured JSON inputs substantially simplify code-eval construction, as in ticker extraction. Claude Haiku 4.5 was deliberately selected as the agent model for its low cost, speed, and "usefully dumb" predictable error patterns suited to demonstration purposes. Judge models should differ from the agent model (preferably a more capable model from a different provider) to avoid self-preference bias.

Experiments combine a dataset, a task (agent execution), and evaluators to enable controlled before/after comparison. Prompt fixes were mapped directly to specific eval findings, moving actionability from 5/16 to 48/48 passing; faithfulness, however, improved only marginally (2/16 to 3/48), indicating that prompt-level fixes do not uniformly address reasoning errors. A model upgrade from Haiku 4.5 to a hypothetical 5.5 tripled faithfulness but left it still poor, while consistency improved because the model became overly conservative, issuing "hold" recommendations 41% of the time at four times the cost. The reported leverage ordering for improvement is data quality > prompts > model choice > hyperparameters.

5. Discussion

These findings suggest that evaluation infrastructure, not model selection, is the binding constraint on agent reliability. The correctness-eval failure illustrates a broader risk: an evaluator can be technically well-implemented yet answer the wrong question, producing misleading signal at scale. This reframes evaluator development as a problem of question selection prior to optimization.

The persistence of low faithfulness scores despite prompt and model upgrades indicates that certain failure classes - particularly compound reasoning errors across retrieved data - may be structurally resistant to superficial interventions, warranting further investigation into retrieval-grounding architectures or verification sub-agents. The pass@K versus pass^K distinction also has direct industry relevance: domains such as financial analysis demand pass^K reliability, a stricter bar than much current benchmark reporting assumes.

Finally, embedding evals into CI - distinguishing guardrails (must-pass) from north stars (target averages) - extends software engineering discipline to probabilistic systems, and the observation that non-engineers can define "good" in plain language for translation into evals suggests a democratization of evaluation authorship beyond ML specialists.

6. Conclusion

This synthesis demonstrates that reliable AI agent development depends on a disciplined, multi-stage evaluation workflow: tracing, manual trace reading, layered code/LLM/human evals, judge validation against golden data, controlled experimentation, and CI integration. Empirical results - actionability gains from 5/16 to 48/48, a faithfulness metric resistant to improvement, and a misconfigured correctness evaluator - underscore that evaluation design decisions dominate outcomes more than evaluator tuning or model upgrades alone. Practitioners should prioritize reading traces before writing evals, validate judges against human judgment, and lock passing capability evals into regression suites to convert informal confidence into measurable evidence.

Sources

• How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude - Laurie Voss, Arize AI - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub