Evals for High-Stakes AI Agents - Doneyli De Jesus & Annabell Schäfer, Langfuse

Shipping reliable AI agents to production requires a systematic, continuous 'AI engineering loop' of tracing, monitoring, dataset building, experimentation, ...

Evals for High-Stakes AI Agents: A Systematic Approach to Production Reliability

Abstract

Deploying large language model (LLM)-based agents to production exposes a measurement gap: non-deterministic systems can fail silently while infrastructure telemetry reports nominal operation. This synthesis examines a systematic remedy - the AI engineering loop, comprising tracing, monitoring, dataset construction, experimentation, and evaluation - as practiced on the open-source Langfuse observability platform, which reports adoption by over 150,000 developers, 21 of the Fortune 50, and ingestion of more than 90 billion observations monthly. The analysis details mechanisms for ground-truth acquisition in regulated domains (expert curation, shadow runs, human-in-the-loop slicing), hybrid evaluation combining LLM-as-judge with deterministic code checks, and integration of evaluation gates into continuous integration and deployment (CI/CD). Findings indicate that neither full automation nor exhaustive manual review is viable; the effective equilibrium automates instrumentation and evaluation execution while reserving human judgment for dataset curation and problem scoping.

1. Introduction

Conventional software reliability practice rests on deterministic verification: a unit test either passes or fails, and system metrics - latency, error rates, throughput - correlate reasonably with user-perceived correctness. Agentic AI systems break this correspondence. An agent may report to a user that a refund has been issued without ever invoking the refund tool; the HTTP response is 200, latency is nominal, and no exception is raised. As articulated in the source material, "AI agents can fail while the system metrics themselves look green." Such failures are frequently discovered weeks later, when users notice the discrepancy between claimed and actual outcomes - a lag that is itself diagnostic of insufficient observability.

This gap motivated the founders of the Langfuse platform, who observed that "there's nothing [like] a unit test for AI applications and we need a solution for it." The central thesis advanced here is that reliability in agentic systems is achieved not through a single artifact - a benchmark, a guardrail, or an automated repair agent - but through a continuous engineering loop in which production behavior is captured, inspected, converted into evaluable datasets, and used to gate changes before and after deployment.

Key terms used throughout this analysis include tracing (capturing the full sequence of steps an agent takes), LLM-as-judge (using a model to score outputs against a rubric), and code-based evaluators (deterministic, rule-based scoring functions). The analysis proceeds as follows: Section 2 establishes the observability foundation and intellectual context; Section 3 analyzes the five components of the loop, ground-truth acquisition strategies, and error analysis workflows; Section 4 presents technical implementation considerations; Section 5 discusses the automation-judgment trade-off; Section 6 concludes with practical implications.

2. Background and Related Work

Tracing is the substrate on which all subsequent evaluation practices depend. It is the agentic analogue of distributed tracing in microservice architectures, extended to record semantic content - LLM calls, tool invocations, inputs, outputs, and metadata - rather than only timing and status codes. Without this inspectable log, the discrepancy between an agent's claimed action and its actual behavior is undetectable until a user reports it.

The platform examined here is positioned as infrastructure rather than application tooling, reflected in reported production usage at organizations including Ramp, Canva, Khan Academy, and Sup. The monthly ingestion figure of 90+ billion observations is reported as trending upward, a consequence of agents taking progressively more steps per user interaction - a structural observation suggesting that observability cost scales with agent autonomy rather than remaining fixed per interaction.

Two evaluation paradigms form the intellectual context for the analysis that follows. LLM-as-judge offers flexibility for subjective or open-ended criteria but introduces stochasticity, since repeated evaluation of identical inputs may yield different scores. Code-based evaluators - for example, regex exact-match checks - produce 100% consistent outputs for the same input but cannot capture nuanced or context-dependent failure modes. The synthesis below argues these are complementary rather than substitutable.

3. Core Analysis

3.1 The Five Components of the AI Engineering Loop

The loop comprises tracing, monitoring, dataset building, experimentation, and evaluation, executed continuously rather than as discrete project phases. Tracing captures every step an agent takes in an inspectable log, forming the raw material for all downstream analysis. Monitoring aggregates these traces into signals at scale, surfacing cost, latency, quality, and compliance issues before they compound. Datasets represent the full range of expected inputs and outputs an agent should handle, and are constructed either in advance or sourced retrospectively from production traces. Experiments isolate a single variable - model, prompt, or tool configuration - to measure its causal impact on outcomes. Evaluations apply LLM-as-judge and code-based deterministic checks against datasets to measure quality at scale. Critically, the appropriate entry point into this loop is use-case dependent: low-stakes applications may begin with lightweight monitoring, whereas regulated or customer-facing systems require upfront investment in dataset rigor and guardrail definition.

3.2 Ground Truth Acquisition in Regulated Domains

Regulated industries face an acute ground-truth problem, since expected outputs are not always self-evident from historical data. Three methodologies are identified. Expert curation involves domain experts directly building and validating datasets, ensuring that edge cases reflect regulatory nuance. Shadow runs have the agent operate in parallel with an existing manual process, enabling direct comparison of agent output against human-generated ground truth without exposing end users to risk. Human-in-the-loop systems slice incoming data over time into categories: cases that can be safely automated and cases that must always receive human review, with this classification itself evolving as confidence in the agent grows.

A demonstrated banking assistant illustrates these principles in practice. Persona impersonation techniques were used to test guardrails against different customer types, revealing whether the agent correctly refused unauthorized requests. Personally identifiable information, such as credit card numbers, is automatically masked before persistence into the platform, addressing compliance requirements at the infrastructure layer rather than relying on downstream review. Scores attached to each trace track regulatory compliance, investment advice flags, intent classification accuracy, and guardrail violations, with multimodal support extending evaluation to speech-to-text transcriptions of customer calls.

3.3 Error Analysis and Evaluator Design

Dataset construction is treated as a human-centered activity even as downstream evaluation is automated. Test cases are built to capture expected behavior, keywords, or intent rather than requiring exact-answer matching, reflecting the open-ended nature of natural language outputs. Datasets can be created via CSV upload, API/CLI/MCP integration, or - notably - sourced directly from production traces, closing the loop between live failures and future test coverage.

Error analysis proceeds by curating representative samples for domain-expert review through annotation queues, using simple scoring mechanisms (yes/no, 1-10 scales) supplemented with free-text comments. An LLM is subsequently used to extract failure-mode categories from these raw human comments, converting unstructured expert judgment into structured taxonomy at scale. This hybrid workflow - human judgment for labeling, LLM processing for categorization - is reported to improve evaluation reliability when balanced against deterministic code evaluators for objectively checkable criteria.

4. Technical Insights

Several implementation details merit attention for practitioners building equivalent systems. First, data store architecture matters at scale: Langfuse migrated from Postgres to ClickHouse to accommodate billions of events per day, an architectural decision reflecting the write-heavy, append-only nature of trace data. Regardless of backend, data remains exportable via S3, API, CLI, or MCP, preserving interoperability for downstream analysis or migration.

Second, deployment flexibility is offered through both self-hosted options (e.g., on AWS EC2) and a fully managed SaaS offering, allowing regulated organizations to retain data residency control where required. Third, automated PII masking functions redact sensitive data, such as credit card numbers, before persistence - a design choice that embeds compliance into the data pipeline rather than treating it as a post-hoc filter.

Fourth, evaluator reliability is addressed through calibration: LLM-as-judge scores are checked against human-labeled samples using an inter-rater reliability approach, with the recommended practice being to start with an LLM-as-judge configuration that catches more false positives and tune sensitivity downward over time as confidence increases. Finally, experiments and prompt versioning integrate directly into CI/CD pipelines: promoting a prompt version can trigger automated evaluation jobs with pass/fail thresholds per evaluator, producing an auditable evidence trail for deployment decisions rather than relying on subjective judgment.

5. Discussion

The findings suggest that agent reliability is fundamentally a measurement and workflow problem rather than a purely algorithmic one. The recurring failure mode - an agent claiming success without performing the underlying action - cannot be caught by infrastructure metrics alone, implying that organizations deploying agentic systems without trace-level instrumentation are operating with an undetected blind spot regardless of model quality.

A second implication concerns the limits of automation. The source material explicitly cautions against fully automated "fix my agent, make no mistakes" approaches, describing their output as "slop" relative to systematic human-curated evaluation. This suggests an industry tendency to over-index on automated remediation tools without first establishing the dataset and labeling infrastructure necessary to judge whether remediation is actually effective. Conversely, fully manual review does not scale to the interaction volumes demonstrated (e.g., 1,400 interactions handled for under $7 over seven days), indicating that some automation of instrumentation and evaluation execution is a practical necessity, not merely an efficiency gain.

An open question concerns generalizability: the regulated-industry methodologies (shadow runs, expert curation) are resource-intensive and may not transfer cleanly to lower-stakes consumer applications, suggesting that future work should characterize the risk thresholds at which each ground-truth acquisition method becomes cost-justified.

6. Conclusion

This synthesis has examined the AI engineering loop - tracing, monitoring, dataset building, experimentation, and evaluation - as a systematic response to the silent-failure problem characteristic of agentic AI systems. The central contribution is the reframing of evaluation not as a one-time benchmark exercise but as a continuous practice integrated into CI/CD, combining LLM-as-judge flexibility with code-based determinism and anchored by human-curated datasets.

Practically, organizations deploying high-stakes agents should prioritize comprehensive tracing before pursuing sophisticated automation, establish ground-truth acquisition methods proportional to regulatory risk, and resist the temptation toward fully automated remediation in favor of a calibrated division of labor between automated execution and human judgment in dataset curation and failure-mode definition.

Sources

• Evals for High-Stakes AI Agents - Doneyli De Jesus & Annabell Schäfer, Langfuse - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub