How to Test AI Agents When No Benchmark Exists - Nathanael Weill, Air Canada

When domain experts and public benchmarks are unavailable, teams can build evaluation sets by inverting the problem - generating known answers first and testin...

How to Test AI Agents When No Benchmark Exists: An Inversion-Based Approach to Enterprise Agent Evaluation

Abstract

Evaluating LLM agents deployed on proprietary enterprise tasks is constrained by two scarcities: the absence of public benchmarks for firm-specific workflows, and limited access to domain experts capable of supplying ground truth. This synthesis, drawn from work by Nathanael Weill at Air Canada, documents an inverted, answer-first evaluation methodology developed across two deployments - a telecommunications RAG assistant and a supplier-contract compliance agent. Rather than collecting questions and soliciting human judgments, the approach fixes a known answer first and then tests whether the agent recovers it. Five synthetic data generation strategies are detailed alongside their failure modes. Reported outcomes include evaluation cadence increasing from one trial per week to dozens per day, 275 automatically generated compliance test cases, and test-contract generation time dropping from four weeks to five minutes. The findings suggest evaluability should be treated as a design constraint rather than a downstream activity.

1. Introduction

Enterprise deployment of LLM-based agents has outpaced the evaluation infrastructure required to govern it. Public leaderboards measure general capability on open benchmarks, but teams building agents for airline operations, corporate functions, commercial planning, or loyalty programs confront tasks for which no benchmark exists and no published technique has established priority. At Air Canada, the AI team builds agents across these domains without access to public benchmarks tailored to specific business tasks. Compounding this, the data scientists and AI engineers constructing the agents are not domain experts themselves, while the experts who could adjudicate correctness are busy, dispersed across business units, and often disagree with one another when consulted.

This scarcity is not merely an inconvenience but a structural barrier to iteration. Evaluation, understood here as the capacity to assign a correctness signal to an agent's output, is the precondition for improvement; without it, engineering effort is directionless. The central thesis advanced in this analysis is that evaluability can be manufactured through inversion: generating an artifact whose correct output is known by construction, then measuring whether the agent recovers it. This reframes a subjective grading problem into a deterministic, code-checkable comparison, converting a slow human-in-the-loop ritual into an automated iteration loop.

The analysis proceeds by first establishing empirical context from a 2024 telecom RAG project (Section 2), then developing the inversion principle and its extension to a multi-step contract-analysis agent, including five synthetic data generation techniques (Section 3). Section 4 consolidates technical findings, Section 5 discusses limitations - particularly those of LLM-as-judge - and Section 6 concludes with practical implications.

2. Background and Related Work

The motivating case is a 2024 question-answering assistant built for telecom engineers navigating a corpus of approximately 10,000 internal documents supplemented by vendor documentation. The project was competitive: two external vendors and one internal team worked from identical documents and an identical task specification. The team's solution was adopted, yet the causal basis for its success remained unclear at the time of selection.

Two observations shaped subsequent practice. First, complex pipelines assembled from published techniques frequently failed to improve results and occasionally degraded existing functionality - "complexity doesn't always help." Second, the evaluation budget permitted approximately one trial per week, conducted as a live expert grading session comparing five chatbots side by side on a large screen, in which experts routinely disagreed. This produced a foundational learning: "If you cannot grade the agent, you can't make it better." This positions the work within evaluation-driven development, where measurement infrastructure precedes and constrains model improvement rather than following it.

3. Core Analysis

3.1 The Inversion Technique for RAG Evaluation

The telecom project's evaluation bottleneck was resolved by inverting the standard question-answering workflow. Instead of sourcing questions from humans and grading answers with humans, the team began with a document page, asked an LLM to generate a question about it - an easy generation task - and then had the agent search for the answer, comparing the retrieved source to the original page via code. This removed subjective human grading from the loop entirely and allowed evaluation volume to increase from one trial per week to dozens per day.

With rapid, automated feedback available, the team discovered that reducing pipeline complexity while tuning the embedding model, chunking strategy, and BM25 alpha parameter improved accuracy more than layering in advanced techniques drawn from publications. This finding reinforces the earlier observation that complexity is not inherently beneficial; it is only useful when it can be validated against a fast feedback signal.

3.2 Extending Inversion to Multi-Step Contract Analysis

The Air Canada 2026 project applied this logic to a substantially harder task: analyzing supplier contracts for compliance and flagging non-compliant clauses, a process that is costly and expert-intensive with no existing benchmark or established technique. The agent pipeline comprises four steps - spotting a clause, finding relevant supporting documents, writing an amendment, and classifying whether to keep the amendment. The document-retrieval step reused the prior RAG evaluation technique directly; the remaining three steps required new strategies, since their outputs were not simple retrieval matches but generative and judgmental artifacts.

3.3 Five Techniques for Generating Synthetic Test Data

Five distinct approaches to answer-first test generation were developed, with mixed success:

1. Diffing past decisions: Comparing original and amended contracts to extract expected flags. This approach failed because the team had not retained original contract versions, illustrating that historical-data techniques depend on data retention practices established well before an evaluation need arises.
2. Rule-based generation: Using known business rules - such as contract caps above $1 million - to generate compliant and non-compliant clauses and check whether the agent flags them correctly. This produced 275 testable cases, the largest automatically generated set in the project.
3. Clause removal: Removing a clause from a signed contract and testing whether the agent detects the omission. This was the hardest task to pass but produced the strongest benchmark; agents frequently missed removed clauses, exposing a significant weakness that other methods did not surface.
4. Invariance testing: Changing supplier name or address, reordering clauses, or rerunning the same contract and checking that results remain consistent. This technique is fully code-checkable and requires no LLM-as-judge, making it particularly cheap to run at scale.
5. Template-based generation: Using existing expert-authored contract templates with blanks, then generating arrays of values to fill them, creating many contracts with known outcomes. This leveraged existing expert artifacts rather than requiring new expert time, and reduced contract generation cycles from four weeks of expert feedback to five minutes of automated scripting.

4. Technical Insights

Several implementation-level findings generalize beyond the two case studies examined. The RAG evaluation pipeline - generate question from page, check retrieval via code - demonstrates that LLM-as-judge can often be replaced by exact-match or structural comparison when the test artifact is constructed with a known answer embedded by design. Trajectory evaluation extends this logic to agentic workflows: by defining expected endpoints or step ordering in advance and generating a scenario to match, an agent's execution trace can be checked against the expected trajectory without subjective grading. Similarly, text-to-SQL style evaluation - generating an input and verifying that the agent retrieves the correct answer - generalizes to any agent type where a ground-truth answer can be fixed prior to execution.

A notable trade-off is that some agent solutions had to be re-engineered away from their original design specifically to make them evaluable, indicating that evaluability constraints can and should influence architecture decisions, not merely follow them. Invariance checks are the cheapest category of test to maintain, since they require no judgment of correctness at all - only consistency - but they cannot surface errors that are reproducible across variations, as demonstrated by the clause-removal results.

5. Discussion

The recurring theme across both case studies is that grading capacity, not model capability, was the binding constraint on improvement. The telecom project's shift from weekly to daily evaluation cadence enabled iteration on concrete pipeline parameters - embedding model, chunking, BM25 alpha - that would otherwise have been invisible within a one-trial-per-week budget. The contract project extends this principle to generative and judgmental outputs, where no single technique suffices; instead, five complementary generation strategies were needed to cover different failure modes, from rule violations to omissions to invariance breaks.

A critical caveat concerns LLM-as-judge. The team found that solving the judging problem well was "almost as hard as solving the right problem," and consequently avoided LLM-as-judge wherever a code-based or exact-match alternative existed. This suggests that answer-first inversion is valuable precisely because it sidesteps the open problem of automated judgment rather than attempting to solve it directly.

Finally, the stated feedback-loop philosophy - that expert feedback should generate better tests and clause-removal ideas rather than directly patch the agent - reframes human expertise as a resource for test design rather than a bottleneck in the grading loop itself, which may generalize to other domains facing similar expert scarcity.

6. Conclusion

This analysis documents a transferable methodology for evaluating agents in domains lacking public benchmarks: fix the answer first, then test whether the agent recovers it. Across a telecom RAG system and a multi-step contract compliance agent, this inversion converted subjective, scarce human grading into automated, code-checkable comparisons, enabling evaluation cadence to scale from weekly to daily and test-case generation to shrink from weeks to minutes. The practical takeaway is that passing synthetically generated tests is a starting point for shipping decisions, not a final validation, and that evaluability should be treated as a design constraint from the outset of agent development.

Sources

• How to Test AI Agents When No Benchmark Exists - Nathanael Weill, Air Canada - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub