If Your AI Scores 100%, Your Eval Is Wrong with Ramakrishnan Lokanathan, eslams.com

Building reliable AI evaluation systems for high-stake applications requires treating evals not as a single score but as a holistic, continuously evolving fr

If Your AI Scores 100%, Your Eval Is Wrong: A Five-Principle Framework for High-Stakes AI Evaluation

Abstract

As generative AI systems move into production, the determining factor for success is increasingly evaluation infrastructure rather than raw model capability. This synthesis examines a five-principle framework for constructing evaluations ("evals") for high-stakes AI applications, drawn from practitioner experience spanning commercial LLM deployment, frontier-model red teaming, and AI safety research. The central argument holds that an eval is not a single score but a continuously evolving system: it must define success at an aspirational ceiling, instrument every stage of a multi-step workflow, gather evidence through heterogeneous paradigms including game-based agentic testing, evolve as models saturate existing benchmarks, and continuously absorb real-world failures. Supporting evidence includes the observation that frontier evals typically plateau near 85-86% rather than 100%, the E-Slams open-source evaluation package spanning approximately 50 games and 90 providers, and reported findings that 95% of organizations realize no return on generative AI investment. Practical implications center on CI-embedded checks, human review, and adversarial testing.

1. Introduction

The term eval, short for evaluation, denotes a structured assessment of system performance against expectation. The analogy to human examinations - the SAT, the GMAT, the bar exam - is instructive precisely because it is deliberately limiting. Each of these instruments comprises three components: a set of tasks, a standard against which responses are judged, and evidence produced by the comparison. Critically, each answers a bounded question. The SAT constitutes one signal of college readiness, not a complete measure of a candidate's capacity.

This boundedness is the first-principles insight underpinning the present analysis:

"No eval or no test is a measure of someone's complete abilities and that should always be clear going into that."

The same constraint applies to large language models (LLMs) and the agentic systems built atop them. A single benchmark score - coding proficiency, writing quality, mathematical reasoning - characterizes one dimension of a system that users experience as an integrated whole. The central research question addressed here is therefore not "how well does the model perform?" but rather: does production performance match the promise made to the consumer?

The analysis proceeds as follows. Section 2 establishes the intellectual and commercial context motivating renewed attention to evaluation. Section 3 presents the five-principle framework constituting the core contribution. Section 4 extracts actionable technical findings and trade-offs. Section 5 situates these findings within broader industry trends, and Section 6 concludes.

2. Background and Related Work

The framework's provenance is practitioner-derived rather than purely academic. Its author, Ramakrishnan, began building machine learning classifiers in secondary school in Cochin, South India, and subsequently worked at an Indian EdTech unicorn. Exposure to a GPT-3 demonstration in 2021 redirected their trajectory; by September 2022 - approximately two months before OpenAI's release of the ChatGPT research preview - the team had committed full-time to GPT-3 API applications. Subsequent work included red teaming for the Nemotron 3 model family and AI safety research at Lawrence Berkeley Lab alongside the London Business School data science and AI initiative. No vendor alignment or sponsorship is claimed.

Prior art referenced includes SWE-bench and Humanity's Last Exam, both characterized as single-signal evaluations admitting a correct answer to a one-shot task. AlphaGo is invoked as a landmark in agentic evaluation, with the caveat that it was narrowly trained for a single game, whereas general-purpose LLMs are rarely subjected to comparable long-horizon, adversarial testing. This gap motivates the game-based paradigm discussed in Section 3.

3. Core Analysis

3.1 Defining Ambitious Success and the 100% Problem

The framework's first principle holds that success criteria must be defined before any test is designed, and that this definition should be aspirational - approximating performance at the level of artificial general intelligence (AGI) rather than current model capability. This requires asking what must happen, what must never happen, and when a process should halt. The resulting heuristic is counterintuitive: a perfect score indicates a flawed instrument.

"If you have an eval that is scoring a 100 of a 100, your eval is wrong, right? And you're not ambitious enough and you're not testing enough."

Empirically, this is consistent with the observation that current frontier model evals typically plateau around 85-86%, rather than approaching 100%. This ceiling is interpreted not as a model limitation to be lamented but as evidence that well-constructed evals should continuously outpace current ability.

3.2 Testing the Whole Workflow

The second principle extends evaluation beyond input-output comparison to every intermediate step of a workflow. An illustrative failure case involved an image-editing instruction to change only the background; the system instead altered the subject, producing an erroneous second handle. Comparing only the initial prompt and final output would have missed this error entirely, since the defect occurred mid-pipeline. For workflows comprising on the order of 100 discrete steps, each step's input, output, and expected outcome requires individual evaluation, with test design tailored to the function of that step - for example, evaluating tool-calling steps specifically for tool-call accuracy rather than overall task success.

3.3 Holistic Evidence Gathering, Including Game-Based Paradigms

The third principle rejects the notion that a single score can characterize an agentic, persistent system. A holistic evaluation architecture instead comprises three elements: automated checks embedded in the continuous integration (CI) pipeline and run on every production deployment; continuous human review to validate that automated eval results are sensible, rather than relying on agents to review their own outputs; and adversarial testing designed explicitly to break the workflow.

A notable extension of this principle is the introduction of "tests that fight back" - continuous, interactive evaluation distinct from one-shot, correct-answer tasks such as SWE-bench. Board games such as chess and Go have long served as proxies for agentic action and long-chain reasoning in humans, and AlphaGo demonstrated AI competence in this domain, though narrowly. To generalize this paradigm to LLMs and agents, the practitioner developed E-Slams (electronic slams), an open-source Python package distributed via PyPI that enables AI agents and models to play approximately 50 games across roughly 90 providers. Game categories include chess and Go (planning under invisible state), poker (decision-making under hidden information), cooperation games (shared-goal coordination), and arcade-style trials. The stated use case is comparative: pitting an older agent against an improved model in direct gameplay to observe reasoning quality and blunder recovery, not merely win/loss outcomes. A demonstration pitting Fable 5.1 against GPT-6 Astra in tic-tac-toe is cited as an example, which concluded in a tie. As summarized in the source material:

"Getting one right answer more and more is not the same as making a good sequence of decisions and you need agentic action and long-chain reasoning."

3.4 Evolving Evals and Incorporating Real-World Failure

The fourth and fifth principles address the temporal dimension of evaluation. Evals must be continuously tested against regression: a model that improves on new tasks can simultaneously lose capabilities present in its predecessor. The cited example is Google Assistant's transition to Gemini, which reportedly lost the ability to set reminders - a regression attributable to insufficient testing of previously functioning cases.

"A model that is extremely better than the old one could very well be incapable of something the old one did very well, for example, setting a reminder."

Relatedly, once models score 85-90% on an existing eval, the eval itself should be made harder rather than treated as validation of sufficient capability. The fifth principle holds that end users are the most important evaluators of a deployed system. Observability tooling, automatic data collection, and simple feedback mechanisms (e.g., thumbs up/down) surface real-world failures, which should then be triaged using standard QA practice: reproduce the issue, localize the point of failure (for instance, a specific tool call), and identify which part of the existing eval suite failed to catch it. Reviewed failures are then folded back into the eval pipeline, closing the feedback loop.

4. Technical Insights

Several implementation considerations follow from the framework. First, CI-embedded automated checks should run on every deployment, but should not substitute for periodic human review, since agents evaluating their own outputs introduce correlated blind spots. Second, workflow-level instrumentation requires step-specific metrics rather than generic pass/fail criteria, implying additional engineering overhead proportional to workflow length - a nontrivial cost for pipelines with on the order of 100 steps. Third, adopting game-based evaluation via tools such as E-Slams trades the determinism of one-shot benchmarks for signal about sustained reasoning and error recovery, at the cost of requiring comparative (agent-vs-agent) rather than absolute scoring. Fourth, the "harden the eval as scores rise" heuristic implies evals are living artifacts requiring ongoing engineering investment rather than one-time deliverables, with resourcing implications for teams accustomed to treating benchmarks as static. Finally, the MIT GenAI divide finding - that 95% of organizations see no return on GenAI investment - is attributed specifically to inadequate eval and production-delivery structures rather than model deficiency, underscoring that eval infrastructure, not model access, is frequently the binding constraint on value realization.

5. Discussion

These findings suggest a reframing of evaluation's organizational role: rather than a gatekeeping checkpoint prior to deployment, evaluation functions as an ongoing production-monitoring discipline comparable to observability in traditional software systems. This reframing has direct bearing on the widely cited MIT finding regarding GenAI returns, suggesting that the "divide" separating successful from unsuccessful deployments is less about model selection and more about whether organizations have built the continuous eval-and-feedback apparatus described here.

A notable tension in the framework is temporal: the practitioner explicitly anticipates that evals will diminish in importance over a roughly ten-year horizon as model capability improves, reducing their function to mere validation rather than active gatekeeping. This raises an open question for future investigation - namely, at what rate of capability improvement does workflow-level and adversarial testing become unnecessary, and whether game-based paradigms like E-Slams will themselves saturate before that point. The framework also leaves underspecified how organizations should weight automated CI checks against human review when the two disagree, an area warranting further empirical study as holistic eval systems mature.

6. Conclusion

This synthesis presents a five-principle framework - defining ambitious success, testing the whole workflow, gathering holistic evidence across diverse paradigms, evolving evals over time, and incorporating real-world failure - as a practitioner-derived response to the problem of eval reliability in high-stakes AI deployment. The core directive distilled from this framework is to "test the job and not just the answer." Practically, teams building production AI systems should treat a saturated benchmark as a signal to intensify testing rather than a milestone of completion, instrument intermediate workflow steps rather than only final outputs, and establish durable feedback loops from user-reported failures back into eval suites.

Sources

• If Your AI Scores 100%, Your Eval Is Wrong - Ramakrishnan Lokanathan, eslams.com - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub