1,368 AI Failures and What They Teach About Evals - Vishal Srivastava, New York Life & Tanmay Sah

AI system evaluation should be risk-based and holistic, covering technical failure, deployment controls, and business context rather than focusing solely on ...

1,368 AI Failures and What They Teach About Evals: A Risk-Based Framework for AI System Evaluation

Abstract

This synthesis presents a risk-based, holistic framework for evaluating artificial intelligence (AI) systems, arguing that model-level technical performance is an insufficient basis for deployment decisions. Drawing on retrospective analysis of 1,368 incidents catalogued in the AI Incident Database, the framework decomposes system failure into three dimensions - technical strength, deployment controls, and business context - operationalized through thirteen sub-dimensions that collectively explain approximately 98% of observed failures. Findings indicate that an average of six sub-dimensions co-occur per incident, and that deployment controls, particularly human mediation and monitoring, show the strongest association with realized harm. The synthesis further proposes a falsification-oriented build-break-fix evaluation loop, informed by confirmation bias research, and extends system-level evaluation to agentic architectures. Practical implications include materiality-based prioritization of use cases, system-level stress testing prior to deployment, and governance of the evaluators themselves as components of the system under review.

1. Introduction

The proliferation of AI systems within enterprises has produced an evaluation bottleneck: the number of deployed use cases now exceeds organizational capacity to review each one individually. Concurrently, documented AI failures have increased exponentially, with the trajectory over the last four years - corresponding to the foundational model era - described as "almost vertical." These two trends jointly motivate the central question addressed in this analysis: how should organizations structure AI evaluation when exhaustive, case-by-case technical review is infeasible?

The thesis advanced is that evaluation should be risk-based rather than frequency-based, and holistic rather than model-centric. As stated in the source material, "the model alone or the technical strength of the model is just one part of the system." Three terms recur throughout. Technical failure denotes defects originating in data, design, or model behavior. Deployment controls denote operational mechanisms - authority limits, human review, exposure constraints, monitoring - that determine whether a defect propagates to a real-world decision. Business context denotes the financial, legal, and scale characteristics that determine the magnitude of consequence once propagation occurs.

This analysis proceeds by establishing theoretical context (Section 2), presenting the three-dimensional failure framework and empirical findings from retrospective incident analysis (Section 3), extending the analysis to agentic systems and falsification-based methodology (Section 4), and discussing broader implications before concluding with practical recommendations (Sections 5-6).

2. Background and Related Work

Two intellectual traditions inform the framework. The first is conventional machine learning (ML) evaluation practice, which assesses model quality on held-out data using aggregate metrics such as accuracy and AUC-ROC. This paradigm assumes a bounded input distribution and a single predictive output - assumptions that degrade when applied to systems that retrieve external context, invoke tools, and execute multi-step action trajectories.

The second tradition derives from cognitive psychology. Peter Wason's 2-4-6 task demonstrated that 79% of participants exhibited confirmation bias, generating test cases intended to confirm a hypothesized rule rather than falsify it. This is relevant to AI evaluation in two respects: AI systems trained on human-generated data may inherit analogous biases, and evaluation suites constructed by human practitioners tend toward confirmatory test design - demonstrating that a system works on expected inputs rather than probing where it fails. The framework treats falsification as a design requirement rather than an optional supplement to evaluation.

3. Core Analysis

3.1 The Three-Dimensional Failure Framework

The framework decomposes AI system risk into three core dimensions. Technical strength comprises six sub-dimensions: data quality, system design, predictive validity/outcome analysis, fairness checks, failure modes, and explainability. Deployment controls comprise four sub-dimensions: decision authority, human mediation, exposure control, and monitoring controls. Business impact comprises three sub-dimensions: financial impact, legal consequences, and scale (number of users affected). Together these thirteen sub-dimensions were applied retrospectively to incidents in the AI Incident Database and were found to account for approximately 98% of documented failures, indicating substantial coverage despite the heterogeneity of AI applications.

A notable finding is that failures are rarely attributable to a single sub-dimension. On average, six sub-dimensions co-occur within a single incident, suggesting that failures emerge from compounding weaknesses across technical, operational, and contextual layers rather than isolated defects. This compounding pattern argues against evaluation approaches that certify a model based on technical metrics alone.

3.2 Deployment Controls as the Dominant Risk Factor

Grouped analysis of the thirteen sub-dimensions reveals a hierarchy of association with harm: deployment controls are most critically associated with harm, followed by business impact, followed by technical failure. Within deployment controls, human mediation effectiveness and harm detection/monitoring controls show the highest association with harm specifically, while scale and system design, though frequently present, are less strongly associated with it. This supports the claim that "deployment controls are far more important" than model quality in isolation - a defective model contained by effective human review or monitoring may produce no realized harm, whereas a technically sound model deployed without adequate oversight can still produce severe consequences.

This finding has direct implications for evaluation prioritization: organizations with constrained review capacity should allocate scrutiny toward deployment architecture - authority boundaries, escalation paths, and monitoring coverage - rather than exclusively toward model benchmarking. A materiality scale is proposed as a mechanism for rank-ordering AI use cases by risk and harm potential, enabling resource allocation proportional to consequence rather than uniform review across all use cases.

3.3 System-Level Evaluation for Agentic Systems

Agentic systems introduce evaluation challenges beyond those addressed by traditional ML metrics. Risk in these systems is decomposed into four components: how often the model fails, whether failure reaches a real decision, whether it goes uncaught, and how severe the resulting consequences are. Critically, the same underlying model can present different risk profiles depending on its deployment setup - authority level, human review presence, and monitoring configuration all modulate whether a technical failure translates into harm.

Testing agentic systems therefore requires evaluation across retrieval accuracy, tool failure handling, human review effectiveness, permission boundaries, and recovery capability - dimensions absent from traditional held-out-data evaluation using accuracy or AUC-ROC. Because agentic systems may employ AI judges or evaluators to assess trajectories and outputs, evaluator reliability itself must be verified, since these evaluators can make mistakes. Consequently, the evaluator is treated as part of the system requiring governance, not as an independent, trusted arbiter.

4. Technical Insights

Several actionable findings emerge from this analysis. First, the 98% coverage achieved by thirteen sub-dimensions suggests that comprehensive risk assessment does not require bespoke frameworks per use case; a standardized taxonomy can be applied across heterogeneous AI applications with high explanatory power. Second, the co-occurrence rate of six sub-dimensions per incident implies that evaluation checklists addressing only one or two dimensions - such as model accuracy and fairness - will systematically miss the majority of contributing risk factors. Third, the empirical dominance of deployment controls over technical failure in harm association suggests a reallocation of evaluation effort: human mediation effectiveness and monitoring coverage warrant equal or greater scrutiny than model benchmarking.

The build-break-fix(-and-break-again) loop operationalizes falsification-based evaluation: test system claims, identify where they break, apply fixes such as restricting system scope or adding supporting controls, and iterate. This process generates a failure region map distinguishing trusted outcomes from uncertain or high-risk regions, providing a structured alternative to one-time validation. A limitation is that this approach requires active adversarial test design, which demands more effort than confirmatory testing and may be resisted given the confirmation bias tendencies documented in Wason's research, where 79% of subjects favored confirming evidence.

For agentic systems, the expanded testing surface - retrieval accuracy, tool failure handling, permission boundaries, recovery capability - implies higher evaluation cost per system, partially offset by the materiality scale's prioritization mechanism. A further trade-off concerns evaluator governance: introducing AI judges to scale evaluation reduces human review burden but introduces a secondary reliability dependency that must itself be audited.

5. Discussion

The framework's central implication is that AI evaluation practice, as currently constituted in much of the industry, overweights model-level benchmarking relative to its empirical association with harm. This finding resonates with broader industry discourse on AI governance, which has increasingly emphasized human-in-the-loop design and monitoring infrastructure, but the retrospective quantification offered here - deployment controls ranking above business impact and technical failure in harm association - provides empirical grounding for what has often been an intuitive claim.

The airline chatbot case illustrates the practical mechanics of this framework: an incorrect policy statement (technical/design failure) reached a customer without human mediation or policy-check controls (deployment control failure), resulting in legal and reputational consequences (business impact). Applying the build-break-fix approach - testing chatbot responses against refund policy questions prior to deployment - could plausibly have surfaced the failure before it reached a customer, illustrating how falsification-based testing operationalizes the three-dimensional framework.

Open questions remain regarding the generalizability of the thirteen-dimension taxonomy to AI incident types not yet well-represented in the AI Incident Database, and regarding the practical cost of scaling build-break-fix loops across the large number of enterprise use cases referenced at the outset of this analysis. Additionally, the governance of AI evaluators as system components raises unresolved questions about who evaluates the evaluator, particularly as organizations increasingly rely on AI-based grading for agentic trajectory assessment.

6. Conclusion

This analysis contributes a risk-based, three-dimensional framework for AI evaluation - spanning technical strength, deployment controls, and business context - validated against 1,368 retrospectively analyzed incidents with 98% coverage via thirteen sub-dimensions. The central finding, that deployment controls show the strongest association with harm, challenges evaluation practices centered narrowly on model performance metrics.

Practically, organizations are advised to: assess use case materiality using a company-specific risk scale; apply system-level stress testing, including falsification-oriented build-break-fix loops, before deployment decisions; and extend governance to evaluators themselves in agentic contexts. The overarching takeaway is that evaluation must address the full AI system based on risk, not isolated technical performance.

Sources

• 1,368 AI Failures and What They Teach About Evals - Vishal Srivastava, New York Life & Tanmay Sah - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub