Inside 847 Production Clinical AI Notes - Sebastian Fox, Composo

AI systems in high-stakes domains like healthcare make dangerous errors that go undetected because evaluators cannot identify what matters in context; the so...

By Sean Weldon

Contextual Evaluation Failure in Production Clinical AI: A Framework for Dynamic Quality Assessment

Abstract

Production artificial intelligence systems in healthcare demonstrate a critical evaluation failure: ambient scribes deployed across approximately one-third of US medical practices exhibit serious errors in 5% of clinical notes, important omissions in nearly 20% of cases, and hallucinations in over 10% of outputs, yet these failures remain systematically undetected. Analysis of 847 production clinical notes reveals that conventional evaluation methodologies - including frontier models equipped with detailed rubrics and deterministic verification checks - fail to identify approximately 20% of clinically significant errors. This failure stems from evaluators lacking domain-specific "taste": tacit, contextual knowledge of clinical significance that resists codification in static rubrics. A continuous Discovery-Capture-Calibrate loop addresses this limitation by identifying failure modes from production outputs, systematically capturing expert judgment, and assembling case-specific evaluation context dynamically rather than relying on pre-specified criteria. This framework demonstrates superior performance and generalizes to high-stakes AI applications beyond healthcare where evaluation requires contextual domain expertise.

1. Introduction

The deployment of ambient scribes - large language model systems that automatically generate clinical documentation from patient-physician conversations - represents a critical inflection point for AI evaluation methodology. These systems have achieved widespread adoption, operating in approximately one-third of United States medical practices without systematic adverse event reporting mechanisms. The largest empirical study of production clinical notes reveals error rates that would be unacceptable in safety-critical applications: 1 in 20 notes contains errors serious enough to cause significant patient harm, nearly 1 in 5 exhibits important omissions, and more than 1 in 10 includes hallucinated content.

The troubling aspect of this phenomenon extends beyond raw error rates. These failures occur in systems that have undergone conventional evaluation processes, suggesting fundamental limitations in current assessment methodologies rather than mere implementation deficiencies. The errors that pose greatest clinical risk are not obvious hallucinations readily apparent on superficial inspection, but rather subtle omissions and context-inappropriate inclusions that appear entirely plausible to evaluators lacking deep domain expertise.

This analysis examines why conventional evaluation frameworks fail in high-stakes AI applications and proposes a dynamic assessment methodology grounded in continuous learning from production outputs. The central thesis posits that evaluation failure stems from a categorical mismatch: while LLMs excel at generating plausible documentation, evaluators lack the domain-specific "taste" - tacit knowledge of what matters clinically in specific contexts - necessary to distinguish superficially correct outputs from clinically sound ones. This knowledge cannot be fully codified in static rubrics, necessitating a fundamentally different approach to quality assessment.

2. Background and Related Work

The asymmetry of verification principle, foundational to AI safety research, holds that verifying correct outputs should be substantially easier than generating them. This asymmetry has historically justified deploying AI systems with human-in-the-loop verification, assuming that reviewers can readily identify errors even when unable to produce correct outputs themselves. However, this principle's applicability depends critically on the cognitive demands of verification relative to generation.

Reinforcement Learning from Human Feedback (RLHF) established precedent for learning quality standards from examples rather than explicit specification. This methodology acknowledges that human preferences and expert judgments often resist complete formalization, necessitating empirical capture of evaluation criteria through demonstrated examples. The success of RLHF in improving model alignment suggests that quality standards in complex domains may be fundamentally empirical rather than specifiable a priori.

In clinical documentation, verification complexity bifurcates sharply along task characteristics. Surface-level transcription errors - sound-alike pharmaceutical substitutions (Humalog → Humulin), condition inversions (hypothyroidism → hyperthyroidism), or negation flips (no evidence of cancer → evidence of cancer) - remain readily detectable through comparison with source audio. For these errors, verification asymmetry holds: checking transcription accuracy requires less expertise than producing accurate transcriptions. However, determining clinical significance of content additions, modifications, or omissions requires domain expertise that transcends simple fact-checking, fundamentally altering the verification task's cognitive demands.

3. Core Analysis

3.1 Empirical Failure Modes in Production Systems

Analysis of 847 production clinical notes from three leading ambient scribe systems reveals systematic failure patterns across three categories: additions (content never stated in conversation), changes (modifications to stated information), and omissions (clinically relevant information excluded from documentation). Critically, these categories share a common characteristic: the technical challenge lies not in detecting the addition, change, or omission itself, but in determining whether the deviation matters clinically.

A representative failure illustrates this distinction. In one case, a patient presented with jaw pain on chewing and new-onset headache, both occurring after age 50 - a clinical constellation suggesting giant cell arteritis, a condition requiring urgent treatment to prevent blindness. The ambient scribe documented this as routine tension headache because a single line about jaw pain never appeared in the generated note. The omission was not detectable through transcription comparison - the words about jaw pain existed in the audio transcript - but required clinical judgment to recognize that this specific detail transformed the diagnostic picture entirely.

Conversely, travel history demonstrates context-dependent significance. A patient reporting recent travel to Lake Malawi presenting with hematuria (blood in urine) requires immediate consideration of schistosomiasis, making travel history clinically critical. The identical travel detail for a patient visiting France with similar symptoms carries negligible diagnostic weight. No static rubric can specify in advance which travel histories matter; this determination requires case-specific clinical reasoning.

3.2 Limitations of Conventional Evaluation Methodologies

Frontier language models equipped with detailed rubrics, worked examples, and deterministic natural language processing checks demonstrate measurable improvement over baseline evaluation but still pass approximately 20% of outputs containing serious errors. This failure rate persists despite substantial engineering investment in evaluation infrastructure, suggesting fundamental rather than implementation-based limitations.

The core problem manifests as evaluator taste deficiency: judges lack knowledge of what constitutes clinical significance in specific contexts. This knowledge exhibits three characteristics that resist codification in static rubrics. First, it is tacit - domain experts possess and apply this knowledge but cannot fully articulate decision criteria. Second, it is contextual - identical details carry different significance depending on case-specific factors. Third, it is dynamic - clinical guidelines evolve, model behaviors change, expert consensus shifts, and institutional standards vary.

The distinction between faithful-to-words and faithful-to-intent errors proves particularly challenging for automated evaluators. When a physician states "let's wait and see" regarding antibiotic prescription, but the note documents "start amoxicillin," the error is not transcriptional - both phrases may appear in the audio - but intentional. The model accurately captured words while fundamentally misrepresenting clinical decision-making. Detecting this requires understanding not just what was said, but what was meant and decided.

3.3 The Discovery-Capture-Calibrate Framework

The proposed evaluation framework implements a continuous three-stage loop addressing the limitations of static rubrics. The Discovery phase identifies failure modes from actual production outputs rather than anticipated error categories. Synthetic test cases, while valuable for known failure modes, cannot cover errors not yet imagined. Real production outputs reveal how systems actually fail in practice, including failure modes that would not occur to system designers.

The Capture phase systematically records expert judgment through focused review sessions where clinicians annotate real outputs. Rather than month-long labeling projects attempting comprehensive coverage, this approach captures expert corrections and judgments on specific cases as they arise. These annotations become training data for evaluation systems, encoding domain expertise in demonstrated form rather than written rules.

The Calibrate phase assembles case-specific evaluation context dynamically for each output. Rather than applying fixed rubrics uniformly, the system retrieves: (1) most similar prior cases that experts have judged, (2) applicable expert corrections from the capture phase, and (3) relevant clinical guidelines and reference documents. This context engineering per output approach provides evaluators with case-appropriate standards assembled on-the-fly rather than frozen in model weights or static prompts.

3.4 Performance Characteristics and Generalization

Empirical comparison across evaluation approaches on identical datasets demonstrates progressive improvement. Initial approaches using frontier models with static rubrics perform better than random but miss most clinically significant errors. Enhanced approaches incorporating deeper rubrics and deterministic checks improve further but still fail to capture contextual significance. The discovery-capture-calibrate approach, assembling dynamic context for each evaluation, demonstrates substantially superior performance on the same dataset.

Critically, the performance difference stems not from increased computational resources or more sophisticated prompting, but from whether the evaluator possesses knowledge of what matters in the specific domain and case. This distinction suggests the framework's applicability extends beyond healthcare to any high-stakes AI application where evaluation requires contextual domain expertise: contract review systems that miss clauses fundamentally altering deal terms, customer support agents promising unavailable refunds, or financial analysis systems overlooking material risk factors.

4. Technical Insights

The implementation of continuous evaluation loops requires infrastructure supporting three technical capabilities. First, production output sampling must capture representative cases including edge cases and potential failure modes, not merely successful outputs. Sampling strategies should oversample for diversity and potential errors rather than optimizing for average-case performance.

Second, expert annotation interfaces must minimize cognitive burden while capturing rich feedback. Rather than requiring experts to complete exhaustive checklists, interfaces should enable rapid marking of specific issues with contextual explanations. The goal is capturing expert judgment efficiently rather than comprehensive labeling. Annotation sessions of focused duration (30-60 minutes) reviewing real cases prove more effective than extended labeling projects attempting complete coverage.

Third, retrieval and context assembly systems must identify relevantly similar prior cases and applicable expert corrections for each new output. Similarity metrics should reflect clinical similarity rather than mere textual overlap - cases involving similar diagnostic reasoning or decision points rather than similar word patterns. The retrieved context must be assembled into evaluation prompts or fine-tuning data that effectively transfers expert judgment to new cases.

A critical trade-off emerges between evaluation latency and accuracy. Dynamic context assembly requires additional computation compared to static rubric application, introducing latency that may be unacceptable for real-time applications. Hybrid approaches can apply fast static checks for obvious errors while routing uncertain cases to context-calibrated evaluation, balancing speed and accuracy based on application requirements.

5. Discussion

The findings illuminate a broader principle regarding AI evaluation in high-stakes domains: verification difficulty scales with the tacitness of quality standards. When quality can be fully specified in advance - as with transcription accuracy or format compliance - static rubrics and deterministic checks suffice. However, when quality depends on tacit domain expertise applied contextually, evaluation requires mechanisms that capture and transfer expert judgment case-by-case rather than attempting complete a priori specification.

This observation challenges conventional deployment paradigms that assume verification asymmetry universally. For tasks requiring contextual expertise, verification may be comparably difficult to generation, necessitating either restricting AI deployment to domains where quality is fully specifiable, or developing evaluation frameworks that continuously learn quality standards from expert judgment on real outputs.

The framework's emphasis on discovering failure modes from production outputs rather than anticipated error categories reflects a fundamental epistemological shift. Traditional software testing assumes failure modes can be enumerated in advance through requirements analysis and threat modeling. However, LLM failure modes in complex domains may be fundamentally unpredictable, emerging from the interaction of model capabilities, domain complexity, and specific use contexts in ways that resist a priori enumeration.

Future research should investigate the minimum expert annotation volume required for effective calibration across different domains, optimal retrieval strategies for identifying relevantly similar prior cases, and methods for detecting when captured expert judgment has become stale due to evolving guidelines or model changes. Additionally, the framework's applicability to domains with less clear expert consensus - such as creative writing or strategic business decisions - remains an open question.

6. Conclusion

Analysis of 847 production clinical AI notes demonstrates that conventional evaluation methodologies fail systematically in high-stakes domains where quality depends on tacit, contextual domain expertise. Frontier models equipped with detailed rubrics and deterministic checks miss approximately 20% of clinically significant errors because they lack "taste" - knowledge of what matters in specific contexts that cannot be fully codified in static criteria.

The Discovery-Capture-Calibrate framework addresses this limitation through continuous learning from production outputs: discovering how systems actually fail in practice, capturing expert judgment on real cases, and assembling case-specific evaluation context dynamically rather than applying uniform rubrics. This approach demonstrates superior performance by aligning evaluation capabilities with the contextual complexity inherent in high-stakes applications.

The practical implication extends beyond healthcare: any AI deployment in domains requiring contextual expertise - legal review, financial analysis, strategic decision support - faces similar evaluation challenges. Organizations deploying such systems should implement continuous evaluation loops that discover failure modes empirically, capture expert judgment systematically, and calibrate assessment case-by-case. The alternative - relying on static evaluation frameworks while hoping errors remain acceptable - amounts to flying blind in domains where failures carry serious consequences.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub