Don't Approve the AI Model. Approve the App with Doneyli De Jesus, Langfuse

In regulated industries like banking, AI teams must certify applications (not models) against custom-built, domain-specific golden datasets and multi-dimensi

Don't Approve the AI Model. Approve the App

Abstract

Public model benchmarks exert substantial influence over procurement and architecture decisions in the artificial intelligence (AI) industry, yet they provide little evidentiary value in regulated environments where neither the underlying data nor the use cases are public. This synthesis examines a deployment pattern observed in banking, in which an institution declined to evaluate models in isolation and instead certified complete applications against internally constructed evaluation suites. The approach combines a golden dataset of domain-authored test cases, execution traces as evidence, and multi-dimensional pass/fail scoring with per-dimension thresholds set by domain experts. A reported case study demonstrates that augmenting a retrieval-augmented agent with a deterministic calculator tool raised numerical accuracy from approximately 60% to above 95%, clearing an 85% gate. The pattern reduced risk and compliance review cycles from weeks or months to minutes, suggesting that evaluation infrastructure, rather than raw model capability, constitutes the binding constraint on regulated AI deployment.

1. Introduction

The deployment of Large Language Models (LLMs) into regulated workflows exposes a structural mismatch between how model quality is reported publicly and how it must be demonstrated internally. Public leaderboards measure general-purpose capability against open datasets that are, by construction, decoupled from any specific institution's data, policies, or operating procedures. Regulated institutions, however, must demonstrate that a particular application, operating on proprietary data under institutional policy, produces outputs that can be verified against a defined standard. These are fundamentally different claims, and conflating them leads teams to over-index on benchmark scores that have no bearing on production readiness.

The central thesis examined here is that regulated AI teams should certify applications rather than models, using custom-built, domain-specific evaluation datasets and multi-dimensional gates whose thresholds are established by domain experts rather than by engineers alone. This position was articulated directly to the presenting team by a banking client:

"We don't certify models here. We certify applications."

Key terminology established in this domain includes the golden dataset (a curated, domain-authored set of test inputs and expected behaviors analogous to a QA suite in software engineering), the trace (the recorded execution path of an agent responding to a golden dataset item, serving as evidence), and the gate (a per-dimension pass/fail threshold, set by domain experts, that an application's scores must clear before a configuration change can ship).

This synthesis is organized as follows. Section 2 establishes why public benchmarks lack evidentiary standing in regulated settings and describes the infrastructure constraints shaping these deployments. Section 3 analyzes the certification pipeline, a case study in tool-augmented numerical reasoning, and the scoring methodologies employed. Section 4 consolidates actionable technical findings and trade-offs. Sections 5 and 6 discuss broader implications and practical takeaways.

2. Background and Related Work

Two structural factors explain the limited utility of public benchmarks in banking contexts. First, benchmarks are not evaluated on internal data or internal use cases, so any claim of performance transfer is unestablished. Second, institutions treat their data, use cases, and model-building processes as proprietary "secret sauce," and therefore avoid publishing the artifacts that would be required to construct a comparable public benchmark in the first place. Some institutions maintain research divisions staffed with doctoral researchers dedicated to training and fine-tuning internal models, further decoupling internal practice from public reporting norms.

A representative use case illustrates the gap: a 10-K analyst tool intended to accelerate investment bankers' analysis of earnings call transcripts and regulatory filings. A general-purpose model possesses no ingrained context regarding how a particular firm operates, which sources it treats as authoritative, or how its analysts phrase and qualify conclusions. The engineering problem is therefore twofold: supplying that institutional context, and reliably evaluating whether generated outputs reflect it. This framing motivates the application-level certification approach analyzed below, which is implemented using a Langfuse-based evaluation pipeline covering dataset management, tracing, and scoring.

3. Core Analysis

3.1 Infrastructure Preconditions for Regulated Deployment

Three infrastructure requirements consistently frame these deployments. On-premises deployment is common, with institutions repurposing legacy data centers and standing up GPU infrastructure to run models locally, driven by data sensitivity. Gated access prohibits outbound internet calls even where model providers offer zero-data-retention guarantees, reflecting residual institutional concern about training-data exposure. Air-gapped operation restricts data access to authorized personnel operating entirely within the bank's own infrastructure. These constraints establish the operating environment within which any evaluation methodology must function; they also explain why reliance on externally hosted benchmarks or evaluation services is frequently untenable.

3.2 The Certification Pipeline: Dataset, Trace, Score

The core methodology produces a pass/fail stamp of approval whenever an application or agent undergoes a configuration change - a modified system prompt, a model swap, or a newly added tool. The pipeline proceeds in three stages. First, a golden dataset serves as the anchor, comprising domain-expert-authored test cases analogous to a QA suite in conventional software engineering. Second, the agent under test is run against each item to generate a trace, which constitutes the evidentiary record of the agent's behavior. Third, each trace is scored across multiple chosen dimensions, producing a pass/fail result per dimension rather than a single aggregate figure. Many banks already maintain golden datasets internally but lack a mechanism to operationalize systematic testing against them, which is the gap this pipeline addresses. Datasets used in reported testing ranged from 150 to 1,500 questions.

3.3 Case Study: Numerical Accuracy via Deterministic Tool Use

A concrete case study demonstrates the diagnostic value of multi-dimensional scoring. An initial model configuration scored 97% on groundedness - indicating strong grounding of responses in internal source documents - but only approximately 60% on numerical accuracy. This gap is attributable to a known limitation: LLMs are next-token predictors and are inherently weak at arithmetic and financial calculation, a limitation that persists regardless of grounding quality. The remediation was architectural rather than model-centric: the agent was given a deterministic calculator tool to offload any task requiring mathematical computation, since such tasks are deterministic and do not require language-model inference. This single change raised numerical accuracy from 60% to above 95%, clearing the bank's 85% threshold gate for that dimension. The resulting agent pipeline - planning, retrieval from official sources, calculation via tool, and multi-dimensional evaluation - illustrates how targeted tool augmentation can resolve failures that no amount of prompt engineering on the base model would address.

3.4 Scoring Methodologies and Hard Gates

Three scoring methodologies are combined rather than used in isolation. Deterministic scores are code-based, producing identical outputs for identical inputs, and are not LLM-mediated. Path or trajectory scoring evaluates whether the agent followed the correct sequence of steps to reach an answer, independent of whether the final answer appears correct - a distinction that matters because a correct-looking output can mask an incorrect or non-compliant process. LLM-as-judge scoring is reserved for qualitative dimensions of an answer that resist deterministic evaluation. The stated justification for blending methods is that reliance on LLM judges alone degrades as solution complexity increases. Critically, evaluation is multi-dimensional and not reducible to a single average: one reported run scored well on groundedness and completeness but zero on regulatory compliance, a dimension treated as binary (0 or 100) that functions as a hard gate-blocker overriding an otherwise passing average.

4. Technical Insights

Several implementation considerations follow from the reported pipeline. The regulatory compliance dimension is treated as binary rather than continuous, reflecting the asymmetric cost of compliance failures relative to other quality dimensions - a design choice with direct implications for how gate logic should be structured in similar systems (hard blockers versus weighted averages). The combination of deterministic, trajectory, and LLM-judge scoring addresses a known limitation of LLM-as-judge approaches at scale, but introduces engineering overhead: each dimension requires its own scoring implementation and threshold, set collaboratively with domain experts rather than unilaterally by engineers. The tool-augmentation case study generalizes beyond this specific deployment: any dimension where failure is attributable to a known LLM weakness (arithmetic, date logic, exact retrieval) is a candidate for deterministic tool offloading rather than prompt-level mitigation. A limitation of the source material is that it does not specify the tool-calling architecture's failure modes - for example, how the system handles cases where the model fails to invoke the calculator tool correctly, which would itself require evaluation. Dataset sizes of 150-1,500 items suggest the approach is tractable for narrow, well-scoped use cases but leaves open how golden datasets scale as application surface area grows across many internal tools.

5. Discussion

The broader implication is that evaluation infrastructure, not model capability, is the binding constraint on regulated AI deployment. The case study shows that a 35-percentage-point accuracy gain came not from a better model but from recognizing, via multi-dimensional scoring, exactly which capability was deficient and routing that capability to a deterministic tool instead. This reframes the "model selection" problem common in less regulated settings as a secondary concern relative to building the measurement apparatus that reveals where any given model, tool configuration, or prompt falls short.

This pattern also connects to a wider industry shift toward agentic architectures composed of multiple components (retrieval, planning, tool use) rather than monolithic model calls, since gating applications rather than models presupposes that quality can be decomposed and attributed across such components. An open question is how domain-expert-authored golden datasets and gate thresholds themselves should be validated or audited over time, since the credibility of the entire certification framework rests on the quality of that initial authorship - a human bottleneck the source material acknowledges but does not fully resolve.

6. Conclusion

This synthesis documents a certification pattern in which regulated institutions evaluate complete applications - golden dataset, execution trace, multi-dimensional score - rather than relying on public model benchmarks disconnected from internal data and use cases. The reported case study demonstrates that diagnosing a specific capability gap (numerical accuracy) and addressing it with a deterministic tool, rather than a different model, produced the decisive improvement, while a human-in-the-loop annotation and golden-dataset-update cycle reduced risk and compliance review time from weeks or months to minutes.

The practical takeaway for AI engineers operating in regulated or high-stakes domains is to write organization-specific benchmarks, gate the application rather than the model to preserve flexibility across model swaps, and involve domain experts directly in setting per-dimension thresholds, since technical performance metrics alone are insufficient to establish production readiness.

Sources

• Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub