Building the AI Agent Is Easy. Proving It's Safe Isn't with Jonathan Osborne, NatWest

In a regulated banking environment, building an AI-powered customer-facing agent is the easy part given modern coding assistants; the hard and critical chall

Building the AI Agent Is Easy. Proving It's Safe Isn't

Abstract

This synthesis examines the engineering and assurance practices developed for deploying generative agentic systems in a highly regulated retail banking environment, drawing on NatWest's migration of its digital assistant Cora from scripted dialogue trees to retrieval-augmented, action-taking agents. The central claim is that agent construction has become inexpensive relative to agent assurance: the binding constraint is demonstrating safety at a scale of approximately 19 million retail customers and 100,000 daily conversations. Four mechanisms are analyzed: a deterministic routing funnel that minimizes exposure to generative inference, quality measurement as a multi-dimensional vector across safety, task, and experience, continuously calibrated LLM-as-judge evaluation, and staged, reversible deployment via shadow mode and feature flags. Findings from human validation and judge-calibration exercises show that aggregate accuracy scores and uncalibrated automated judges systematically conceal failure concentrations, with direct implications for how evaluation infrastructure should be designed.

1. Introduction

Agentic systems - large language model (LLM) applications that retrieve grounded content and execute consequential actions such as payments or address changes - introduce a materially different risk profile from the scripted conversational systems they replace. NatWest, the second-largest retail bank in the United Kingdom, serves approximately 19 million retail customers through Cora, which handles roughly 100,000 conversations per day. As of this year, approximately one third of Cora journeys are migrating to AI-generated answers backed by retrieval-augmented generation (RAG), curated knowledge bases, and action-capable agents.

The defining risk is not model quality in isolation but blast radius: at this scale, a small proportion of incorrect utterances does not translate into a defect backlog, but into real customer money, regulatory scrutiny, and reputational exposure. Given that modern coding assistants render agent implementation comparatively trivial, the operative question becomes what architectural, evaluative, and operational apparatus is required to demonstrate that such an agent is safe before and after it reaches production.

This analysis proceeds by contrasting scripted and agentic failure surfaces (§2), examining four principal risk-reduction mechanisms - routing, multi-dimensional quality measurement, judge calibration, and staged rollout (§3) - consolidating actionable technical findings (§4), and discussing broader implications for regulated AI deployment (§5).

2. Background and Related Work

The limitation motivating migration is illustrated by a representative mortgage query: "Can I overpay my mortgage without being charged?" A keyword-matching scripted system fails to resolve this intent and forces the customer into a menu, typically terminating in escalation to a human agent. A RAG-based approach answers directly - for example, that overpayment of up to 10% of balance per annum incurs no early repayment charge - and accommodates topic digression, such as a subsequent request to close the account. Traditional decision trees handle such digressions poorly, since their failure surface is enumerable but their coverage is not.

The same capability that resolves this limitation generates its inverse liability: an agent able to answer any question is also able to answer any question incorrectly. The 10% figure may not apply to a specific customer's product terms, with direct financial consequence. This motivates four failure modes distinct from scripted-system failures: being confidently wrong (hallucination delivered with full assurance), being susceptible to prompt injection (the input itself becomes adversarial), being right but unhelpful (accurate, on-policy responses that loop without resolution), and non-determinism (identical inputs yielding divergent outputs, which invalidates assertion-based regression testing). A scripted system's failure surface is fully bounded and exhaustively testable; an agent's is not, which is the foundational premise for the architecture and evaluation regime described below.

3. Core Analysis

3.1 The Routing Funnel: Shrinking the Attack Surface

Rather than exposing every customer utterance to generative inference, Cora routes requests through a layered funnel designed to minimize generative involvement wherever deterministic handling suffices. A contextual filter first removes non-linguistic events such as button clicks, eliminating approximately 20% of traffic. Remaining utterances pass through deterministic pre-classification rules that route using utterance and metadata with no machine learning involvement, followed by a gated relevancy rephrasing step for detailed, multi-context questions. Only at this stage does an intent classifier - a neural network operating over embeddings, not a generative model - predict the correct backend. A final policy rule layer verifies that the given customer should actually access that backend.

This architecture is justified on four grounds: access control, auditability (every routing decision carries a code, fully reproducible for regulators), resilience (a fallback answer remains available if generative inference is unavailable), and governance (the funnel is version-configured and can be altered without full redeployment). The funnel's function is not to replace evaluation but to reduce the proportion of interactions for which generative, non-deterministic behavior is even possible.

3.2 Quality as a Vector, Not a Scalar

A single aggregate accuracy figure - "94% accurate" - conceals whether answers are grounded, whether they resolve the customer's issue, or whether the system recognizes when to escalate to a human. Quality is instead decomposed into three independently monitorable dimensions: Safety (could the response harm the customer?), Task (did it accomplish its purpose?), and Experience (how did it feel to interact with?). Decoupling these dimensions distinguishes hallucination from a "right but insufferable" interaction - different failure classes requiring different remediation.

This decomposition is corroborated by human validation of a payment agent: across 111 conversations spanning 37 journeys and 4 brands, scored by three independent human raters each, a headline 83% pass rate concealed a 54-percentage-point spread - 97% on happy-path scenarios, 76% on edge cases, and only 43% on topic-change scenarios. Inter-rater agreement itself varied sharply by metric: 95% agreement on bias/fairness (an objective criterion) versus only 64% agreement on intent understanding (a subjective criterion). This divergence has a direct methodological consequence: because human agreement on intent understanding is itself only 64%, an LLM-as-judge cannot be trained to match human judgment as unambiguous ground truth on that dimension.

3.3 Calibrating LLM-as-Judge Evaluation

The case for actively calibrating automated judges, rather than trusting them at face value, is illustrated by an internal complaints agent launched in February, for which approximately 14,000 cases were scored by both human validators and an LLM judge. On a faithfulness metric, the judge's mean score was 0.97 with narrow spread, versus a human mean of 0.83 with substantially wider spread. The average error between judge and human scores was only 0.2, which superficially suggests reasonable alignment - but correlation between the two was near zero, indicating the judge was not discriminating good cases from bad ones and was therefore functionally worthless despite its low average error.

A second finding concerns model substitution: swapping only the judge model, with identical inputs and outputs held constant, caused the margin above a configured decision threshold to collapse from comfortably above 0.8 to approximately 0.2 above threshold. The judge model is therefore itself part of the system's configuration surface and can silently introduce breaking changes. Partial mitigation was achieved by selecting a judge model from a different model family than the agent being evaluated, reducing shared blind spots between judge and judged.

3.4 Staged, Reversible Deployment

Risk is further managed through deployment mechanics rather than pre-release testing alone. Shadow deployment allows a new model to classify live customer utterances while the existing scripted system continues to respond, producing real production data with near-zero customer-facing risk. This resolves a structural deadlock common to regulated industries, in which release approval requires evidence of reaction to live data, but obtaining live data normally requires release. Rollout then proceeds brand by brand with percentage-based exposure and KPI-driven go/no-go checkpoints, some automated and some requiring manual sign-off. Feature flagging provides a global kill switch reverting all traffic to the scripted system, as well as the ability to isolate individual backends globally or along specific dimensions such as brand, within seconds.

4. Technical Insights

The production evaluation pipeline captures live conversations between agents and guardrails using open standards (OTel, OpenInference), sampling 5% of traffic by default plus all "interesting" conversations - handoffs, failures, long conversations, and expressions of customer frustration. Scoring occurs via LLM-as-judge across the safety/task/experience vector, with traces, scores, and experiments logged in Comet ML. The environment is fully air-gapped with no general internet access; the judge model is reached through a private path via an internal GenAI gateway, and Comet is consumed as a signed partner SaaS application through AWS, reflecting a zero-tolerance policy on PII leaving the network.

Evaluation cadences differ by purpose: traces are captured live, offline scoring runs every four hours, and regression testing is embedded in CI/CD with a hard gate blocking merges until behavioral impact is understood - treating continuous evaluation as production infrastructure rather than a one-off notebook exercise. Monitoring also differs by component type: generative components require regression gates because prompt or context changes can have outsized behavioral effects, with span-by-span trace visualization used to localize slowness or failure; the traditional embedding-based intent classifier instead drifts gradually as conversational patterns evolve, monitored via input feature drift and F1 score against human labels, with input feature drift noted as an early canary that precedes measurable F1 decline. Comet is used to unify both observability needs - tracing for generative components and model performance monitoring for classical ML - within a single platform.

In the complaints agent, human feedback was implemented as a mandatory gate rather than a voluntary rating: a complaint handler cannot rerun an investigation without first providing feedback explaining the rerun. This produced a labeled dataset of approximately 14,000 colleague and customer labels, substantially larger than typical for comparable projects. The two internal projects cross-pollinated: the complaints project's judge-generosity finding led to the judge calibration and regression practices now applied more broadly, while the complaints agent is adopting the Comet-based observability architecture originally built for Cora.

5. Discussion

The recurring pattern across these findings is that automated proxies for quality - a single accuracy number, an unscrutinized judge score, an average error metric - tend to understate risk by masking variance. A router that is 94% accurate but 100% confident in incorrect cases is characterized as more dangerous than one that is 85% confident and recognizes its own uncertainty, because the former offers no signal by which downstream systems or operators can intervene. This suggests that confidence calibration, not just accuracy, should be treated as a first-class evaluation target for production agents.

The judge-calibration findings also carry implications beyond this specific deployment. LLM judges exhibit a default tendency toward generosity, and this tendency must be actively countered through distributional comparison - examining spread and correlation rather than averages alone - and through awareness that the judge model itself is a configuration dependency subject to silent drift upon substitution. This reframes "LLM-as-judge" evaluation from a static scoring utility into a component requiring its own versioning, regression testing, and change management discipline, analogous to the agent it evaluates.

A broader implication concerns organizational allocation of effort. The reported experience suggests that agent construction, aided by contemporary coding assistants, is no longer the binding constraint on deployment velocity in regulated contexts; the evaluation and observability "spine" - routing, multi-dimensional scoring, judge calibration, and reversible rollout infrastructure - is the durable and reusable asset across projects, while the agent itself is comparatively disposable.

6. Conclusion

This analysis has shown that safe deployment of agentic systems in a regulated, large-scale retail banking context depends less on agent construction than on a surrounding apparatus of deterministic routing, multi-dimensional quality measurement, continuously calibrated automated judging, and staged, reversible rollout mechanics. Empirical evidence from human validation and judge-calibration exercises demonstrates that single-number quality metrics and uncalibrated LLM judges systematically conceal failure concentrations that matter at scale.

The practical takeaway is that evaluation infrastructure - not the generative model itself - merits primary engineering investment, with judge models treated as versioned, regression-tested components rather than static scoring tools, and deployment treated as a sequence of small, reversible decisions rather than a single release event.

Sources

• Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub