Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait - Al Chen, Cisco

Deploying AI agents in production at banks requires rigorous, evidence-based processes across CVE remediation, disaster recovery, and data leakage prevention...

Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait

Abstract

This synthesis examines a three-pillar operational framework for deploying AI agents in production within regulated banking environments: build pipeline and Common Vulnerabilities and Exposures (CVE) remediation, data resiliency and disaster recovery (DR), and sensitive information leakage prevention. Drawing on field experience deploying on-premises, air-gapped AI observability infrastructure across five major banks, the analysis argues that demonstrated evidence, rather than asserted compliance, is the primary determinant of deployment velocity. Mechanisms examined include agent-driven exploit verification harnesses, record-level correctness checks for ClickHouse backup and restore operations, jurisdiction-aware data replication topologies, and a dual LLM-as-judge pattern that prevents leakage of personally identifiable information (PII) and material non-public information (MNPI) through reasoning traces. Findings indicate that time-to-production can compress from approximately twelve months to several weeks when evidentiary artifacts, rather than claims, are presented to compliance and security reviewers. Trade-offs in computational cost and organizational friction are discussed.

1. Introduction

Deploying artificial intelligence systems inside regulated financial institutions differs fundamentally from conventional Software-as-a-Service (SaaS) deployment. A standard SaaS AI product retains outbound egress to the public internet, creating a persistent channel through which sensitive data may leave the institutional perimeter. Banks, by contrast, frequently require air-gapped, on-premises deployment, which removes that channel but introduces a substantially more complex operational flow: container images must be sourced and internally approved, failover must be engineered rather than purchased as a managed service, and any observability output must be sanitized before reaching a human reviewer.

The central thesis of this analysis is that time-to-production in such environments is governed not by a vendor's technical capability but by that vendor's ability to produce verifiable evidence satisfying compliance and security functions. These functions behave as rigorous examiners operating with substantial budgets, analogous to an instructor who demands answers to the fourth decimal place specifically to exclude results obtained by chance rather than demonstrated method. Engineering effort, therefore, must be redirected from building features toward producing artifacts of proof.

This paper organizes the problem using a Three Pillars Framework: (1) the build pipeline and CVE remediation, (2) data resiliency and disaster recovery, and (3) prevention of PII/MNPI leakage. Section 2 situates these pillars within the broader context of legacy financial software. Section 3 analyzes each pillar's mechanisms in detail. Section 4 distills technical insights with implementation guidance. Section 5 discusses implications and open questions.

2. Background and Related Work

Financial software infrastructure has historically resisted modernization because the consequences of failure - disrupted checking accounts, mortgages, credit cards - are borne directly by consumers, producing institutional conservatism disproportionate to typical enterprise software risk tolerance. This conservatism manifests as a requirement that new infrastructure simultaneously satisfy heterogeneous vulnerability scanners, DR mandates, and data residency statutes, each enforced by a distinct internal constituency with independent authority to block deployment.

Two structural features compound the problem. First, institutions standardize on different container vulnerability scanning tools - Sysdig, Wiz, and Prisma are commonly encountered - each applying distinct severity classifications and distinct service-level agreements (SLAs) for remediation. Second, institutions run differing versions of the same vendor images, multiplying the vulnerability surface combinatorially across multiple images, multiple CVEs per image, and multiple banks each running a different version. The Three Pillars Framework used throughout treats each domain not as a checklist item to be marked complete, but as an ongoing evidence-production problem requiring repeatable, automatable processes.

3. Core Analysis

3.1 Build Pipeline and CVE Remediation

Compliance velocity is directly coupled to an institution's confidence in a vendor's CVE remediation process; no credible on-premises platform can proceed without one. Open-source transient dependencies frequently introduce CVEs that engineering teams cannot remediate directly, forcing a choice between bumping package versions or rebuilding against updated base Docker images. This choice is complicated by inter-bank disagreement: one institution may require a higher package version while another, citing its own CVE concerns, requires a lower one, producing conflicting constraints across a five-bank deployment footprint.

A key finding is that standard scanning tools verify only package version strings, not whether the underlying exploit remains present in application code. As one practitioner observes, "it's not enough just to say we bumped the package version up to a specific number. We have to provide the evidence to show that the exploit is no longer actually in the application code." This gap motivates an agent-driven verification approach using Harbor tasks: a test harness is constructed by writing instructions, provisioning a Docker image environment, writing exploit code, and testing whether remediation actually neutralizes the exploit. Harbor Modal or Daytona sandboxes enable parallel trials to evaluate agent performance across many CVE-remediation attempts simultaneously, producing a scalable system that outputs a structured table of fixable versus unfixable CVEs with accompanying explanations - the evidentiary artifact compliance teams require.

3.2 Data Resiliency and Disaster Recovery

Disaster recovery is treated by banks as a mandatory requirement rather than optional infrastructure hygiene. A significant technical constraint arises because ClickHouse, the primary database underlying AI observability platforms, lacks a managed cross-region failover service, unlike Redis or Postgres - a limitation acknowledged in ClickHouse's own documentation. The resulting pattern involves a primary-region ClickHouse instance backing up to a storage bucket, with a scheduled job restoring that backup to a disaster-recovery region's replica pods.

A critical limitation is that the ClickHouse backup tooling, maintained by Altinity (a ClickHouse competitor), reports only whether the backup and restore jobs succeeded, not whether the underlying data is correct at a given point in time. Stale or corrupted data can result from bad disks, bad nodes, bad migrations, or replication lag - none of which a simple job-success log would surface. Consequently, additional verification steps are required: comparing row counts between source and restored tables, and confirming replica convergence within a region before relying on restored data.

A related concern is data residency: because storage buckets do not inherently respect jurisdictional boundaries, residency compliance becomes an implicit property of backup-and-restore architecture rather than an explicit control. The adopted solution replicates only identifiers and metadata across regions while retaining core trace data within a single region, satisfying jurisdictional requirements without sacrificing cross-region operational visibility.

3.3 Preventing PII/MNPI Leakage

Banks may believe their redaction mechanisms are comprehensive while retaining side-channel leakage paths for PII and MNPI. Redaction can be implemented at the software development kit (SDK) level, the gateway level, or in middleware, but UI-level redaction alone is insufficient: the reasoning text produced by an LLM-as-judge evaluating a trace may contain unredacted sensitive information even when the displayed record itself is clean. A specific failure mode arises when a judge's system prompt includes a line intended to assist human verification, which can cause the judge to reproduce or un-redact PII/MNPI within its own reasoning output.

The proposed mitigation persists the judge's score unconditionally while suppressing the reasoning text from the user interface whenever an automated check flags the record as containing sensitive information. An enhanced solution runs a second LLM-as-judge call on the already-redacted input, generating a clean reasoning output suitable for display. This dual-judge pattern is not without cost: achieving zero leakage and verifiable data sovereignty requires an additional LLM inference call per request, a direct trade-off between compliance assurance and operating expense.

4. Technical Insights

Several implementation-level findings generalize beyond the specific deployments examined. First, Harbor task harnesses provide a reusable pattern for converting CVE remediation claims into verifiable, agent-tested evidence, applicable to any environment requiring exploit-level proof rather than version-level assertion. Second, the absence of managed failover for ClickHouse implies that any AI observability platform built on it must independently engineer and verify - not merely execute - backup/restore correctness, since job-success signals from tools like Altinity's backup utility are necessary but insufficient. Row-count comparison and replica-convergence checks represent minimal viable verification steps. Third, jurisdiction-aware replication (metadata only, cross-region; trace data, single-region) offers a template for reconciling observability needs with data residency law without full data duplication. Fourth, the dual LLM-as-judge pattern demonstrates that redaction must be evaluated at every stage of a pipeline, including model-generated reasoning artifacts, not solely at ingestion or display points - a distinction with direct relevance to any system employing LLMs as evaluators over sensitive data.

5. Discussion

The findings collectively suggest that AI deployment velocity in regulated industries is primarily an evidence-engineering problem rather than a model-capability problem. Vendors asserting deployment timelines of "a few weeks" without accounting for a specific institution's scanning tools, image-approval processes, GPU procurement workflows, and security review cadence are likely to underestimate actual timelines by months. The data indicate that delays are overwhelmingly attributable to internal human and organizational factors within banks - absent image-approval pipelines, unclear GPU sourcing, undefined RBAC configuration - rather than to the underlying AI technology itself.

This observation connects to a broader industry pattern in which legacy financial infrastructure persists precisely because the cost of verification failure is borne by consumers holding checking accounts, mortgages, and credit cards. Future work might formalize the CVE remediation scale/system described here into a generalized benchmark for cross-institutional vulnerability tracking, and further investigate automated methods for continuous data-correctness verification beyond row-count comparison, such as checksum-based or statistical sampling approaches for ClickHouse replica validation.

6. Conclusion

This analysis has presented a three-pillar framework - build pipeline/CVE remediation, data resiliency/DR, and PII/MNPI leakage prevention - derived from deploying on-premises AI observability infrastructure at five major banks. The central contribution is the demonstration that "showing your work," through agent-tested exploit remediation, record-level backup verification, and dual-judge redaction patterns, functions as the primary lever for compliance velocity. Practically, engineering teams pursuing bank deployments should prioritize building reusable evidence-generation infrastructure over feature development, recognizing that per-request cost increases and organizational friction are structural trade-offs rather than solvable inefficiencies.

Sources

• Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait - Al Chen, Cisco - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub