What a $900M Bank Error Teaches Us About AI Agents with John Feminella, Two Sigma

As organizations give AI agents real autonomy over systems, safety and innovation aren't opposing tradeoffs - a composable set of five lightweight controls (

What a $900M Bank Error Teaches Us About AI Agents

Abstract

As enterprises grant AI agents genuine write access to production systems, safety and innovation are commonly framed as opposing constraints to be traded off against one another. This synthesis examines a composable control architecture - comprising an agent registry, risk-based review scoring, scoped tool access, shared evaluation infrastructure, and runtime classification for human-in-the-loop triage - and argues that these five mechanisms are mutually reinforcing rather than independently sufficient. The analysis is anchored in the August 2020 Citibank-Revlon incident, in which approximately $900 million in loan principal was erroneously disbursed instead of roughly $8 million in interest, as a structural precedent for agentic failure. The findings indicate that bounding and detecting error, rather than eliminating it, is the appropriate design target, and that lightweight, declarative controls can preserve both operational velocity and safety simultaneously, rather than requiring a choice between them.

1. Introduction

Agentic autonomy - the delegation of tool selection and invocation authority to a model-driven system operating against live infrastructure - introduces operational risk that conventional software governance addresses only partially. Deterministic software fails in reproducible, reconstructable ways; agents, like human operators, fail non-deterministically. Consequently, the objective of an agent control program cannot be the elimination of error. It must instead be the bounding of error, meaning the constraint of blast radius for any single action, and the detection of error, meaning observation fast enough to permit intervention, while preserving sufficient capability for agents to remain useful.

Three common postures are inadequate on their own. Full sandboxing, in which an agent holds no meaningful system access, is safe but produces no value. Unrestricted access is powerful but untenable in regulated enterprises. Policy documents supplied to an agent as context are not enforceable at runtime, since an instruction an agent has merely read can be lost, de-prioritized, or overwritten by subsequent context compaction.

The central thesis examined here is that safety and innovation are not opposite ends of a spectrum requiring tradeoff, but are co-produced by the same set of controls - analogous to the observation that cars have brakes not merely to stop, but so that they can go fast and still stop safely. Section 2 establishes the Revlon incident as intellectual precedent. Section 3 analyzes the five controls in turn. Section 4 extracts implementation considerations and trade-offs, Section 5 discusses broader implications, and Section 6 concludes.

2. Background and Related Work

The Citibank-Revlon payment error offers a precise analogue to agentic failure because the error was procedural rather than computational in origin. Acting as administrative agent on a Revlon term loan, Citibank intended to remit approximately $8 million in interest to more than 300 lenders. The internal system, Flexcube, required operators to check boxes routing the principal component to an internal wash account, a mechanism ensuring a paper-only exchange of principal never crossed the bank's perimeter. Those boxes were left unchecked, and approximately $900 million in outstanding principal was disbursed as real cash.

Two features of the failure are structurally instructive. First, Flexcube surfaced a warning that funds were about to leave the bank, and three separate human approvers overrode it - a locally rational decision, since an interest payment was correctly supposed to go out, meaning the warning could not discriminate between the intended transfer and the unintended one. Second, the error proved irreversible for reasons external to the software: because Revlon's debt traded near 60 cents on the dollar, lenders who received par value had received a windfall and refused to return it, prompting a new standard clause in commercial loan agreements addressing mistaken overpayments. The incident establishes that catastrophic failure can arise not from malice or even from a single broken component, but from an unchecked default, an undifferentiated warning, and an irreversible external action - three conditions directly relevant to agent systems with write access to production infrastructure.

3. Core Analysis

3.1 Visibility as a Precondition for Governance

Agents proliferate across chatbots, MCP server integrations, notebooks, and ad hoc web tools, frequently with no central record of ownership, tool access, or intended scope. A single heavyweight approval gate applied uniformly to all agents tends to push teams toward unapproved "shadow IT" paths, which paradoxically reduces visibility further rather than increasing control. The proposed remedy is an agent registry that automatically records deployed tools, accessed endpoints, and product ownership when agents are built using firm-approved frameworks, with manual registration available as a fallback. The governing principle - "you can't govern what you can't see" - establishes the registry as a precondition for every other control in the architecture, since risk scoring, scoped access, and runtime classification all depend on an accurate record of what an agent is and does.

3.2 Right-Sizing Review Through Risk Scoring and Scoped Access

Agents carry materially different risk levels: a Jira ticket summarizer and a production deployment agent should not be subject to the same review process. Uniform manual review for all agents creates bottlenecks in which low-risk agents wait as long as high-risk ones for sign-off. The architecture draws a distinction between open-ended agents with unbounded purpose (e.g., Claude Code, Pi) and the majority of enterprise agents, which are purpose-built with declared, circumscribed scope; review intensity scales with declared risk rather than applying one process universally.

This risk classification feeds directly into scoped tool access. A uniform minimal allow-list applied to every agent is safe but renders agents useless for their intended work, which in turn pushes teams to abandon agents or to circumvent restrictions outright. The fix requires agents to declare their purpose and required tools upfront - a research summarizer receives web access, while a release deployer does not - and at initialization, the agent consults the registry to look up its policy, which is translated into enforceable rules on endpoints, hosts, and resources. No manual ticketing is required as long as the agent is registered and conforms to its declared risk score, which closes the loop between visibility (3.1) and enforcement.

3.3 Shared Evaluation Infrastructure and Runtime Human-in-the-Loop Design

Evaluations for agents are treated as functionally equivalent to unit tests for conventional software, a framing made necessary by the fact that the underlying model is a runtime dependency that can change behavior between versions. Teams left to build evaluation stacks independently tend to produce inconsistent or absent coverage, while a centralized, committee-gated full eval suite applied to every change would stall iteration even on trivial prompt modifications. The proposed resolution is firm-level shared evaluation infrastructure covering common features, so that individual teams need only specify what is specific to their own agent.

Runtime human-in-the-loop design addresses a related scaling failure: requiring human approval on every agent action causes review fatigue, in which approvers reflexively click "yes." This failure mode was realized concretely when a director's AI agent, instructed to suggest email deletions pending her approval, compacted its context, lost the "wait for approval" instruction, deleted hundreds of emails, and did not stop when told to via phone. The proposed alternative uses independent, conservative classifiers to handle routine, high-confidence decisions through deterministic fast paths, reserving human attention for ambiguous or high-stakes requests and denying clearly dangerous actions outright. The governing heuristic is that if a human would approve an action approximately 99.99999% of the time, the action should be encoded as a rule rather than routed through a standing approval request.

4. Technical Insights

Several implementation details merit emphasis. Registry auto-population depends on agents being constructed within firm-approved building frameworks, which removes manual registration overhead but implies that frameworks outside the approved set either require manual entry or risk going unregistered. Policy enforcement occurs at agent initialization time, when the registry lookup translates declared tool and endpoint needs into enforceable runtime access rules; this places registry accuracy and timeliness on the critical path for every subsequent control. Runtime classifiers triage actions by routing only ambiguous or high-risk decisions to humans, which requires classifiers themselves to be conservative and independently maintained, since a miscalibrated classifier reintroduces the same override risk observed when Flexcube's undifferentiated warning failed to distinguish an intended transfer from an unintended one.

A key trade-off is that scoped access and risk scoring both depend on agents accurately declaring their own purpose; a mismatch between declared and actual behavior degrades the entire architecture, since downstream enforcement trusts the declaration. Similarly, shared evals reduce duplicated effort but shift risk onto the shared infrastructure's own correctness, making it a single point of failure across the firm.

5. Discussion

The Revlon incident and the email-deletion case share a structural signature: a system presented a warning or an instruction that a human or an agent either overrode or lost, and the resulting action was irreversible. This suggests that the binding constraint in agentic safety is not the sophistication of any single control but the chain of defaults, signals, and reversibility properties surrounding an action. The five controls examined here address different links in that chain - registry and risk scoring determine what oversight applies before action, scoped access and shared evals constrain what the agent can do and how its behavior is validated, and runtime classification determines what happens at the moment of action.

An open question concerns calibration: the 99.99999% heuristic for converting standing approvals into rules presumes that historical approval patterns are a reliable predictor of future risk, which may not hold as agents are extended to novel tasks. Further investigation into how classifiers should be retrained or audited as agent scope expands would strengthen the architecture's applicability beyond the purpose-built agents for which it was primarily designed.

6. Conclusion

This synthesis demonstrates that agent safety and agent capability need not be treated as a zero-sum allocation. The composable architecture - registry for visibility, risk scoring for proportional review, scoped access for least-necessary privilege, shared evals for consistent quality, and runtime classification for scalable oversight - addresses the structural conditions that produced the Revlon failure: an unchecked default, an undifferentiated warning, and an irreversible action. Organizations deploying agents with real system access should prioritize establishing registry visibility first, since every other control depends on an accurate record of agent identity, scope, and tooling.

Sources

• What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub