A Bank Trained Its Own Foundation Model to Catch Fraud - Freddy Lecue, Wells Fargo

Wells Fargo built a transaction-based foundation model, pre-trained from scratch on billions of customer transactions, that serves as a reusable backbone for...

A Bank Trained Its Own Foundation Model to Catch Fraud: An Analysis of Wells Fargo's Transaction Foundation Model

Abstract

This synthesis examines the design, training, and deployment of a transaction foundation model (TFM) developed at Wells Fargo: a decoder-only transformer pre-trained from scratch on billions of customer transactions to serve as a reusable backbone across multiple downstream financial tasks. The work addresses a structural gap in applied machine learning for financial services, where the dominant data modality is tabular and event-sequential rather than linguistic, rendering off-the-shelf pre-trained language models unsuitable. The approach substitutes transaction sequences for natural-language token sequences, trains with a causal language modeling objective, and fine-tunes the resulting backbone for classification. On fraud detection, the model exceeded an incumbent XGBoost production system on both detection rate and false-positive rate while achieving inference latency roughly an order of magnitude below the 250ms decision window. Reported business impact is measured in millions of dollars of annual savings, with the same backbone now under evaluation for credit line management, anti-money laundering, and marketing.

1. Introduction

Foundation models have reorganized natural language processing around a simple pattern: pre-train once on broad data, adapt cheaply to many tasks. Porting this pattern to enterprise domains governed by structured, tabular data has proven less straightforward, since the pretraining corpora, tokenization schemes, and architectures developed for text do not map cleanly onto transaction records. Financial services exemplify this gap. Debit, credit, digital, and check transactions are described by structured fields - merchant identity, amount, channel, timestamp, location - rather than free text, and the incumbent modeling paradigm, gradient-boosted decision trees (XGBoost), depends on extensive hand-engineered feature pipelines built around fixed, pre-aggregated vectors.

The central claim examined here is that the foundation-model paradigm can be transferred to this domain by redefining the base modality itself. As stated in the source material: "So instead of having languages as the base, the base is now transactions." A transaction foundation model is thus defined as a transformer pre-trained on sequences of customer transactions using a self-supervised objective, intended for reuse across multiple downstream classification tasks rather than built bespoke for a single use case.

This raises two empirical questions: whether a transformer trained from scratch on transaction sequences learns representations general enough to transfer across business functions, and whether such a model can surpass a mature tree-ensemble baseline on both predictive accuracy and the strict latency constraints of real-time payment authorization. The analysis below covers data representation and tokenization, architecture and training methodology, fraud-detection outcomes, and the broader implications for applying foundation-model methodology to structured enterprise data.

2. Background and Related Work

Three bodies of work underpin the design. The transformer architecture, through self-attention, allows modeling of dependencies between a candidate transaction and an arbitrarily long history of prior transactions - a capability absent from XGBoost's fixed-feature-vector formulation. Chinchilla scaling laws, which relate model parameters, training tokens, and compute budget, provide a principled basis for selecting parameter count and decoder depth rather than relying on ad hoc architecture search. Finally, the standard self-supervised objectives of language modeling - causal language modeling (CLM), predicting the next token from preceding context, and masked language modeling (MLM), reconstructing corrupted positions - furnish candidate pretraining objectives transferable to sequences of discrete transaction events.

The incumbent baseline, XGBoost, represents prevailing practice in tabular finance modeling, typically paired with large hand-crafted feature sets. Its structural limitation is that it evaluates each transaction largely in isolation against engineered aggregate features, rather than attending dynamically over full historical sequences, motivating the shift to a sequence-modeling architecture.

3. Core Analysis

3.1 Data Representation and Feature Reduction

The model is trained on billions of monthly transactions spanning debit, credit, digital, and check channels, organized along five dimensions: who (customer/merchant), what (purchase type), how (channel), when, and where. A notable methodological choice is the reduction of thousands of hand-engineered features down to approximately 100 raw "core" features. Rather than encoding domain knowledge through manual feature construction, the transformer is allowed to learn relationships among raw fields directly, shifting the locus of representation learning from feature engineers to the model itself.

3.2 Tokenization and Vocabulary Design

Because transaction data mixes numerical values (amounts, time deltas), categorical values, and identifier-type fields (zip codes, IP addresses), tokenization cannot reuse schemes built for natural language. The pipeline proceeds through value-to-token mapping, tokenization, and embedding lookup, with careful attention paid to which raw values become discrete tokens to avoid an unmanageably large vocabulary. This is identified in the source material as a critical design axis: "the recipe or the secret sauce is really how you do the tokenization and then how you're going to do the post training." The vocabulary itself is domain-specific, precluding reuse of pre-trained language-model vocabularies or embeddings.

3.3 Architecture, Scaling, and Training Efficiency

The model is a decoder-only transformer trained via a causal language modeling objective to predict the next transaction/token in a customer's sequence, mirroring autoregressive language modeling but over transactional events rather than words. Chinchilla scaling-law mathematics guided decisions on parameter count, decoder layers, and tokens-per-parameter ratios. An initial assumption of a 20x tokens-per-parameter ratio was revised downward to approximately 10x following experimentation, indicating that optimal scaling ratios established for text corpora do not transfer directly to transaction sequences.

A 1-billion-parameter model was selected as the initial production candidate, reported as offering the best return on investment relative to size, with an 11-billion-parameter version in subsequent development. Training employed mixed precision, distributed data parallelism, large micro-batch sizes, RoPE positional encoding, flash attention, and activation checkpointing - a standard large-model training toolkit applied to a novel modality. Initial pre-training required approximately two weeks, and the model was trained entirely from scratch, with zero reuse of pre-trained open weights; only architectural concepts were adapted from the literature.

3.4 Downstream Adaptation: Fraud Detection

The pretrained backbone was fine-tuned for fraud classification as the first downstream task. The attention mechanism's capacity to relate a candidate transaction to the complete historical sequence of past transactions is presented as the key mechanistic advantage over XGBoost, which relies on static aggregated features rather than dynamic sequence context. In production, inference latency was approximately 10x faster than the required 250ms decision window, while detection rate improved and false-positive rate simultaneously decreased relative to the XGBoost baseline - a joint improvement notable because these two metrics typically trade off against one another. As stated in the source: "if you decline a transaction which is not fraudulent, then that person is going to call," underscoring that false-positive reduction carries direct operational cost implications beyond raw accuracy metrics.

4. Technical Insights

Several implementation considerations generalize beyond this specific deployment. First, feature reduction from thousands of engineered variables to roughly 100 core raw fields suggests that transformer-based sequence models can substitute representation learning for manual feature engineering in tabular/event-sequence settings, provided sufficient pretraining data exists. Second, scaling-law assumptions imported from language modeling require empirical validation in new modalities; the revision from a 20x to 10x tokens-per-parameter ratio indicates that transaction sequences may carry different information density than text. Third, tokenization design is identified as a primary determinant of downstream performance, particularly for mixed numerical/categorical/identifier fields, with vocabulary size control as a key constraint. Fourth, standard large-model training efficiency techniques (mixed precision, RoPE, flash attention, activation checkpointing, distributed data parallelism) transferred successfully to this non-text domain without apparent modification. A limitation implicit in the account is that detailed ablations on tokenization choices and the 11-billion-parameter model's performance are not yet reported, leaving open questions about scaling behavior beyond the initial 1-billion-parameter deployment.

5. Discussion

The reported outcome - simultaneous improvement in detection rate, false-positive rate, and latency relative to a mature XGBoost production system - carries significance beyond a single use case, since it suggests that foundation-model methodology can be competitive with, rather than merely an alternative to, specialized tabular models in latency-critical financial infrastructure. This runs counter to a common assumption that transformer-based models, despite representational advantages, are too slow for sub-second decisioning; the 10x latency margin reported here addresses that concern directly for this deployment.

The reusability claim - a single backbone extended toward credit line management, anti-money laundering, and marketing - is the more structurally significant contribution, as it reframes the economics of model development from per-task construction toward amortized infrastructure investment, analogous to the foundation-model pattern in NLP. However, the source material does not yet report quantitative outcomes for these additional downstream tasks, so the generality of the backbone remains partially unverified pending further publication. Open questions include how the model's performance scales at 11 billion parameters, how tokenization choices affect transfer across tasks with different label distributions, and whether the causal language modeling objective remains optimal or whether masked objectives offer complementary benefits for classification-oriented downstream tasks.

6. Conclusion

This analysis demonstrates that a decoder-only transformer, pre-trained from scratch on billions of raw transaction records using a causal language modeling objective, can be fine-tuned to outperform a production XGBoost fraud model on detection rate, false-positive rate, and inference latency simultaneously. The core technical contributions are a reduction in hand-engineered features to roughly 100 core fields, a domain-specific tokenization pipeline for mixed tabular data types, and empirically revised scaling-law ratios for transaction sequences. Practically, the work suggests that organizations with large volumes of structured, sequential transactional data may benefit from investing in a shared pretrained backbone rather than maintaining siloed, task-specific tabular models, provided sufficient engineering attention is paid to tokenization design and latency-constrained inference requirements.

Sources

• A Bank Trained Its Own Foundation Model to Catch Fraud - Freddy Lecue, Wells Fargo - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub