Memory-Aware Agent Harnesses for Anti-Money Laundering - Nacho Martinez, Oracle

Building effective AI agents depends less on the frontier model itself and more on the 'agent harness' - the customizable data layer (memory, context enginee...

Memory-Aware Agent Harnesses for Anti-Money Laundering: A Technical Synthesis

Abstract

This synthesis examines the proposition that large language model (LLM) agent performance is governed less by the frontier model selected than by the agent harness - the configurable data layer comprising memory, context engineering, tool/skill management, and the agent loop. Drawing on a technical workshop centered on an autonomous bank fraud-detection agent built on Grok 4.3 and Oracle Cloud Infrastructure, the analysis decomposes the harness into constituent subsystems, surveys storage trade-offs between file-based and transactional database approaches, formalizes a taxonomy of agent memory, and documents retrieval techniques for managing tool and skill proliferation. Quantitative observations indicate tools and skills may occupy approximately 80% of a chat completions request (35,000-100,000 tokens), that context window performance degrades well before advertised limits, and that semantic caching mitigates a potential four- to fivefold cost increase. Implications for practitioners building compliance-grade autonomous agents are discussed throughout.

1. Introduction

The rapid commoditization of frontier reasoning models has shifted the locus of engineering differentiation away from model selection and toward system design. Where early systems competed on raw model capability, contemporary practice indicates that two teams consuming the identical model can produce agents of markedly different quality depending on the scaffolding surrounding that model. This observation motivates sustained attention to what is termed the harness.

An AI agent is defined as a computational entity powered by an LLM supplying reasoning, augmented by a database supplying memory, extended through tools enabling action, and grounded in inputs providing perception - analogous to the human brain, memory, extremities, and senses, respectively. The harness is the scaffolding mediating these four faculties; it determines, exhaustively, what an agent is capable of doing. As stated in the source material, "these harnesses actually control everything that the AI agent can do."

The central thesis is that the harness, rather than model selection, constitutes the highest-leverage object of engineering attention: "you give it a different configuration then the agent - the same LLM frontier AI model that you're consuming - becomes better or worse." This paper situates the harness within the broader agent stack (§2), analyzes its core subsystems through the lens of a bank fraud-detection workshop (§3), extracts implementation-level findings (§4), and discusses open questions and industry trends (§5).

2. Background and Related Work

The agent stack may be stratified into four layers: the application layer, the model serving as reasoning core, infrastructure, and compute. Of these, only the intermediate data layer is substantially controllable by most engineering teams; compute provisioning and frontier model weights are largely externally determined. The data layer comprises the agent loop, a gateway or Model Context Protocol (MCP) interface to external data, the memory layer, context engineering machinery, and the tool/skill registry.

Contemporary AI applications fall into four recognizable forms: LLM chatbots, retrieval-augmented generation (RAG) applications, LLM-driven workflows, and AI agents. The most capable production systems - Claude Code, Codex, and comparable coding agents - are not pure instances of any single form. They constitute agentic systems fusing deterministic workflow automation with genuine agentic autonomy. This hybridization motivates the workshop's design choice to combine scripted SQL retrieval (via MCP) with autonomous reasoning for suspicious-activity classification.

3. Core Analysis

3.1 Storage Substrates: Files versus Databases

A live methodological dispute separates "files maximalists" from "database maximalists." The file-oriented camp has converged on git worktrees to resolve concurrent writes by multiple agents to shared files. This solution is functional but architecturally regressive, since ACID transactional consistency addressed the same concurrent-write problem three decades earlier: "the workaround has been already solved by databases 30 years ago. And we're trying to reinvent the wheel with some of this." The practical resolution favors a hybrid: files for large unstructured data, databases for structured, concurrency-sensitive state.

3.2 Memory Architecture and Context Degradation

Agent memory divides into short-term memory (the context window, which vanishes after a conversation concludes) and long-term memory, further subdivided into episodic (past conversations, tool logs, summaries), procedural (skills, workflows), and semantic (user preferences, implicit context) categories. A critical finding is that context window performance degrades as token count rises, well before advertised limits of 256k or 1 million tokens - limits enabled technically by the KV cache but not reflective of sustained model reasoning quality: "the smaller the context window is the better it will solve a problem... this applies for every type of problem that you can think of in humanity." This motivates two complementary context-engineering techniques: summarization, which generates a condensed restart point, and compaction, which externalizes conversation chunks to a database or file referenced by ID.

3.3 Tool and Skill Retrieval at Scale

In the fraud-detection workshop, the agent must select from a growing library of investigative skills (e.g., pattern recognition for structuring, such as a $1 million deposit followed by three days of $50,000 ATM withdrawals). Because each skill consumes 3,000-5,000 tokens, naively loading an entire skillbox is infeasible once the library exceeds a handful of entries. The workshop instead applies a toolbox/skillbox pattern: tools and skills are embedded as vectors, and similarity search retrieves only the top-k most relevant candidates (e.g., top 3) per query. A reranker can be layered atop this retrieval step to further refine selection among the top-k set, preventing unnecessary items from consuming context budget. Over time, a skill distillation process - converting completed agent runs into new skill.md files appended to the skillbox - allows the system to accumulate institutional knowledge, an approach referenced to the Hermes agent architecture.

3.4 Hybrid Search and Scalability

Retrieval quality is further improved via hybrid search, combining semantic (embedding-based) search with keyword matching through Reciprocal Rank Fusion (RRF), which the workshop found to outperform either method in isolation. This matters because regular-expression or keyword matching degrades quadratically as repository size grows linearly, whereas embedding-based retrieval (using vectors of 384 to 1,000+ dimensions) scales more gracefully while capturing semantic relationships not accessible through literal text matching - a material consideration as a bank's skill and SQL-query library expands across transaction-monitoring categories.

4. Technical Insights

Several quantitative findings carry direct implementation consequences. Tools and skills can occupy more than 50% - and in the measured instructional hierarchy, roughly 80% (35,000-100,000 tokens) - of a typical chat completions request, despite occupying the lowest position in the instruction hierarchy (system prompt > developer instructions > user query > tools/skills). This asymmetry between token footprint and instructional priority suggests that indiscriminate tool loading is both costly and architecturally misaligned with intended model behavior.

Semantic caching, as implemented by OpenAI and comparable providers, substantially mitigates this cost by avoiding recomputation of repeated tool/skill content; absent such caching, requests could cost four to five times more. Harness efficiency varies considerably across implementations: the referenced Pi harness uses approximately 6,000 tokens in its initial system prompt/request, compared to 30,000-40,000 tokens in less optimized harnesses - a tenfold difference attributable entirely to harness design rather than model choice.

The workshop's agent loop was configured with a maximum of 8 iterations and 60 seconds of reasoning time per iteration, an explicit cost and runaway-behavior control mechanism. On the database-integration side, MCP's principal value is enabling pre-validated, proven SQL queries rather than requiring the agent to generate SQL at inference time - a meaningful safeguard given that 15+ distinct SQL dialects exist across database engines, each with potential for hallucinated or engine-incompatible syntax. Finally, continual learning strategies were categorized into parametric memory (weight modification, e.g., LoRA fine-tuning), representational memory (embeddings), and nonparametric memory (the memory layers and context-engineering techniques constituting the bulk of this analysis) - with the workshop's focus concentrated entirely on the nonparametric category as the most immediately actionable for practitioners.

5. Discussion

These findings collectively reposition the engineering problem of agent development. Rather than treating model upgrades as the primary lever for capability gains, the evidence suggests that harness-level interventions - memory architecture, retrieval strategy, and context budget management - produce comparable or greater performance differentials using the same underlying model. This has direct implications for regulated domains such as anti-money-laundering compliance, where auditability of retrieval (via pre-validated MCP queries) and bounded agent autonomy (via iteration limits) are not merely performance optimizations but risk-management requirements.

A notable tension remains unresolved between files-based and database-based storage paradigms, with the field apparently reconstructing solutions - transactional consistency - that database systems have offered for decades. This suggests an opportunity for closer integration between the agent engineering and database systems communities rather than parallel reinvention.

Open questions include the long-term reliability of skill distillation pipelines (whether automatically generated skills maintain quality without human review), the generalizability of the toolbox/skillbox pattern beyond the finance domain, and the degree to which reranking overhead is justified across varying skill-library sizes.

6. Conclusion

This synthesis demonstrates that harness design - spanning memory architecture, context engineering, tool/skill retrieval, and agent loop configuration - functions as the dominant controllable variable in agent performance, with measured effects (80% token share for tools/skills, 4-5x cost differentials without caching, tenfold harness efficiency gaps) exceeding what model selection alone typically achieves. Practitioners building autonomous agents, particularly in regulated domains like fraud detection, should prioritize investment in retrieval architecture, bounded agent loops, and database-backed memory consistency over continual frontier model migration. The fraud-detection workshop demonstrates these principles are implementable today using commodity infrastructure, suggesting the harness is not a theoretical construct but an immediately actionable engineering discipline.

Sources

• Memory-Aware Agent Harnesses for Anti-Money Laundering - Nacho Martinez, Oracle - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub