Better Search Took This AI Agent From 36% to 98% Accuracy with Aamir Shakir, Mixedbread

The real bottleneck in building effective AI agents for knowledge work is not model intelligence but access to and retrieval of the right data; traditional R

Better Search Took This AI Agent From 36% to 98% Accuracy

Abstract

This synthesis examines the claim that enterprise AI agent performance is bottlenecked by data retrieval rather than model intelligence, drawing on a case study from Mixedbread, a retrieval-focused research organization. Conventional Retrieval-Augmented Generation (RAG) pipelines combine independently sourced OCR, chunking, embedding, and vector database components into what the source terms the "Frankenstein stack," a lossy architecture that degrades at production document scale. The analysis traces retrieval's historical performance curve from BM25 (1994) through embedding models surpassing it in 2022, then details a late interaction, multimodal retrieval alternative that generates multiple contextualized embeddings per document rather than compressing content into a single vector. A reported hedge fund deployment spanning over one billion documents shows answer accuracy rising from 36% to 98%, with a threefold reduction in token spend. The findings suggest that retrieval fidelity, not reasoning capacity, is the dominant constraint on agent accuracy in knowledge-intensive enterprise settings, with practical implications for how organizations should allocate engineering investment between model selection and data infrastructure.

1. Introduction

As AI agents are deployed against proprietary enterprise corpora, their practical accuracy increasingly depends on the quality of evidence surfaced to the reasoning model rather than the model itself. The source material advances a specific position: "the real real bottleneck is actually access to your data and to your knowledge." This reframes a common assumption in agent development, where effort is concentrated on model selection and prompting strategy while the retrieval layer is treated as a solved or secondary concern.

Two terms anchor the discussion. Retrieval refers to the process of identifying relevant evidence from a corpus prior to generation. The "unbundling of intelligence and data" describes a projected market structure in which reasoning models become interchangeable commodities chosen independently of the data access layer, with the latter layer determining realized system performance. If this unbundling holds, organizations optimizing solely for model quality are addressing the wrong constraint.

This analysis is organized as follows. Section 2 situates the discussion within the historical trajectory of retrieval technology. Section 3 examines the specific failure modes of conventional RAG architectures and the proposed late interaction alternative. Section 4 distills implementation-relevant technical findings. Sections 5 and 6 discuss broader implications and practical takeaways for teams building retrieval-dependent agents.

2. Background and Related Work

Lexical retrieval via BM25, a term-frequency keyword matching method underlying systems such as Elasticsearch, dates to 1994 and remained the dominant approach for text search for approximately 27 years. Dense embedding-based search surpassed BM25 performance only in 2022. The source stresses that improvements since 2022 have been largely linear rather than step-function, in contrast to the discontinuous capability jump associated with ChatGPT in generative modeling. This context matters because it establishes that RAG, as widely deployed today, inherited an immature and slowly-evolving retrieval substrate rather than a mature one.

The organizational background for the case study is a research lab that focused exclusively on retrieval for two and a half years before productizing. The resulting commercial system vertically integrates its own models, its own database (to support zero data retention), and its own ingestion pipeline, rather than composing third-party components. It has been live for five months, with banks and hedge funds onboarded within the trailing three months at a described scale of billions of documents, operating under ISO compliance with a "bring your own bucket" deployment mode.

3. Core Analysis

3.1 Failure Modes of the Conventional RAG Pipeline

The canonical RAG pipeline - parsing or OCR, chunking, embedding, vector database storage, and reranking - composes five independently sourced stages, each selected from a different vendor. The source reports that evaluating and deploying such a stack typically consumes six to eight months and four to five engineers, internally termed the "Frankenstein stack." Each stage introduces information loss: OCR struggles with binary or scanned PDF formats, blocking downstream use of multimodal content; chunking severs documents into fragments that lose surrounding context; and single-vector embedding compresses an entire document or chunk into one point in space.

This last limitation is illustrated concretely. SEC filings containing tables of net income and revenue across different years and companies can produce embeddings that are nearly indistinguishable as single points, even though the underlying figures differ substantially. The result is retrieval that performs adequately against small test corpora of tens or hundreds of documents but degrades sharply in production, where volumes are orders of magnitude larger. A secondary consequence is agent behavior: when retrieval underperforms, agents compensate by spawning additional sub-agents and issuing more aggressive queries, which fills context windows and increases cost disproportionately as traffic scales. The source notes that naive lexical search (comparable to grep-style search in developer tools) exacerbates this further by returning high volumes of unranked, irrelevant results, with a single query sometimes costing $10 to $20 in token spend before any answer is produced.

3.2 Late Interaction and Multimodal Retrieval

The proposed alternative is a late interaction model, which generates multiple embeddings per document and per query rather than one embedding per document. This multi-vector representation permits fine-grained description of information within a single page or filing - for example, distinguishing net income from net revenue within the same document - because each embedding retains localized detail rather than averaging it away. Each embedding is additionally contextualized with surrounding information (e.g., recognizing that a filing pertains to a specific company) since it is produced in a single pass over the document rather than through isolated chunking.

A second architectural distinction is multimodal indexing: documents are processed as page screenshots rather than relying solely on OCR-extracted text, which preserves figures, diagrams, and layout information that text extraction would otherwise discard. The system still outputs layout information and OCR text for downstream consumption by non-multimodal language models, and supports bounding box citations that trace an answer back to its exact source location on a page. For qualitative or multi-part queries, the system escalates to an agentic search model trained to perform multihop reasoning across multiple decomposed queries, described in the source as comparable in quality to unspecified reference models ("Astra and fable") while being 40 times faster and cheaper.

3.3 Case Study: Hedge Fund Deployment

A hedge fund managing over one billion documents was onboarded within one and a half weeks, using the retrieval system within internal agent platforms for analyst and fund manager decision support. Prior to switching from a conventional chunking-embedding-vector-database stack, answer accuracy - graded against human expert answers - was 36%. After migration, accuracy rose to 98%. The source attributes a meaningful portion of this gain to multimodal handling beyond text-only question answering, and reports a threefold reduction in LLM token spend, attributed to the prior stack consuming most tokens on locating and parsing irrelevant information rather than on reasoning itself.

4. Technical Insights

Several implementation-relevant findings emerge. First, multi-vector (late interaction) embeddings address a structural limitation of single-vector representations: fine-grained distinctions within a document (e.g., adjacent financial line items) are not recoverable once collapsed into a single embedding point, regardless of embedding model quality. Second, visual/multimodal indexing is presented as a more faithful alternative to OCR-dependent pipelines for documents containing tables, figures, and layout-dependent meaning, at the cost of requiring the retrieval system to process raw page imagery. Third, bounding box citation support is positioned as a mechanism for traceability and verification, relevant to compliance-sensitive domains such as finance. Fourth, agentic search is reserved for complex or qualitative queries requiring multihop reasoning, rather than applied universally, suggesting a tiered retrieval strategy based on query complexity. Trade-offs are not fully specified in the source: no latency or cost breakdown is given for the agentic search tier beyond a sub-200-millisecond target for standard search, and the comparative benchmarks ("40 times faster and cheaper," parity with "Astra and fable") are vendor-reported without methodology disclosed.

5. Discussion

The central implication is that the much-discussed unbundling of intelligence and data may be already underway in practice: the hedge fund case study holds the downstream reasoning task roughly constant while varying only the retrieval layer, and attributes a 62 percentage-point accuracy gain almost entirely to that change. This is consistent with the broader observation that retrieval technology has historically advanced far more slowly than generative modeling - 27 years for embeddings to surpass BM25, versus the discontinuous leap in generative capability around ChatGPT - meaning that organizations may be pairing frontier reasoning models with retrieval infrastructure that has not kept pace.

A notable gap is the absence of independent benchmarking; all figures (36%, 98%, 3x cost reduction, 40x speed/cost advantage) are vendor-reported and self-selected as a case study, and no comparison against alternative late-interaction or multimodal retrieval systems from other vendors is provided. Future investigation would benefit from standardized, third-party benchmarks across document types and query complexity classes, as well as disclosure of agentic search methodology and failure cases.

6. Conclusion

This analysis supports the broader claim that retrieval fidelity, rather than model intelligence, is frequently the binding constraint on enterprise agent accuracy, and that single-vector embedding and OCR-dependent chunking pipelines are architecturally unsuited to large, heterogeneous document corpora. The practical takeaway for teams building retrieval-dependent agents is to evaluate multi-vector and multimodal retrieval approaches before or alongside reasoning model selection, particularly in domains with dense tabular or visually structured content such as financial filings. Given the reliance on a single vendor-reported case study, organizations should validate comparable gains against their own corpora and human-graded accuracy benchmarks before generalizing these results.

Sources

• Better Search Took This AI Agent From 36% to 98% Accuracy - Aamir Shakir, Mixedbread - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub