How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI

Large language models have exponentially improved reasoning capabilities, but retrieval systems have stagnated, creating a 'knowledge gap' that limits agent ...

By Sean Weldon

Abstract

Large language models have achieved exponential improvements in reasoning capabilities, yet retrieval systems have evolved incrementally, creating a critical "knowledge gap" that constrains agent performance in knowledge-intensive tasks. This analysis examines the disparity between oracle performance - 93% on BrowseComp Plus and 64% on Office QA Pro - and actual agent performance with default tools (9% and 8% respectively), demonstrating that the primary bottleneck is knowledge access rather than reasoning capability. Through architectural innovations including multi-tool search frameworks (overview, semantic, filter, and grep tools), semantic query reformulation techniques, and two-stage training combining supervised fine-tuning with reinforcement learning, the knowledge gap can be substantially reduced. The trained agent achieves NDCG@10 of 0.4 on Oblique Congress benchmark compared to 0.18 baseline, with production implementations reaching 93.4% accuracy on Snowflake Cortex QA, demonstrating that targeted interventions in retrieval architecture yield performance improvements comparable to advances in reasoning capability.

1. Introduction

The rapid advancement of large language models has transformed artificial intelligence capabilities across knowledge-intensive domains. While reasoning abilities have improved exponentially from GPT-3.5 to GPT-5.5, a critical asymmetry has emerged: retrieval systems have evolved incrementally over the same period, improving slowly despite their foundational role in information access. This disparity creates what can be termed the knowledge gap - a performance bottleneck where sophisticated reasoning capabilities remain underutilized due to inadequate information retrieval mechanisms.

The magnitude of this gap becomes evident through oracle performance benchmarking, which measures theoretical maximum performance when agents receive perfect document retrieval. On BrowseComp Plus, oracle performance reaches 93%, yet agents using default tools achieve only 9% - an 84-point deficit. Similarly, Office QA Pro demonstrates 64% oracle performance versus 8% with standard tooling. These findings indicate that the primary constraint on agent performance is not reasoning capability but rather access to relevant knowledge documents. When agents receive appropriate documents, their reasoning capabilities prove sufficient for task completion, but in noisy corpus environments, performance degrades sharply.

This analysis examines the structural causes of poor retrieval performance in LLM agents, proposes architectural and methodological solutions through multi-tool search frameworks and reinforcement learning training, and presents empirical evidence demonstrating substantial performance improvements. The investigation reveals that semantic search implementations can recover most of the performance gap - reducing the BrowseComp Plus deficit from 84 points to merely 3 points (90% versus 93% oracle), while Office QA Pro performance nearly closes completely.

2. Background and Related Work

2.1 The Reasoning-Retrieval Asymmetry

LLM reasoning capabilities have followed an exponential growth trajectory, with successive model generations demonstrating marked improvements in complex problem-solving, logical inference, and multi-step reasoning tasks. In contrast, information retrieval systems have evolved incrementally over two decades, with traditional approaches like BM25 (Best Matching 25) remaining dominant for keyword-based search despite the emergence of neural and semantic search alternatives. This asymmetry creates a fundamental mismatch between what models can reason about and what information they can effectively access.

Semantic search systems leverage neural embeddings to capture conceptual similarity beyond keyword matching. Unlike BM25, which optimizes for term frequency and document length normalization, semantic search encodes queries and documents into dense vector representations, enabling retrieval based on semantic proximity. However, agents trained primarily on coding tasks or exposed predominantly to traditional web search patterns fail to leverage these advanced capabilities effectively, instead defaulting to grep-style regular expression patterns or keyword-on-keyword formulations optimized for BM25 rather than neural search architectures.

2.2 Benchmark Bias and Query Formulation Patterns

Existing benchmarks introduce structural biases that reinforce suboptimal search patterns. BeerNano and similar evaluation frameworks employ entity-based "caveman-style" queries - short, keyword-focused formulations that structurally favor BM25 over semantic search. Agents trained on these benchmarks learn to mimic human web search patterns, guessing keywords to increase query-document overlap rather than formulating natural language descriptions that leverage semantic search capabilities. This training bias perpetuates inefficient retrieval strategies even when more powerful tools are available.

3. Core Analysis

3.1 Multi-Tool Search Agent Architecture

The proposed architecture employs four complementary search tools designed to address different retrieval scenarios. Overview search returns up to 50 chunks with summaries, providing broad corpus coverage for initial exploration. Semantic search returns the top 10 full chunks based on neural embedding similarity, enabling deep retrieval of conceptually relevant content. Filter chunks enables metadata-based retrieval for structured queries, while grep tool supports keyword matching for precise term-based searches.

The architecture constrains agents to a maximum of four search rounds with parallel search capability within each round, balancing exploration depth against computational efficiency. Agents receive the initial user query, semantic search results preview, and metadata facet hints for search planning at initialization. Critically, agents deduplicate chunks across rounds to prevent context window bloat while maintaining a comprehensive information set. This design prioritizes speed while preserving exploration possibilities, outputting a final ranked list of relevant chunks for downstream reasoning tasks.

3.2 Prompt Engineering for Semantic Query Formulation

Traditional prompt patterns inadvertently reinforce BM25-style query formulation. The research identifies several techniques to break these patterns and encourage semantic query generation. Goal framing instructs agents to articulate needed evidence before writing queries, separating information need identification from query construction. Tool diversity prompting explicitly encourages agents to select appropriate tools based on task requirements rather than defaulting to a single pattern.

Most significantly, query instruction reframing replaces the directive "write a search query" with "write one concise sentence describing what it wants to find." This subtle linguistic shift disrupts learned BM25 patterns and encourages natural language formulation suitable for semantic search. Few-shot examples in prompts demonstrate effective query formulation and illustrate how to divide complex input queries into explorable aspects. Providing original query semantic search results exposes agents to corpus language and content characteristics, informing subsequent exploration strategies with domain-specific terminology and document structure patterns.

3.3 Two-Stage Training Methodology

The training methodology employs a small LLM for the agent to optimize speed and efficiency, implementing a two-stage approach. Supervised fine-tuning with a larger teacher LLM provides initial behavioral guidance, establishing baseline search strategies. Subsequently, on-policy reinforcement learning with custom search rewards refines agent behavior through direct interaction with the retrieval environment.

The retrieval reward function combines NDCG (Normalized Discounted Cumulative Gain) metrics on the final ranked list with LLM-based judging using structured rubrics evaluating result relevance, chunk relevance, and ranking plausibility. This hybrid approach balances objective ranking quality with nuanced semantic assessment. The trajectory reward function employs LLM judges to evaluate query naturalness, exploration sufficiency (detecting both excessive and insufficient search), and overall search strategy quality. Through this dual reward structure, agents optimize for improved tool choice, higher semantic query quality, enhanced exploration patterns, and operational efficiency.

4. Technical Insights

4.1 Performance Metrics and Benchmark Results

Empirical evaluation demonstrates substantial performance improvements across multiple benchmarks. On the Oblique Congress benchmark, the trained agent achieves NDCG@10 of 0.4, compared to a GPT multi-hop agent baseline of 0.18 - representing a 122% improvement. The Snowflake Cortex QA benchmark shows Mixedbread genetic search achieving top-1 ranking with 93.4% accuracy using Gemini 3.5 Flash, demonstrating that production implementations maintain strong performance with reduced computational requirements compared to larger model architectures.

The oracle performance analysis reveals the magnitude of retrieval impact: BrowseComp Plus shows an 84-point gap between oracle (93%) and default tools (9%), while Office QA Pro demonstrates a 56-point gap (64% versus 8%). Implementing Mixedbread semantic search reduces the BrowseComp gap to merely 3 points (90% versus 93% oracle), recovering 96% of the theoretical maximum performance. These results quantify the knowledge gap and demonstrate that retrieval optimization yields performance gains comparable to reasoning capability improvements.

4.2 Implementation Considerations and Trade-offs

The four-round maximum search architecture with parallel query execution represents a deliberate trade-off between exploration depth and latency. This constraint prevents excessive iteration while enabling sufficient refinement for complex information needs. Chunk deduplication across search rounds manages context window constraints but requires careful implementation to preserve information coherence when chunks appear in different retrieval contexts.

The small LLM agent design prioritizes speed and cost efficiency, accepting modest reasoning capability reductions in exchange for substantial operational benefits. This design choice proves viable because the architecture offloads complex reasoning to downstream components, focusing the search agent specifically on retrieval optimization. The reinforcement learning training approach requires careful reward function design to avoid degenerate solutions - agents might optimize for superficial metrics without improving actual retrieval quality if rewards lack nuanced semantic evaluation.

5. Discussion

The findings demonstrate that the knowledge gap between reasoning and retrieval capabilities represents a critical bottleneck in agent performance, yet this gap proves amenable to targeted architectural and training interventions. The substantial performance recovery achieved through semantic search implementation - from 9% to 90% on BrowseComp Plus - indicates that retrieval optimization offers returns comparable to advancing model reasoning capabilities. This insight has significant implications for resource allocation in agent development, suggesting that investments in retrieval infrastructure may yield higher marginal returns than continued scaling of reasoning models alone.

The structural causes of poor query formulation - training bias toward coding tasks, benchmark design favoring BM25 patterns, and mimicry of human web search behavior - reveal that current agent development practices inadvertently teach suboptimal retrieval strategies. Breaking these patterns requires deliberate intervention through prompt engineering, architectural constraints that encourage tool diversity, and training methodologies that directly optimize retrieval quality. The success of query instruction reframing demonstrates that subtle linguistic modifications can substantially alter agent behavior, highlighting the importance of prompt design in shaping search strategies.

The two-stage training methodology combining supervised fine-tuning with reinforcement learning addresses the challenge of optimizing for complex, multi-dimensional retrieval quality. Traditional supervised approaches struggle to capture the nuanced trade-offs between exploration depth, query quality, and efficiency, while pure reinforcement learning faces sample efficiency challenges in retrieval environments. The hybrid approach leverages teacher model knowledge for initialization while enabling direct optimization of retrieval-specific objectives through environment interaction. Future research might explore alternative reward formulations, investigate transfer learning across retrieval domains, and examine the scalability of these techniques to larger corpus sizes and more complex information needs.

6. Conclusion

This analysis establishes that the knowledge gap between exponentially improving reasoning capabilities and incrementally advancing retrieval systems represents a primary bottleneck in agent performance on knowledge-intensive tasks. Oracle performance benchmarking quantifies this gap at 84 points on BrowseComp Plus and 56 points on Office QA Pro, demonstrating that reasoning capability suffices when appropriate documents are provided. Through multi-tool search architectures, semantic query reformulation techniques, and two-stage reinforcement learning training, this gap can be substantially reduced, achieving NDCG@10 of 0.4 compared to 0.18 baseline and recovering 96% of theoretical maximum performance on certain benchmarks.

The practical implications extend beyond specific architectural choices to broader principles of agent design. Prioritizing retrieval optimization alongside reasoning capability development, deliberately counteracting training biases toward keyword-based search patterns, and implementing hybrid reward functions that balance objective metrics with semantic quality assessment emerge as key strategies for closing the knowledge gap. As language models continue advancing, ensuring that retrieval systems evolve commensurately becomes increasingly critical for realizing the full potential of agent capabilities in real-world knowledge-intensive applications. The demonstrated performance improvements suggest substantial room remains for further enhancement in LLM search tool integration, representing a high-value direction for continued research and development.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub