Stop Chunking Like It's 2022 - Yuval Belfer, AI21 Labs

Chunking is not dead - there is no single optimal chunk size because the best chunk size is query-dependent; using a multi-scale indexing approach with multipl...

By Sean Weldon

Stop Chunking Like It's 2022: A Case for Query-Dependent Multi-Scale Retrieval

Abstract

This synthesis examines a persistent claim in applied retrieval research: that Retrieval-Augmented Generation (RAG) and its constituent chunking stage have been rendered obsolete by agentic search. Drawing on experimental work from Yuval Belfer (AI21 Labs), this paper argues instead that chunking remains essential but is poorly optimized - specifically, that no single chunk size is universally optimal because the ideal granularity is query-dependent. Through duplicated-dataset experiments across QMSum, NarrativeQA, a Seinfeld trivia corpus, and FinanceBench, an oracle analysis reveals a 20-40% recall gap between any fixed chunk size and the theoretically best per-query choice. A proposed remedy - multi-scale indexing combined with Reciprocal Rank Fusion (RRF) - recovers most of this gap at a 2-5x memory cost and negligible latency penalty, offering a practical, low-complexity upgrade path for production retrieval systems.

1. Introduction

Retrieval-Augmented Generation has become a recurring target for premature obituaries. Commentators periodically declare that RAG, the Model Context Protocol (MCP), and document chunking have been superseded by agentic search - an approach in which a language-model agent autonomously navigates a corpus using primitives such as grep, ls, and find, reasoning its way to relevant content rather than relying on pre-built indices. This narrative has intuitive appeal: agentic exploration can be effective on small, well-structured codebases or documentation sets where a handful of queries are issued against a modest number of files.

The claim breaks down at scale. When corpora grow large and query volumes increase, agentic traversal becomes computationally and economically inefficient compared to indexed retrieval. A more accurate diagnosis, per this analysis, is that what agentic search has displaced is not retrieval itself but retrieval tuning - the manual, per-deployment optimization of retrieval-time hyperparameters (re-rankers, hybrid search weights, query rewriting) that many practitioners historically avoided or under-invested in. Chunking, by contrast, has remained neglected on both sides: it is neither replaced by agentic methods nor meaningfully improved since chunking heuristics popularized around 2022 (fixed chunk sizes near 512 tokens, 10-20% overlap).

This paper's central research question is whether a single, fixed chunk size can serve as an optimal default across the diverse queries a retrieval system must answer. The evidence assembled here - spanning meeting transcripts, novel-length narrative text, trivia-style factual queries, and financial documents - indicates that it cannot. The analysis proceeds by first framing chunking as an information-theoretic problem (Section 2), then presenting controlled experimental evidence for query-dependent optimal granularity (Section 3), followed by a description of the multi-scale indexing architecture and its trade-offs (Sections 4-5), and closing with implications for retrieval system design (Section 6).

2. Background and Related Work

A typical RAG pipeline decomposes into two stages with strikingly different levels of engineering attention. Pre-processing and chunking occurs once, at ingestion time, and is treated as infrastructural "plumbing" - configured with default parameters and rarely revisited. Retrieval, by contrast, is executed per query and is the locus of continuous experimentation: embedding model selection, re-ranking, hybrid sparse-dense search, and query rewriting all receive iterative attention. This asymmetry is consequential because chunking decisions made once at ingestion impose a hard ceiling on what downstream retrieval tuning can recover; a badly chunked corpus cannot be fully rescued by a better retriever.

Prior work has approached this ceiling from a complementary angle. Contextual retrieval, as described by Anthropic, augments individual chunks with document-level context before embedding, aiming to reduce the loss of surrounding context that occurs when a passage is chunked in isolation. The approach examined in this synthesis does not enrich individual chunks but instead questions the premise that a single segmentation granularity should be chosen at all. Evaluation throughout relies on Recall@K, the fraction of queries for which a relevant document appears among the top K retrieved results, supplemented by MTEB-style benchmark comparisons for broader validation.

3. Core Analysis

3.1 Chunking as Lossy Compression

Chunking is, in the framing advanced here, fundamentally a form of lossy compression: regardless of the segment size selected, some information is discarded. The nature of that loss differs at each end of the size spectrum. Large chunks preserve document-level structure and long-range relationships - useful for aggregate or thematic queries - but their embeddings average over heterogeneous content, degrading discriminative precision for narrowly scoped factual questions. Small chunks produce sharp, high-precision embeddings for localized facts but sever the surrounding context needed to answer questions that depend on relationships spanning multiple passages. A illustrative case is the "World Cup data" scenario, in which siloed, folder-based data requires aggregation across many chunks to answer aggregate queries - an operation that is inherently inefficient regardless of chunk size chosen.

3.2 Experimental Evidence for Query-Dependent Optimality

To test whether any fixed chunk size dominates across query types, the underlying datasets were duplicated six times, each copy chunked at a different fixed size ranging from approximately 100 to 2000 tokens. These parallel indices were evaluated against QMSum (meeting transcripts), NarrativeQA (novel-based question answering), an in-house Seinfeld trivia dataset, and FinanceBench.

The Seinfeld dataset furnishes a concrete illustration. A query about "Jerry's favorite shirt" ranks the correct passage first when using a small chunk size (100 tokens), where fine-grained detail is preserved. A structurally similar query about "Jerry's nemesis" (Newman) requires substantially larger chunks to rank well, since the relevant information is distributed across a broader narrative context. Plotting recall against chunk size for each dataset produces intersecting curves - the ranking of fixed chunk sizes by performance is not consistent across datasets or query types, meaning no single size dominates universally.

An oracle experiment formalizes this observation: for each individual query, the best-performing chunk size was selected retrospectively (a "genie" that always picks correctly). Comparing this oracle performance against any single fixed chunk size reveals a 20-40% recall gap, consistent across QMSum, NarrativeQA, the Seinfeld dataset, and FinanceBench. This gap represents the performance ceiling being forfeited by committing to any one static chunking configuration.

3.3 The Multi-Scale Indexing Solution

The proposed remedy avoids selecting a single chunk size at ingestion time. Instead, the database is duplicated across multiple chunk/window sizes (six sizes were used in the reported experiments). At query time, n parallel retrieval calls are issued - one against each differently-chunked index. Because chunks of differing sizes are not directly comparable in score space, retrieval is performed at the level of full documents rather than raw chunks, enabling a consistent basis for ranking comparison across scales.

The n resulting rankings are then merged using Reciprocal Rank Fusion (RRF), a simple, training-free formula that aggregates rank positions across multiple ranked lists - described in the source material as a form of "voting" rather than re-ranking. RRF was found to outperform other merging methods tested, without requiring any specialized model training.

4. Technical Insights

Several implementation considerations follow from this architecture. First, the memory overhead is a constant factor rather than an asymptotic cost: storing n chunked copies of a database corresponds to an O(1)-to-O(n) overhead, empirically observed as a 2-5x multiplier depending on how many chunk sizes are indexed. Second, latency impact is minimal because the n retrieval calls execute in parallel, and RRF's computation over the resulting rankings is inexpensive - no additional model inference is required at merge time. Third, retrieving full documents rather than raw chunks is a necessary design choice, since embeddings or scores produced at different chunk granularities are not directly comparable and would otherwise bias the fused ranking toward one scale.

Across the evaluated datasets, the multi-scale approach matched or exceeded the best-performing fixed chunk size, yielding 20-40% recall improvements consistent with the oracle gap identified in Section 3.2, and 10-40% improvement on MTEB-style benchmarks. Open questions remain around the optimal number and spacing of chunk sizes - the specific values used (approximately 50, 100, 200, and larger increments) were arbitrarily selected rather than derived from a principled search procedure. Additionally, alternative fusion algorithms beyond RRF have not been exhaustively explored and may yield further gains.

5. Discussion

The findings reframe the "RAG is dead" discourse. Rather than validating the obsolescence of retrieval infrastructure, the evidence suggests that retrieval tuning - not retrieval or chunking - may be the component genuinely displaced by agentic workflows in narrow contexts. Chunking itself remains an active bottleneck, and one that has received disproportionately little engineering attention relative to its downstream impact on recall.

The query-dependent nature of optimal chunk size has broader implications for how retrieval systems are evaluated and deployed. Benchmark practices that report a single "best" chunk size for a corpus may obscure substantial per-query variance, understating the achievable performance ceiling. This suggests future evaluation protocols should report oracle or upper-bound comparisons alongside fixed-configuration results, as done here, to reveal such gaps.

The multi-scale approach's reliance on RRF - a decades-old, training-free rank aggregation method - also underscores a broader pattern: substantial retrieval gains can be achieved through architectural reorganization rather than more sophisticated modeling. This is a notable counterpoint to trends favoring increasingly complex re-ranking and learned fusion models.

6. Conclusion

This analysis demonstrates that chunking remains a live and underexplored problem in retrieval system design, contrary to claims of its obsolescence. The core contribution is empirical: a demonstrated 20-40% recall gap between any fixed chunk size and per-query optimal selection, replicated across four distinct datasets. The proposed multi-scale indexing architecture, paired with Reciprocal Rank Fusion, recovers most of this gap at a bounded memory cost and negligible latency penalty, requiring no specialized model training.

For practitioners, the practical takeaway is straightforward: rather than fixing a single chunk size at ingestion, retrieval systems should consider indexing corpora at multiple granularities and fusing results at query time. Future work should address principled selection of chunk-size sets and explore fusion algorithms beyond RRF that may further close the remaining performance gap.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub