It’s Tokens All The Way Down: How RLMs are Different - Kevin Madura, AlixPartners

Recursive Language Models (RLMs) treat context as a symbolic object living in a REPL environment rather than as tokens stuffed into a context window, allowin...

By Sean Weldon

It's Tokens All The Way Down: How RLMs Are Different

Abstract

Recursive Language Models (RLMs) represent an architectural departure from conventional long-context handling by treating context as a symbolic object addressable within a programmatic environment - typically a Python REPL - rather than as tokens inserted directly into a model's context window. This synthesis examines the mechanisms by which RLMs delegate subtasks to recursive sub-model calls, insulating the root context from intermediate computation and thereby mitigating context rot. Drawing on benchmark evidence including an accuracy improvement from 2.6% to 45.4% on a long chain-of-thought task, and favorable cost-performance comparisons against GPT-5 with BM25 retrieval, this analysis situates RLMs against RAG, tool calling, and coding agents. Findings indicate RLMs are most advantageous for decomposable, high-volume-context tasks, with open-source implementations (Predict RLM, Fast RLM, DSPy, Compound) already deployed in production settings spanning invoice consolidation, security auditing, and agent-harness optimization.

1. Introduction

The prevailing approach to supplying external information to large language models (LLMs) treats the context window as a fixed, finite resource that must be curated before inference. This premise has produced an entire engineering discipline - context engineering - concerned with chunking, embedding, retrieval ranking, and pruning. A well-documented failure mode of this paradigm is context rot, wherein model reliability degrades as the context window fills with information, producing what practitioners informally term the "dumb zone."

Recursive Language Models (RLMs) propose an alternative premise: rather than pre-selecting which fragments of context a model requires, the full context is materialized as a variable within an execution environment - typically a Python REPL - that the model queries and manipulates programmatically. Instead of exchanging JSON or string payloads with external tools, as in conventional tool calling, an RLM interacts with context symbolically and can recursively delegate subtasks to sub-language-model calls, whose intermediate results remain within the environment rather than being surfaced to the root context window.

As stated in the source material, an RLM "treats the context as an object that it can interact with symbolically in its environment." This distinction - context as an addressable variable rather than as inlined tokens - forms the central thesis of this analysis: that such indirection improves both accuracy and cost-efficiency on long-context and decomposable tasks relative to RAG, tool calling, and standard coding agents.

This synthesis proceeds by establishing the intellectual origins of RLMs (Section 2), analyzing their mechanisms and benchmark performance (Section 3), extracting implementation-level technical insights (Section 4), and discussing broader implications and open questions (Sections 5-6).

2. Background and Related Work

The RLM concept traces to a proposal by Omar, an adviser on the RLM research, describing DSPy-like techniques for summarizing arbitrary-length input. The critical conceptual move was not to optimize context management within the existing paradigm, but to challenge the premise that context windows must be deliberately managed at all.

Philosophically, RLMs are described as "bitter lesson pilled" - an allusion to the observation that general methods leveraging computation tend to outperform methods encoding human-designed structure as models scale. As articulated in the source material: "As models get better, you should be able to defer more and more to the model for it to figure out on its own what it needs to do." This principle underlies the RLM design choice to specify only a deterministic shell - inputs, outputs, and guidance - while leaving implementation details to the model itself, a mental model borrowed directly from DSPy.

Three prior architectural families provide the comparative baseline. RAG retrieves candidate passages and inserts them into the context window, inheriting context rot as retrieval breadth increases. Agents return strings to a calling process, offering no tight coupling between caller and callee state. Tool calling and codec-style interfaces pass JSON or string payloads to be interpreted elsewhere, with results still landing in the same context window and thus reproducing the original constraint. RLMs are distinguished by storing both context and sub-computation results as variables in a shared environment rather than as serialized messages - a design pattern notably echoed in Anthropic's recently released "workflows" feature, which the RLM paper is cited as having directly influenced.

3. Core Analysis

3.1 Mechanism: Context as a REPL Variable

The defining mechanism of an RLM is that the full context is loaded as a variable within a REPL environment rather than tokenized into the model's active context window. The root LM inspects, slices, and transforms this variable programmatically and can delegate subtasks to sub-LM calls - potentially instances of the same model - operating within the same environment. Because only the results of these sub-computations are returned to the root context, the root model's effective context load remains small even when the underlying data is large. As the source material notes, "RLMs somewhat get around this problem because the context itself doesn't fill up as quickly."

This mechanism directly addresses a documented limitation of standard LLMs: unreliable execution of simple aggregation tasks over long inputs. The illustrative example - summing 12 numbers scattered across 30,000 tokens - is unreliable for a base LLM operating purely over token context but is straightforwardly solved via regex or code execution when the RLM treats the input as a queryable object.

3.2 Benchmark Evidence

Empirical evidence supports the mechanism's practical value. On long-context benchmarks such as ULong and BrowseComp, the RLM approach outperformed comparison baselines, including tool-calling configurations using GPT-5 combined with BM25 retrieval, which proved both more expensive and lower-performing. Independent testing on a long chain-of-thought benchmark (attributed to Raymond) reported an accuracy increase from 2.6% to 45.4% when applying the RLM approach relative to a non-RLM baseline - a substantial margin suggesting the mechanism is not merely incremental but addresses a structural limitation of context-window-bound inference.

RLMs were reported to perform particularly well on tasks amenable to code-based reasoning, including logic puzzles, chess, and chemistry problems, consistent with the hypothesis that symbolic, programmatic manipulation of context is advantageous when subproblems admit deterministic or algorithmic solutions.

3.3 Comparative Positioning Against Existing Paradigms

Relative to RAG, RLMs avoid the need to pre-determine retrieval granularity, since the full corpus can be held as a variable and queried selectively rather than chunked and embedded in advance - demonstrated in the invoice consolidation case study (Trampoline AI), which applied Predict RLM without manual chunking or embedding strategy design. Relative to coding agents such as Claude Code, comparative testing showed a "bloated" approach when solving equivalent tasks, suggesting that unstructured agentic coding does not automatically capture the efficiency benefits of the RLM's tighter environment coupling.

4. Technical Insights

Several implementation-level patterns recur across the open-source RLM ecosystem (Predict RLM, Fast RLM, DSPy, Axe, Compound):

A notable trade-off is that RLM effectiveness depends on tasks being decomposable and amenable to programmatic manipulation; tasks lacking this structure may not benefit proportionally.

5. Discussion

The evidence assembled here suggests that RLMs address context rot not through better curation but through architectural avoidance - by ensuring the root model is simply never exposed to the full volume of context in token form. This reframes context engineering less as a retrieval-optimization problem and more as an environment-design problem, in which the central task becomes specifying appropriate input/output schemas and delegation boundaries rather than tuning chunk sizes or embedding models.

The convergence of Anthropic's "workflows" feature with RLM-style variable-based intermediate state storage suggests this architectural pattern is gaining broader industry traction beyond a single research lineage. This raises an open question, acknowledged in the source material, regarding what becomes possible if base models are explicitly post-trained to be "RLM aware," potentially altering how models internally represent delegation and recursion rather than treating it as an external scaffold.

Underexplored areas include large-output generation tasks, where the RLM paradigm's advantages for large input handling may or may not transfer symmetrically.

6. Conclusion

This synthesis has examined RLMs as an architectural alternative to token-window-bound context handling, demonstrating measurable benefits on long-context and decomposable tasks, including a documented accuracy improvement from 2.6% to 45.4% on a chain-of-thought benchmark. Practically, RLMs are best suited to tasks involving large or dense context, meaningful decomposition, and code-amenable subproblems, while offering limited advantage for tasks that fit comfortably within existing context windows or require minimal latency. As open-source implementations mature and industry adoption (e.g., Anthropic's workflows) accelerates, RLMs warrant continued empirical evaluation as a candidate default for long-horizon, high-volume-context production systems.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub