Your Agent Doesn't Need an LLM for Every Decision with Raymond Weitekamp, OpenProse
Reasoning-heavy tasks can often be decomposed into many small, fast, cheap "decisions" (system one/decision models like Jev) composed recursively, and when p
By Sean WeldonYour Agent Doesn't Need an LLM for Every Decision: Recursive Decision Models as an Alternative to Pure Generation
Abstract
This synthesis examines Recursive Decision Models (RDM), an architecture that treats reasoning not as a monolithic generative process but as the recursive composition of many small, fast, inexpensive classification calls. RDM combines Recursive Language Models (RLM), developed at MIT, with so-called system one or decision models, exemplified by Type Safe's Jev. The architecture rests on three rules: context remains external to the model, decisions may recursively invoke sub-decisions to a bounded depth, and the system must function with zero generative models while remaining composable with Large Language Models (LLMs) and agents. Demonstrations span game-playing harnesses, hierarchical taxonomy search over 12,000 categories, multimodal image localization, and real-time audio interaction at approximately 45 ms latency. The architecture underpins OSO (Operating System One), a decision-first computer-use agent oriented toward touch-free human-computer interaction, suggesting that cost and latency gains are available wherever an action space can be enumerated.
1. Introduction
The dominant paradigm in applied artificial intelligence equates capability with generation: tasks are framed as token-production problems, and improvements are sought through larger models, longer contexts, and extended chains of thought. Every action an agent takes, in this framing, routes through an LLM call. The architecture examined here, presented by Raymond Weitekamp of OpenProse, inverts that assumption by asking a direct question:
"What if reasoning was actually many small decisions?"
The terminology derives from Type Safe's launch of Jev, which invoked Daniel Kahneman's distinction between system 1 (fast, intuitive, automatic cognition) and system 2 (slow, deliberate, effortful cognition). Naming remains contested within the field, with competing framings including decision models, zero-shot classifiers, and the proprietary system one branding. A further refinement, attributed to Type Safe's Ali, holds that Jev produces judgments rather than decisions: the surrounding code, not the model, makes the final decision. This distinction matters architecturally because it locates control flow in deterministic software rather than in a probabilistic model.
This reframing matters because most agentic systems today route every branch point - tool selection, state checks, routing logic - through a generative model, incurring latency and cost that may be unnecessary when the action space is enumerable. The analysis that follows establishes the theoretical lineage of RDM (Section 2), examines its architecture and demonstrated instantiations (Section 3), extracts implementation-level findings (Section 4), discusses broader implications (Section 5), and concludes with practical takeaways (Section 6).
2. Background and Related Work
Three intellectual strands converge in RDM. First, Recursive Language Models (RLM), developed by Alex Sang and Omar at MIT, establish the structural template: a root model decomposes a problem and recursively invokes sub-models, bottoming out in a leaf-level LLM call. The RLM group also produced research on speculative tool-calling, which informs the concurrency pattern discussed in Section 3.3.
Second, decision models such as Jev supply a qualitatively different primitive than generative LLMs. A decision model cannot emit free text; it can only select among supplied options. Generative models exhibit divergence - even given infinite tokens, the output space remains open-ended - whereas decision models exhibit convergence, narrowing deterministically to a single selection. Weitekamp frames the two as complementary, "yin and yang," rather than competing approaches.
Third, an older tradition of small-model interaction design informs the applied work discussed in Section 3.4: the Princeton Laptop Orchestra and Rebecca Fiebrink's Wekinator, a real-time machine learning framework that mapped laptop accelerometer, trackpad, and keyboard input to musical control using k-nearest neighbors, single-layer neural networks, and PCA-like dimensionality reduction.
3. Core Analysis
3.1 The Three Rules of RDM
RDM is defined by three constraints. First, context stays outside the model: deterministic code slices and supplies context rather than loading it wholesale into a context window. Second, a decision may call another decision, recursively, up to a maximum depth, at which point a single system-one decision call is made - a direct structural analogy to RLM's leaf-level LLM call. Third, the architecture must work with zero generative models at all, while still being able to compose with LLMs and agents when needed. This third rule is what distinguishes RDM from a conventional agent framework: generation is optional, not foundational.
3.2 Pure-Decision Demonstrations
Several demonstrations establish that nontrivial behavior can emerge from decision calls alone, with no generative component. "Jev Plays Doom" combines two coupled decision harnesses - one governing first-person shooting actions, one governing bird's-eye map exploration - to play the game using pure decision calls. A Factorio demonstration uses a matrix of decision calls to identify broken production lines and inserters and navigate to repair them, with the presenter acknowledging imperfect pathfinding as a current limitation.
Type Safe's hierarchical search cookbook applies the same principle to information retrieval: beam search over a 12,000-category Shopify product taxonomy is executed via recursive hierarchical Jev calls rather than a conventional beam search, exploring only the next sublayer of categories at each step rather than loading the full taxonomy into context. This avoids context-window overload entirely. Related multimodal examples - recursive zoom-in feedback loops on images, and a golden-spiral-style recursive halving of images to localize objects - extend the same convergence pattern to visual search.
The clear limitation of this "decisions all the way down" approach is that all possible actions must be pre-enumerated programmatically. It is well suited to games, browsers, and computer-use contexts where the action space is finite and enumerable, and poorly suited to open-ended tasks requiring novel output.
3.3 Composing Decisions with Generative Models
Pure decision trees are symmetric and pre-defined in shape; introducing generative elements produces organic, emergent call-graph shapes instead. A smart-home demonstration (adapted from Type Safe's cookbook) illustrates the pattern: decisions handle enumerable states, such as an "off to bed" routine, while an LLM is invoked generatively only when the task cannot be resolved by decision alone - for example, "let's watch a film."
Two architectural constructs support this composition. The map is a shared interface of pointers (e.g., "TV on" bound to a boolean-flip function) that lets the decision model select among predefined options while the LLM can add, edit, or modify map entries based on observed user behavior; the map can be multi-dimensional, representing embeddings in the sense proposed by Kieran Clausen (Every) for using Jev to construct low-dimensional, semantically meaningful embeddings. The arbiter is a gating mechanism allowing many fast, cheap decision models to run speculatively in parallel while preventing write collisions on shared state - analogous to CPU read/write locking, and inspired by speculative tool-calling research from MIT's RLM group.
3.4 Real-Time Interaction and OSO
The applied motivation for this work is explicitly physical: the presenter describes repetitive strain injury from conventional keyboard, mouse, and touchscreen interaction as the impetus for touch-free interface design. The "Earshot" demonstration, a precursor to OSO, layers a recursive decision model that transcribes audio and triggers actions on top of a very small real-time model (via "Chai"/Chuck AI) operating on 40-100 ms audio slices against a supervised teacher model. An arbiter layer learns over time when to trust the small fast model over the teacher, reducing reaction latency from an initial large lag to approximately 45 ms in a live demonstration set to Philip Glass's "Einstein on the Beach." A related example, Matt Mastracci's "Hunch," uses a WebGPU-based doodle classifier built on embedding Gemma 2 to achieve roughly 150 ms real-time classification, reinforcing that small embedding/classifier models can support responsive interaction without generative latency.
OSO (oso.computer) is the culminating system: a computer-use agent built on RDM that begins with voice input, farms tasks to coding agents (Claude Code, Codex) or RLM agents when generation is required, but is "decision first" by default. It continuously learns user preferences and uses the arbiter to delegate to the fastest trustworthy small model over time. The presenter argues that the industry's shift toward terminal-based coding agents reflects dissatisfaction with existing visual and touch interfaces, not any inherent advantage of the terminal itself.
4. Technical Insights
Several implementation-level findings are notable. The first RDM implementation was built as a DSPy module, chosen partly because the Compound AI/DSPy team quickly added support for System One API calls, and because DSPy's structured approach helped validate RDM module design; a standalone Rust package named RDM was subsequently built for fast, portable deployment. The hierarchical search example demonstrates a concrete cost-avoidance mechanism: context-window overload is avoided by exposing only the next sublayer of a taxonomy at each recursive step rather than the full 12,000-category space. The Earshot pipeline shows that online supervised learning combined with a trust-arbitration layer can compress reaction latency by roughly an order of magnitude, from an unspecified large initial lag to approximately 45 ms. The arbiter pattern generalizes a concurrency-control concept - read/write locking - to probabilistic decision models, enabling parallel speculative execution without corrupting shared state. The principal trade-off throughout is explicit: decision-only composition requires the action space to be enumerable in advance, which bounds its applicability to domains such as games, browser automation, and computer use rather than open-ended generation tasks.
5. Discussion
The broader implication of RDM is that the choice between "decision model" and "generative model" need not be treated as a binary architectural commitment at the system level. Instead, the two can be composed recursively, with generation reserved for the subset of calls that genuinely require open-ended output and decisions handling the remainder at substantially lower latency and cost. This reframes a common agent-design question - "which model should power this agent?" - into a decomposition problem: which parts of this task space are enumerable, and which require divergence.
An open question concerns how far the enumeration constraint can be relaxed. The presenter's own Factorio demonstration acknowledges imperfect pathfinding, suggesting that spatial or continuous-action domains remain harder to express as pure decision trees than discrete, menu-like domains such as smart-home control or taxonomy search. Similarly, the arbiter's learned trust mechanism, while shown to reduce latency substantially in the Earshot demo, has not been evaluated for failure modes where the fast model is trusted inappropriately.
RDM also intersects with a visible industry trend: the rise of terminal-based coding agents as a reaction against unsatisfying visual interfaces. Framing OSO as a decision-first alternative to continuous LLM-mediated interaction positions RDM not merely as a cost-optimization technique but as a candidate interface paradigm, particularly for touch-free or latency-sensitive interaction.
6. Conclusion
RDM contributes a disciplined way to reserve generative model calls for the subset of a task that actually requires divergence, while handling the enumerable remainder through fast, cheap, recursive decision calls. The three governing rules - external context, bounded recursive decision depth, and functionality without any generative component - provide a concrete design discipline rather than a loose heuristic. Demonstrated applications, from game-playing harnesses to taxonomy search to sub-50 ms audio interaction, indicate that the approach is viable wherever the action space can be enumerated in software. Practically, teams building agentic systems may benefit from auditing which of their LLM calls are truly generative versus selections among known options, and migrating the latter to decision models composed via patterns such as the map and the arbiter.
Sources
- Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse - Original Creator (YouTube)
- Analysis and summary by Sean Weldon using AI-assisted research tools
About the Author
Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.
Sean Weldon builds apps and custom software at 22Atlas.