Tokens Should Have Jobs - Katelyn Lesse & Angela Jiang, Anthropic

Tokens are not fungible; assigning tokens distinct 'jobs' (advising, grading, dreaming) beyond just executing produces better outcomes than simply increasing...

By Sean Weldon

Tokens Should Have Jobs: A Synthesis of Anthropic's Multi-Strategy Agentic Framework

Abstract

Contemporary agentic systems built on large language models implicitly treat inference tokens as a fungible resource, using total budget as the primary lever for improving task outcomes. This synthesis examines an alternative thesis presented by Anthropic researchers Katelyn Lesse and Angela Jiang: that tokens should have jobs, with portions of a fixed compute budget allocated to distinct functional roles - advising, grading, and dreaming - beyond simple task execution. Evaluated on a benchmark of financial analysis tasks, these composite strategies demonstrate that outcome quality is governed as much by token allocation architecture as by token quantity. At a fixed budget of 600,000 tokens, execution alone achieved 76% accuracy, while advising and grading strategies reached approximately 90%. Under stricter pass/fail scoring requiring 100% accuracy, execution's true total token cost rose to roughly 1.8 million tokens per perfect answer, exceeding that of advising and grading strategies. The findings suggest concrete architectural principles for reliability-constrained agentic deployments.

1. Introduction

The dominant paradigm for improving large language model agent performance is budget scaling: granting more tool calls, longer reasoning chains, additional retries. This approach rests on an implicit assumption that inference tokens are interchangeable units of capability, such that expenditure and performance are monotonically related. Anthropic's research challenges this assumption directly, asking: "Are all these tokens actually fungible?"

This synthesis examines the proposed alternative - that tokens should be deliberately assigned differentiated jobs within an agentic system. An executor performs the primary task. An adviser intervenes mid-task to validate correctness of next steps. A grader evaluates completed output against a predefined rubric, prompting iteration on failure. A dreamer inspects transcripts across rounds and distills learnings into persistent memory for future use. The combination of an executor with one or more of these auxiliary roles constitutes a strategy.

The central thesis is that, under a fixed compute constraint, strategic allocation of tokens across these roles produces materially better outcomes than simply increasing the token budget available to a single undifferentiated executor. This analysis covers the design of the advising, grading, and dreaming strategies; the experimental methodology used to compare them against pure execution on a financial analysis benchmark; the resulting accuracy and token-efficiency metrics under both partial-credit and pass/fail scoring regimes; and the implementation architecture that enables these strategies to be composed in practice.

2. Background and Related Work

Standard single-agent harnesses provide a model with tools, a task specification, and a termination condition, with performance improvements pursued primarily through expanding the agent's operational budget. This design treats the token stream as homogeneous - tokens spent on planning, tool invocation, or self-correction are assumed to be drawn from the same undifferentiated pool and to contribute comparably to task success.

Individually, the constituent strategies echo established patterns: advising resembles supervisory or critic-in-the-loop designs; grading resembles rubric-based LLM-as-judge evaluation with iterative refinement; dreaming resembles memory consolidation across task episodes. The distinguishing contribution of this work is treating these roles as competing allocations of a single fixed budget, measured under a common accounting framework, rather than as unconstrained additions layered onto an already-uncapped execution loop. The empirical substrate for this comparison is a benchmark of financial analysis tasks modeled on expert human analyst workflows - a domain selected specifically because it is reliability-constrained rather than tolerant of partial credit.

3. Core Analysis

3.1 Strategy Definitions and Mechanisms

Three auxiliary roles are examined alongside the baseline executor. In the advising strategy, the executor calls out to an adviser during task execution to validate the correctness of upcoming steps - illustrated by a sales agent that flags overdue follow-ups or stalling deals before proceeding. In the grading strategy, the executor's output is checked against a predefined rubric by a grader; failing outputs trigger further iteration by the executor until rubric criteria are satisfied, as in a customer service agent processing refund requests against explicit approval criteria. In the dreaming strategy, a dreamer reviews the executor's transcripts and findings after a task round, writing distilled learnings to memory that is retrieved by the executor in subsequent rounds - demonstrated through a recruiting agent that improves across successive candidate-review cycles using accumulated feedback.

3.2 Oneshot and Fixed-Budget Experiments

Initial experiments contrasted strategies without controlling for token expenditure. In a oneshot setting, pure execution achieved only 15% accuracy while consuming 39,000 tokens; dreaming achieved higher accuracy but consumed approximately 600,000 tokens - an order-of-magnitude difference that confounds any direct comparison of strategy quality.

To isolate the effect of allocation architecture from raw budget size, a second experiment fixed the token budget at 600,000 for all strategies. Under this constraint, execution rose to 76% accuracy, while advising and grading strategies reached approximately 90%. Because all strategies operated under an identical budget, the residual performance gap - execution at 76% versus advising at 89% - constitutes what the researchers term alpha: value attributable to allocation architecture rather than token quantity.

3.3 Pass/Fail Scoring and True Total Token Cost

Partial-credit accuracy metrics, while informative, misrepresent utility in domains where any deviation from a correct answer renders output unusable. As stated in the source material, "In this domain for this task if you're not 100% accurate it's actually not useful." Re-scoring the fixed-budget experiments as binary pass/fail against a 100%-accuracy threshold revealed that pure execution achieved a perfect-pass rate of only about 42%, while more complex strategies reached up to 75%.

This reframing motivates a true total token cost metric, which accounts for the expected number of runs required to obtain a single perfect answer. Because execution passes only 42% of the time, achieving a perfect answer requires an expected 3 runs at 600,000 tokens each, yielding a true total cost of approximately 1.8 million tokens - substantially exceeding the nominal 600,000-token budget. Advising and grading strategies, with higher first-pass success rates, were shown to be more token-efficient than execution or dreaming once this run-multiplication effect is incorporated.

4. Technical Insights

Several implementation-relevant findings emerge from this analysis. First, token budget and token allocation are separable variables, and the latter can produce gains - an estimated 13-percentage-point accuracy improvement at fixed budget in this benchmark - that budget expansion alone does not guarantee. Second, evaluation methodology materially affects strategy ranking: partial-credit accuracy favors different strategies than pass/fail-at-100% scoring, and the choice of metric should match the reliability requirements of the deployment domain. Third, true total token cost, rather than nominal per-run budget, is the appropriate unit for cost-efficiency comparisons in reliability-constrained settings, since low first-pass success rates multiply effective cost through required retries.

On the architectural side, the described implementation stack layers a meta-harness for multi-agent orchestration atop a base agent harness (Anthropic's cloud managed agents), with primitives designed to make composition of executor, adviser, grader, and dreamer roles straightforward. A noted limitation is that strategy selection currently appears to be a manual design choice rather than an automated one; the stated goal is for the platform to dynamically construct appropriate strategies over time, which remains an open engineering target rather than a demonstrated capability.

5. Discussion

These findings carry implications beyond the specific financial analysis benchmark. They suggest that agentic system design should treat token allocation topology as a first-class design variable, alongside model selection and prompt engineering, particularly in domains where reliability requirements are strict. The divergence between partial-credit and pass/fail rankings underscores a broader methodological point: benchmarks that report only mean accuracy may obscure the practical unreliability of a strategy in production settings where partial correctness has no value.

A remaining gap is generalizability: the benchmark is domain-specific (financial analysis), and it is not established whether the same magnitude of alpha or the same ranking of strategies (advise/grade over execute/dream) holds across qualitatively different task types, such as creative generation or long-horizon planning. Additionally, the cost model assumes independent, identically distributed run outcomes when estimating expected retries, which may not hold if failures are correlated with specific task subclasses. Future work could examine automated strategy selection or dynamic strategy composition conditioned on task characteristics, an ambition the researchers explicitly flag as a goal for the platform layer.

6. Conclusion

This synthesis has examined the thesis that tokens should be assigned differentiated jobs within agentic systems rather than treated as a fungible pool governed only by budget size. Evidence from a financial analysis benchmark demonstrates that at fixed token budgets, advising and grading strategies substantially outperform pure execution, and that under realistic pass/fail scoring, these strategies are also more token-efficient once the cost of repeated runs is accounted for.

The practical takeaway is that engineers building reliability-constrained agentic systems should evaluate whether restructuring token allocation across advisory, grading, or memory-consolidation roles yields better return than simply expanding execution budget, and should measure success using cost and accuracy metrics appropriate to the domain's tolerance for partial correctness.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub