Give the Agent a Budget, Not a Token - Sachin Malhotra, Anthropic

Agent systems in production require a budget-based governance model with four primitives - asymmetric verbs, rate limits, trip wires, and the undo test - rather ...

By Sean Weldon

Abstract

Production deployment of autonomous agents reveals fundamental limitations in token-based access control systems, which provide only binary permissions that either constrain agents to ineffectiveness or expose infrastructure to catastrophic failures. This paper presents a budget-based governance framework comprising four primitives - asymmetric verbs, rate limits, trip wires, and the undo test - that enable graduated constraints across multiple dimensions of agent behavior. Drawing from a production incident where an agent deleted approximately 200 workloads in 90 seconds, this analysis demonstrates how budget-based governance parallels human organizational onboarding by replacing static yes/no permissions with dynamic budgets enforced through infrastructure-layer proxies. The framework separates intent shaping (text layer) from damage bounding (infrastructure layer), ensuring that prompt injection cannot circumvent safety constraints. This approach provides a practical path for deploying autonomous agents in production environments while maintaining operational safety through composable primitives that bound blast radius, enable recovery, and facilitate continuous improvement.

1. Introduction

The deployment of autonomous agents in production environments has exposed critical inadequacies in conventional access control mechanisms. While token-based access control has served traditional software systems adequately, it proves fundamentally insufficient when agents perform real work with potential for cascading failures. The binary nature of token permissions creates an untenable trade-off: overly restrictive permissions render agents unable to accomplish their intended tasks, while overly permissive access creates catastrophic operational risk.

A production incident illustrates this failure mode with stark clarity. An agent deleted approximately 200 workloads within 90 seconds when a pipeline stage evaluated to nothing, causing a filter to drop out and a selector to match everything rather than the intended subset. This incident impacted 20 engineers and demonstrates that the fundamental problem lies not in agent intelligence or intent - the model itself did not fail - but in the absence of graduated constraints that could bound the blast radius of a single logical error.

This paper examines a budget-based governance model that replaces boolean permissions with graduated constraints across four dimensions: action volume, execution velocity, autonomous recovery capability, and observability. The framework introduces four composable primitives that provide comprehensive governance without categorical denial of capabilities. This approach explicitly parallels human organizational onboarding, recognizing that effective governance requires escalation paths and structural limits rather than simple prohibition of entire verb categories.

2. Background and Related Work

2.1 Limitations of Token-Based Access Control

Traditional access control systems employ tokens as bearer credentials that grant specific scopes of access. In this paradigm, an agent either possesses a token with particular permissions or lacks access entirely. This model assumes that permissions can be correctly specified upfront and remain static throughout the agent's operational lifetime. However, production systems exhibit dynamic access requirements that violate this assumption.

The static nature of token scopes creates a maintenance burden where permissions must be continuously broadened as agent responsibilities expand. An agent initially denied certain operations will eventually require legitimate access to those same operations. This progressive broadening of permissions increases risk exposure over time. Furthermore, narrowing token scope does not provide a sustainable solution - the agent inevitably needs legitimate access to operations it was previously denied, forcing a choice between operational utility and safety.

2.2 The Human Onboarding Analogy

Organizations do not onboard junior engineers by denying them entire categories of operations indefinitely. Instead, they provide graduated access with escalation paths, structural limits, and monitoring. New engineers are not prevented from deploying code; rather, they deploy through staged environments, with code review requirements, and with the ability to roll back changes. This model recognizes that categorical denial is both impractical and counterproductive, while unbounded access without oversight creates unacceptable risk.

The budget-based governance framework applies this organizational wisdom to agent systems, replacing the question "Does the agent have this token?" with four graduated questions: How much can the agent do? How fast can it act? What can it undo autonomously? Who is monitoring its actions?

3. Core Analysis

3.1 The Four Primitives of Budget-Based Governance

The budget-based governance model comprises four primitives that compose to provide comprehensive constraints without categorical denial of capabilities. Each primitive addresses a distinct dimension of risk and enables different aspects of safe autonomous operation.

Asymmetric verbs distinguish operations by their failure characteristics rather than their semantic category. Operations that fail loudly - where failures are immediately visible to humans - can be delegated to agents with confidence that humans can correct visible errors. For example, an unskip_test operation or page_human action fails loudly; if the agent invokes these incorrectly, the error is immediately apparent. Conversely, operations that fail silently - where bugs can reach production undetected - must involve human oversight. A skip_test operation exemplifies silent failure; if the agent skips a test incorrectly, the error may only manifest when broken code reaches production.

Rate limits provide temporal bounds on blast radius by implementing a ceiling that refills over time. An agent receives a fixed budget of disruptive actions per time window that can be executed without approval. When the budget is exceeded, requests bounce with the current count, creating a hard ceiling on damage from a single loop iteration. Critically, rate limit sizes vary by context: an agent receives higher budgets for operations within its own namespace and lower budgets for shared resources. A bypass flag exists for legitimate overrides but refuses to function inside agent sessions, forcing humans to run commands directly when extraordinary circumstances require exceeding normal limits.

Trip wires enable post-hoc behavioral monitoring rather than pre-hoc allow lists. Allow lists represent static guesses made upfront about agent needs and do not improve over time. Trip wires record actions after the fact with actor identity stamps and improve continuously as operational data accumulates. A trip wire pages on-call after a limit is crossed, acting as a smoke detector rather than a lock on the door. In one production example, a trip wire detected an agent launching excessive investigation threads per hour; the fix required only a one-line addition to agent context about correlating failures before launching threads. Trip wires watch aggregate behavior rather than individual calls, enabling rapid improvement through minimal context adjustments.

The undo test determines authorization requirements based on two questions: Can the agent reverse the action autonomously? How severe is the impact if the agent makes an error? If the agent can roll back the change and the blast radius is acceptable, the system logs the action and permits autonomous execution. If either answer is negative, the operation requires a second key held by someone else, with an audit record. This differs from asymmetric verbs - verbs ask whether failure is noticed, while the undo test asks whether recovery is possible. In a feature flag example, an agent has full control over canary deployments (ramping 0-100%, toggling on/off) but cannot promote to production; it can only propose such changes for human approval.

3.2 Implementation Architecture: Text and Infrastructure Layers

The budget-based governance framework operates across two distinct layers with different enforcement characteristics. The text layer comprises prompts, context files, and markdown documentation that shape agent intent. This layer works approximately 80% of the time and guides the agent toward appropriate behavior through natural language instruction. However, the text layer provides no enforcement guarantees - it influences what the agent attempts but cannot prevent unauthorized actions.

The infrastructure layer enforces rules deterministically through a proxy that mediates all agent actions. The proxy sees each action, checks budget constraints, and returns HTTP 403 errors if limits are exceeded. Critically, the proxy does not know or care why the agent wants to act; it only stamps identity and enforces limits. This separation ensures that sophisticated prompt injection cannot circumvent safety constraints - clever prompting may shape intent at the text layer, but the infrastructure layer bounds damage regardless of what the agent believes it should do.

Each agent session has a dedicated proxy running alongside it. Every outbound call goes through this proxy after the context file is read. The proxy holds real credentials and stamps every call with an identity it already knows, not what the agent claims in request headers. This architecture ensures that identity comes from infrastructure rather than agent self-assertion, preventing budget reset attacks where an agent might change header values to circumvent rate limits.

3.3 Identity as Infrastructure Requirement

Proper identity management proves essential to budget enforcement. If an agent could set its own identity in request headers, it could reset budgets by changing the header value, transforming hard constraints into suggestions. The proxy-based architecture prevents this by stamping every call with an identity the proxy already knows from session initialization, independent of agent claims.

The proxy stamps identity as a label on Kubernetes jobs, and child jobs inherit the same identity. All downstream safeguards key on this stamp, enabling consistent enforcement across complex operation chains. Per-session identifiers enable differentiation between concurrent agent sessions, allowing operators to identify which specific session exhibits anomalous behavior when trip wires trigger. This granular identity tracking facilitates rapid diagnosis and targeted intervention without disrupting well-behaved concurrent sessions.

3.4 Primitive Composition in Practice

The four primitives compose to provide comprehensive governance without requiring all checks for every operation. Verbs ask what the agent is touching; rate limits ask how often; trip wires catch what happened; the undo test sizes all three. In the motivating incident where an agent deleted approximately 200 workloads in 90 seconds, a rate limit would have capped deletions at tens of workloads per hour. The undo test would have revealed that running jobs cannot be undeleted in other users' namespaces, requiring human approval for such operations.

This composition mirrors human onboarding checklists: what can they touch, how much rope do they get, who signs off, how do we know it's working. Not all checks apply to every write scenario - only relevant primitives constrain specific actions. This selective application maintains operational velocity while bounding risk in proportion to potential impact.

4. Technical Insights

4.1 Concrete Implementation Details

Production implementations of budget-based governance reveal specific technical patterns. A rate limit admission webhook caps deletions at a fixed number per hour per resource kind per namespace. Trip wires track metrics such as investigation threads launched per hour for given test job failures, paging on-call when baselines are exceeded. Feature flag services grant agents full control over canary dials (0-100% ramp, toggle on/off) while scoped keys prevent direct production promotion.

The proxy architecture requires careful session management. Context files are read first, establishing agent goals and constraints. All subsequent outbound calls route through the dedicated session proxy, which stamps identity and enforces budgets before forwarding requests. This sequencing ensures that agents understand their constraints before attempting actions while maintaining deterministic enforcement regardless of agent behavior.

4.2 Trade-offs and Limitations

The budget-based approach trades upfront specification precision for operational flexibility and continuous improvement. Allow lists provide certainty about permitted operations but require perfect foresight and create maintenance burden. Budget-based governance accepts that initial constraints may be imperfect but enables rapid adjustment through trip wire feedback and minimal context modifications.

The framework assumes that most agent errors are recoverable or bounded rather than catastrophic. For operations where errors are truly unrecoverable or have extreme blast radius, traditional human-in-the-loop approval remains appropriate. The undo test explicitly identifies such operations, requiring second keys and audit records. This hybrid approach applies graduated autonomy where appropriate while maintaining strict controls for high-stakes operations.

5. Discussion

The budget-based governance framework addresses a fundamental tension in production agent deployment: the need for operational autonomy conflicts with the requirement for safety guarantees. Token-based access control resolves this tension through categorical denial, accepting reduced utility to minimize risk. Budget-based governance resolves it through graduated constraints that bound blast radius while preserving operational capability.

This approach has broader implications for agent safety research. Much current work focuses on improving model capabilities or alignment, assuming that better models will make safer agents. The budget-based framework suggests that infrastructure-layer constraints provide complementary safety guarantees independent of model quality. Even perfectly aligned models can encounter logical errors or unexpected system states; infrastructure constraints ensure that such errors have bounded impact regardless of model behavior.

The parallel to human organizational onboarding suggests generalizable principles. Organizations have developed sophisticated approaches to managing human agents with imperfect information, uncertain intent, and potential for error. These organizational patterns - graduated access, escalation paths, monitoring, and recoverability - translate effectively to autonomous agents. This suggests that agent safety may benefit more from organizational design principles than from purely technical approaches focused on model behavior.

Future work should investigate the composition of budget primitives for specific operational contexts. Different environments may require different primitive weightings - development environments might emphasize velocity over constraints, while production systems require stricter limits. Understanding how to tune primitive parameters for different risk profiles remains an open question. Additionally, the relationship between trip wire sensitivity and operational noise requires further investigation to optimize signal-to-noise ratios in production monitoring.

6. Conclusion

This paper presents a budget-based governance framework for production agent systems that replaces token-based access control with four composable primitives: asymmetric verbs, rate limits, trip wires, and the undo test. The framework enables graduated constraints across multiple dimensions - action volume, execution velocity, recovery capability, and observability - while maintaining operational utility. Implementation through infrastructure-layer proxies ensures that safety constraints cannot be circumvented through prompt injection or agent manipulation.

The practical implications are significant for organizations deploying autonomous agents in production environments. Rather than choosing between constrained-but-safe and capable-but-risky agents, the budget-based approach enables agents that are both operationally effective and safely bounded. The framework's parallel to human onboarding provides intuitive mental models for operators and facilitates organizational adoption.

Organizations implementing autonomous agents should consider budget-based governance as an alternative to expanding token scopes. The primitives compose naturally and can be adopted incrementally, starting with rate limits for high-risk operations and expanding to full budget-based governance as operational experience accumulates. This measured adoption path provides immediate safety improvements while building toward comprehensive governance frameworks that enable truly autonomous production agents.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub