Training Your Own Agent Models With RL Is Now Practical with Jason Mancuso, Modal

Owning and fine-tuning your own agent models via RL - rather than treating foundation models as swappable backends - is now a practically achievable and incr

Training Your Own Agent Models With RL Is Now Practical

Abstract

This synthesis examines the argument that treating foundation models as interchangeable backends is strategically untenable for organizations deploying agents at scale, and that reinforcement learning (RL) on proprietary agent harnesses is now practically achievable. Three contributions are consolidated. First, a rationale is presented for preferring RL over supervised fine-tuning (SFT) for agentic workloads, grounded in the branching structure of multi-turn agent sessions. Second, a stabilization recipe is described - Group Relative Policy Optimization (GRPO) augmented with masked importance sampling, score centering, and top-P sampling with mask replay - validated by ablation on a Qwen 3.6 35B mixture-of-experts (MoE) model. Third, infrastructure results are reported showing that RL weight deltas are 100-1,000x sparser than dense weights, permitting approximately 1,000x communication compression, decoupling of inference from the training cluster, and heterogeneous multi-cloud rollouts that reduced rollout cost by 15%.

1. Introduction

The prevailing practice among AI engineers is to consume frontier models as static backends, substituting one for another as new releases arrive. The central thesis advanced here is that this constitutes a losing strategy for any organization competing simultaneously on agent economics (the cost and latency of serving agentic workloads) and agent experience (the reliability and task-fit of agent behavior). Three forces drive this conclusion: customers demanding reliability service-level objectives (SLOs) for agent deployments must ultimately own their inference stack, since open-source models deployed without adaptation frequently degrade in quality under production load; owning all layers of inference permits teams to advance along the latency, quality, and cost curves concurrently rather than trading one against another; and proprietary datasets absent from pretraining corpora define tasks on which off-the-shelf models such as Claude or GPT fail outright, leaving fine-tuning as the only viable path.

A distinct but related trend is the vertical integration of companies across the agent stack to construct differentiated value, a pattern most visible today in coding agents but anticipated to extend into other industries as agentic competition intensifies. This pattern reframes model ownership as a competitive requirement rather than a research luxury.

The analysis proceeds from this strategic framing to the methodological question of which training paradigm suits agents, then to the mechanics of RL instability, the proposed correction recipe, and the infrastructure required to execute it economically at multi-cloud scale.

2. Background and Related Work

The recipe described is anchored in GRPO, introduced by DeepSeek in the development of DeepSeek R1 and now treated as a cornerstone of RL for large language models (LLMs). GRPO provides the baseline policy-optimization objective; the contributions discussed are corrections layered atop it rather than replacements for it.

Implementation rests on two open components: Miles, an RL framework, and SGLang, an inference engine. A structural observation underpins the feasibility claim: teams already operating agents in production typically possess the constituent parts of an RL environment without recognizing them as such. An agent harness supplies the environment dynamics, while existing evaluation suites and A/B tests supply the reward signal. Training tasks of sufficient quality can be synthesized by mining production traces for failure cases, reducing dependence on manual curation.

3. Core Analysis

3.1 Why RL Rather Than SFT for Agentic Workloads

SFT trains models on recorded production traces, but agent sessions form branching trees whose shape depends on the number and content of intervening turns. A model trained on one observed branch may, in deployment, encounter an entirely different branch never represented in its training data, making SFT brittle for multi-turn agents. RL addresses this by allowing the model to explore these branches directly during training, which is the mechanism typically invoked when practitioners describe RL as "generalizing better" than SFT. This distinction motivates the broader claim that organizations with production agent deployments already possess the raw ingredients - harness plus evals - needed to construct an RL environment.

3.2 Mechanics and Sources of RL Instability

The RL training loop separates two roles across hardware: inference GPUs execute the sampler, producing rollouts, which feed a trainer that performs gradient updates; updated weights are then broadcast back to the samplers. Running this loop sequentially - rollout, then train, then update - leaves compute idle during each phase transition. Asynchronous RL instead runs rollout and training concurrently, raising GPU utilization but introducing staleness, in which rollouts are generated from a model version that has since been superseded by the trainer. A second, independent source of instability is systems-level mismatch: subtle discrepancies between what the trainer and the sampler each believe they are training on, arising from differences across layers of the serving and training stack (e.g., numerical precision, kernel implementations).

3.3 The Stabilization Recipe

The proposed recipe begins from GRPO and layers three corrections on top. Masked importance sampling corrects token-wise mismatch arising from both staleness and systems-level discrepancies. Score centering removes accumulated drift that persists across training steps even after token-wise correction has been applied. Top-P sampling with mask replay addresses a more severe failure mode: when the trainer samples a token that the sampler's top-P cutoff (commonly P = 0.95-0.98) had excluded, producing catastrophic mismatch at that token. Notably, score centering becomes an exact correction, rather than an approximation, once combined with top-P sampling - an interaction effect rather than two independent fixes.

This recipe was ablated on a Qwen 3.6 35B MoE model trained on code environments and evaluated on Stable Bench Pro and its Hard-51 subset. The full recipe produced markedly more stable training than any subset of its constituent techniques, indicating that the components are complementary rather than redundant corrections for the same failure mode.

3.4 Infrastructure: Sparse Deltas and Heterogeneous Scheduling

Modal aggregates GPU capacity across roughly 15-20 providers, spanning major hyperscalers and multiple Neo Clouds. This breadth improves reliability and scheduling flexibility but fragments the available compute pool, making large contiguous training clusters (16-32+ nodes) difficult to schedule, whereas clusters of 4-8 nodes schedule readily. Because agent RL typically allocates roughly two-thirds of cluster GPUs to inference rather than training, an 8-node cluster yields only 2-3 nodes of effective training capacity - compounding the scheduling constraint.

The key enabling observation is that model weight update deltas during RL are 100x to 1,000x sparser than the dense weights themselves, even though the full weights at each step remain large and dense. This sparsity permits off-the-shelf compression that reduces trainer-to-rollout-engine communication by approximately 1,000x, which in turn allows inference GPUs to be moved entirely outside the training cluster and distributed across the full multi-provider fleet. In the Qwen 3.6 run, the training cluster was located in GCP while inference GPUs were spread across the broader hyperscaler and Neo Cloud fleet.

4. Technical Insights

Several findings carry direct implementation relevance. Rollouts were run simultaneously across heterogeneous GPU types - B200, H100, A100, and H200 - which increased trainer/sampler mismatch by 4-5x relative to homogeneous hardware due to kernel- and math-level differences between accelerators. Despite this amplified mismatch, the correction recipe held training reward and gradient norm as stable as a homogeneous B200-only configuration, indicating that the recipe's corrections are robust to hardware-induced numerical divergence, not solely to staleness.

This robustness had a direct economic payoff: routing rollouts to cheaper, lower-demand GPU types reduced total rollout cost by 15% relative to using only high-demand B200/B300 hardware, and this reduction was achieved using basic, off-the-shelf inference settings per GPU type - without per-GPU tuning or quantization optimization. A further test split H200 rollouts across two pools, one running BF16 and one running the more aggressive FP8 quantization; the correction recipe absorbed the resulting mismatch without separate handling, suggesting the recipe generalizes across precision regimes as well as hardware families. A limitation implicit in these results is that they were obtained on one model family and one domain (code environments); generalization to other model scales or task domains is not established within the material presented.

5. Discussion

The combination of a stabilized correction recipe and sparse-delta infrastructure shifts RL training of agent models from a research-lab capability to an operationally accessible one. The infrastructure contribution is arguably as consequential as the algorithmic one: decoupling inference from the training cluster via sparse delta compression directly addresses the scheduling bottleneck that fragmented, multi-provider GPU fleets impose, converting a hardware-allocation problem into a bandwidth problem that compression resolves cheaply.

This has implications for the broader "vertical integration" thesis introduced at the outset. If the primary obstacles to owning agent RL were scientific instability and infrastructure scarcity, and both are now substantially mitigated, then the strategic argument for model ownership gains practical force: the barrier shifts from feasibility to organizational investment and harness maturity. A remaining open question concerns how the recipe performs on tasks and model families outside code generation, where reward signals may be noisier or less verifiable than in code environments with executable tests.

6. Conclusion

The central contribution of this work is demonstrating that stable, cost-effective RL training of agent models is now achievable through the combination of an extended GRPO recipe - masked importance sampling, score centering, and top-P sampling with mask replay - and infrastructure exploiting the extreme sparsity of RL weight deltas. Practically, organizations with existing agent deployments already possess the harness and evaluation infrastructure needed to begin RL training, and heterogeneous multi-cloud GPU scheduling, previously a liability, can be converted into a cost advantage. The next step for practitioners is evaluating whether their existing production telemetry can be mined for RL training tasks, and whether their inference stack can absorb the kind of heterogeneous hardware mixing shown here to reduce cost without sacrificing stability.

Sources

• Training Your Own Agent Models With RL Is Now Practical - Jason Mancuso, Modal - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub