'Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads - Sitanshu Gupta, CoreWeave'

Coreweave builds a single, unified inference platform that supports multiple consumption models (serverless and dedicated) and diverse workload shapes, using...

By Sean Weldon

Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads

Abstract

This synthesis examines the architecture of a unified large language model (LLM) inference platform designed to serve heterogeneous consumption models and workload shapes without bespoke redesign per use case. Drawing on CoreWeave's platform design, the analysis details how serverless, provisioned throughput, and dedicated consumption tiers share a common control plane, routing layer, and multi-engine execution substrate. Central to the platform's economics is KV-cache aware routing, which prioritizes cache locality over naive load balancing, motivated by the observation that 80-90% of input sequence length in agentic workloads is repeated across requests, and that prefill computation is strongly compute-bound. Complementary levers - NVFP4 quantization, speculative decoding with customer-trained speculators, optional prefill-decode disaggregation, and cache offloading to high-bandwidth storage - translate architectural decisions into measurable price-performance gains. The findings suggest that vertical mobility across workload types is achievable through configurable infrastructure rather than workload-specific redesign, with implications for platform teams building multi-tenant inference systems at scale.

1. Introduction

Inference has become the dominant recurring cost in production deployment of large language models, surpassing the one-time cost of training for many organizations. Unlike training, which is a relatively homogeneous batch computation, inference traffic is heterogeneous by nature: it includes latency-critical streaming interactions, throughput-oriented batch jobs, and long-context agentic reasoning tasks, each carrying distinct service-level agreements (SLAs) and cost structures. A common industry response has been to construct vertically specialized infrastructure stacks per workload class. This fragments engineering investment and strands capacity that could otherwise be shared across workload types.

The central thesis examined in this synthesis is that a single inference platform can serve multiple consumption models - the commercial and operational terms under which customers access compute - and multiple workload shapes - the statistical and latency profile of request traffic - provided that two structural conditions are met. First, the routing layer must be aware of key-value (KV) cache state rather than reactive only to instantaneous load. Second, a common set of performance levers must be exposed as configurable options rather than hard-coded into workload-specific stacks. The stated terminal objective is delivery of price-performance benefit to the customer, treated as the unifying metric across all consumption models and workload types.

This synthesis proceeds in four parts. Section 2 establishes background on prefill/decode decomposition and the techniques the platform integrates. Section 3 analyzes consumption models, workload shapes, and the KV cache optimization strategy that ties them together. Section 4 consolidates technical insights and trade-offs. Section 5 discusses broader implications for inference platform design.

2. Background and Related Work

Transformer inference decomposes into two phases with divergent hardware characteristics. Prefill processes the full input prompt in parallel across all tokens and is compute-bound; decode generates output tokens autoregressively, one at a time, and is memory-bandwidth-bound. The KV cache - stored attention keys and values from previously processed tokens - is the artifact linking these phases and enabling reuse of computation across turns of a conversation or across requests sharing a common prefix.

Several established techniques inform the platform's design. Prefill-decode disaggregation physically separates the two phases onto distinct resource pools so that each can be scaled and parallelized according to its own bottleneck characteristics. KV cache offloading, exemplified by techniques such as LMCache and MoonCake, relocates cache blocks from high-bandwidth memory (HBM) to cheaper high-bandwidth storage tiers for later reload, avoiding full eviction and recomputation. Speculative decoding uses a smaller draft model to propose token sequences that are verified in parallel by the target model, raising output throughput as a function of the model's acceptance length. Low-precision quantization, specifically NVFP4, is a four-bit floating-point format targeting recent NVIDIA accelerator generations. The platform additionally integrates three inference engines - vLLM, SGLang, and TensorRT-LLM - operating across multiple GPU hardware generations, reflecting the premise that engine suitability is workload- and customer-dependent rather than universal.

3. Core Analysis

3.1 Consumption Models

The platform exposes three purchasing modes over shared infrastructure. Serverless billing is per token, requiring no customer management of hardware, clusters, or orchestration, with access via API or UI and a broad model catalog. Dedicated consumption gives customers explicit knowledge of and control over their hardware, with self-managed model deployment and performance tuning, billed per GPU per hour, and isolated via private gateways to prevent noisy-neighbor interference. Provisioned throughput occupies a middle position: a serverless variant in which capacity is explicitly carved out for a customer to avoid noisy-neighbor effects, while retaining per-token billing. This tiering allows customers to select a point on the spectrum between operational simplicity and infrastructural control without switching platforms.

3.2 Workload Shape Taxonomy

Four workload shapes are identified: agentic, chat, real-time voice/video, and batch. Agentic and chat workloads share high input sequence lengths paired with low output sequence lengths, but agentic workloads impose substantially lower latency tolerances in multi-turn contexts. Voice and video workloads are streaming and uniformly latency-sensitive. Batch workloads carry loose SLAs, sometimes spanning hours, and are scheduled opportunistically to consume spare capacity - for example, dedicated capacity is reused for batch workflows during off-peak hours via API-driven scale-up and scale-down scheduling. Notably, the proportion of agentic workload relative to other shapes is continuously increasing, a trend with direct implications for cache reuse strategy, discussed in Section 3.3. The scheduling challenge is described metaphorically as playing "tetris" across the time dimension to maximize utilization of underlying infrastructure given these divergent shapes.

3.3 KV Cache-Aware Routing and Optimization

The request flow begins at a control plane responsible for authorization, rate limiting, usage tracking, and observability necessary for SLA compliance. From there, a router performs KV-cache-aware routing, targeting explicit deployments for provisioned-throughput or multi-tenant customers. Across heterogeneous capacity spanning zones and regions, the router applies a defined priority order: KV cache locality first, followed by least-loaded fallback when no cache-local option exists.

This priority ordering is motivated by a specific empirical observation: 80-90% of input sequence length in agentic workloads is often repeated across requests, making cache hits highly valuable. Because prefill is compute-bound and expensive, avoiding its recomputation through cache hits yields substantial cost savings - reflected directly in pricing, where uncached input tokens are priced higher than cached input tokens. This reuse principle extends to multi-turn chat and agentic conversations, where KV cache offloading to high-bandwidth storage (via LMCache and MoonCake) allows fast reload of cache state for delayed follow-up requests rather than full eviction and recomputation. Prefill-decode disaggregation remains optional within this framework, applied only where use-case economics justify the added architectural complexity.

4. Technical Insights

Several implementation-level findings merit explicit attention. First, prefill's compute-bound nature makes it the most expensive phase of inference in aggregate cost terms, which structurally justifies asymmetric pricing between cached and uncached input tokens. Second, the router's two-tier priority - cache locality then least-loaded fallback - represents a deliberate trade-off: it accepts potential short-term load imbalance in exchange for avoiding compute-bound recomputation, a trade-off that only makes sense given the high repeat rate of agentic input sequences.

Third, speculative decoding is implemented as a customer-specific pipeline: customers supply their own datasets, speculators are trained asynchronously, and the resulting models are deployed into production to improve acceptance length and output throughput. This represents a customization layer distinct from a one-size-fits-all speculator. Fourth, NVFP4 quantization and speculative decoding are identified as the two most significant performance levers currently in use, suggesting that precision reduction and decoding parallelism are the primary near-term axes for throughput gains, ahead of architectural changes such as disaggregation.

Fifth, dedicated customers retain configurability over both inference engine choice (vLLM, SGLang, TensorRT-LLM) and whether to enable prefill-decode disaggregation, indicating that the platform treats these as tunable parameters rather than fixed defaults. Finally, benchmark evidence - top rankings on Kimi 2.6/2.7 via artificial analysis, and speed competitive with Fireworks on GLM via open-router actual user traffic - illustrates a limitation worth noting: synthetic benchmark results (artificial analysis) and real-traffic benchmark results (open router) may diverge, and both should be considered when evaluating platform performance claims.

5. Discussion

The broader implication of this architecture is that workload heterogeneity need not necessitate infrastructural fragmentation. By treating consumption model and workload shape as orthogonal, configurable dimensions layered atop a shared control plane and routing substrate, the platform avoids the engineering overhead of maintaining parallel stacks per use case. This is consistent with an industry trend toward multi-tenant, multi-engine inference platforms that abstract hardware heterogeneity behind a common routing and scheduling layer.

The increasing share of agentic workloads relative to chat, voice/video, and batch traffic is a significant trend with compounding effects: as agentic traffic grows, the value of KV-cache-aware routing and offloading increases correspondingly, since these workloads exhibit the highest degree of repeated input structure. This suggests that investment in cache locality infrastructure is likely to yield increasing returns over time rather than diminishing ones.

Open questions remain regarding the precise thresholds at which prefill-decode disaggregation becomes cost-effective, and how cache offloading tiers are sized relative to the temporal distribution of multi-turn conversation gaps. The source material does not quantify the latency or cost trade-off of HBM-to-storage offloading in detail, representing an area for further empirical investigation.

6. Conclusion

This synthesis has outlined a platform architecture in which consumption model flexibility (serverless, provisioned throughput, dedicated) and workload shape diversity (agentic, chat, voice/video, batch) are reconciled through KV-cache-aware routing and a shared set of performance levers - quantization, speculative decoding, and optional disaggregation - rather than through workload-specific redesign.

The practical takeaway for platform engineers is that cache locality should be treated as a first-class routing signal, prioritized ahead of load balancing, particularly as agentic workloads with high prefix repetition come to dominate traffic mix. Future work should focus on quantifying the cost-latency trade-offs of cache offloading tiers and establishing clearer decision criteria for when prefill-decode disaggregation justifies its added complexity.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub