Operating Distributed Inference Systems at Scale - Nishant Gupta & Naman Ahuja, Meta

AI inference at hyperscale is fundamentally an orchestration and control-plane problem, not just a model or kernel optimization problem, and reliability, cos...

By Sean Weldon

Operating Distributed Inference Systems at Scale: An Orchestration Perspective

Abstract

This synthesis examines the claim that large-scale artificial intelligence (AI) inference is fundamentally an orchestration and control-plane problem rather than a model or kernel optimization problem. Drawing on operational experience at hyperscale, the analysis contrasts classical microservice serving architectures with Large Language Model (LLM) inference systems across dimensions of request shape, batching, state management, scaling economics, and failure modes. Key findings include the multiplicative growth of agentic workload demand, the entanglement of routing, caching, and scheduling decisions, and the necessity of treating reliability as a control-plane property rather than an edge-level concern. A four-quadrant optimization framework - avoid, share, move, and delay the work - is presented alongside a proposal to adopt cost per successful task as the canonical optimization metric. Practical implications include workflow-aware scheduling, admission control, and the emergence of a unified inference control plane analogous to Kubernetes for compute orchestration.

1. Introduction

Inference has transitioned from a research artifact appended to a product into a foundational hyperscale infrastructure workload growing at a substantial rate. This shift carries consequences for how engineering resources are allocated across the AI stack. A prevailing assumption in the field holds that inference quality and efficiency are determined primarily by model architecture and low-level kernel optimization. The evidence presented in this analysis suggests a different locus of control: the dominant determinants of cost, reliability, and user-perceived latency reside in the distributed systems layer situated between a user prompt and GPU execution.

Several terms recur throughout this analysis. Prefill denotes the compute-bound phase in which an input prompt is processed; decode denotes the memory-bandwidth-bound phase in which output tokens are generated autoregressively. The KV cache refers to per-request key-value state retained across decode steps, resident in high-bandwidth memory (HBM). Time to first token (TTFT) and time between tokens (TBT) serve as principal latency service-level objectives (SLOs). Agentic workloads describe multi-step workflows in which a model issues many sequential calls, frequently without human intervention.

The central thesis is that reliability, cost, and performance in AI inference must be managed holistically across the entire distributed stack, because its layers are mutually entangled rather than independently optimizable. This analysis covers the historical trajectory of AI infrastructure relative to the cloud computing era, the structural divergence between microservice and LLM serving, the hidden decision points behind a single prompt, and the economic and reliability consequences of cascading failure, concluding with the emergence of a unified inference control plane.

2. Background and Related Work

The cloud computing era offers a instructive precedent. Beginning around 2008, value initially accrued to virtual machine (VM) virtualization - the capability of slicing physical hardware into isolated units. Within roughly a decade, value migrated upward the stack: to cluster schedulers such as Borg, Kubernetes, and Mesos, then to service meshes, autoscalers, and higher-order platforms. The scarce capability ceased to be raw compute provisioning and became compute coordination.

AI infrastructure appears to be traversing an analogous trajectory, compressed into a few years rather than a decade. The initial phase consisted of simple models executed directly on GPUs. A second phase introduced serving frameworks - vLLM, Triton, SGLang - addressing batching and memory management at the node level. A third phase is now emerging: an orchestration layer responsible for routing, KV cache management, prefill-decode disaggregation, and multi-model multiplexing. This progression implies that the next competitive frontier is not the best model or fastest kernel, but the surrounding ecosystem and control plane governing their deployment.

3. Core Analysis

3.1 Structural Divergence from Microservice Architectures

Classical microservice serving assumes short, largely uniform requests, stateless execution, and inexpensive, quickly-provisioned compute units. LLM inference violates each of these assumptions. Request length varies from 50 to 100,000 tokens, with prefill and decode phases exhibiting distinct compute profiles. Without continuous inflight batching, throughput can collapse by an order of magnitude, whereas classical stacks batch only optionally at the load balancer. State management differs substantially: LLM serving requires expensive per-request KV cache state that is costly both to construct and to discard, in contrast to the largely stateless nature of microservices. Scaling units diverge as well - GPUs are approximately 100 times more expensive and 10 times slower to acquire than the CPUs underlying traditional microservice infrastructure, making overprovisioning economically punitive. Failure modes compound these differences: a GPU failure mid-decode can drop thousands of in-flight tokens and trigger queue buildup, a scenario with no direct analogue in simple microservice restarts.

3.2 Agentic Demand and Multiplicative Scaling

Classical web serving capacity scaled linearly with users: doubling users doubled queries per second (QPS) and infrastructure requirements proportionally. Agentic serving capacity instead scales as a product of users, calls per user, and tokens per call. Empirically, chatbots generate approximately one model call per turn, co-pilot systems generate 10-20 calls, research agents generate approximately 50 calls, and autonomous workflows can generate thousands of calls with no human in the loop. This multiplicative structure implies that capacity cannot be planned using microservice-style forecasting; it instead requires elasticity, workload-aware scheduling, and admission control mechanisms capable of responding to demand that scales combinatorially rather than linearly.

3.3 The Prompt as a Distributed Transaction

A single prompt traverses numerous infrastructure stages: authentication, model selection, region selection, admission control, cache lookup, GPU execution, and batching. Of these, only the actual prefill/decode step requires the model itself; the remainder constitutes infrastructure. Consequently, "the intelligence might lie in the model but the economics, the reliability, and the user experience are all in the infrastructure." The full lifecycle - gateway, router, cache lookup, scheduler, hardware selection (Nvidia, AMD, in-house silicon), serving runtime, and streamed response - resembles a distributed transaction in which each hop can retry, timeout, fall back, or fail. Partial failure semantics are considerably harder than in standard remote procedure calls (RPC): a GPU preemption occurring after 200 tokens have already streamed cannot simply be retried, since doing so wastes already-completed and costly compute. Because only the control plane observes the entire workflow, reliability "cannot be built at the edge" and must instead be a property of the control plane itself.

3.4 Scheduling Complexity and Cost Economics

Traditional microservice bin-packing considers three to four dimensions, typically CPU, memory, and availability zone. Inference scheduling instead requires awareness across at least seven axes: GPU type, HBM headroom, KV cache state, model weight warm/cold state, tenant priority, workflow context, and latency budget. Workflow-aware orchestration becomes essential when, for example, a request already at step three of five has accrued substantial cost, such that failure at that point wastes disproportionate compute relative to early-stage failures. This context should influence admission decisions, priority assignment, and retry strategy.

Cascading failure represents a particularly consequential dynamic: a degrading GPU raises latency, prompting client retries, which increase queue depth, saturate healthy GPUs, and trigger further retries, ultimately producing regional failures. KV cache recovery compounds this problem, since cold pools must warm up before absorbing traffic while hot pools simultaneously absorb excess load. Mitigations include circuit breakers at the routing layer, admission control that rejects rather than queues excess load, load shedding tied to QPS, and retry budgets.

4. Technical Insights

The four-quadrant optimization framework organizes mitigation strategies applicable across serving stacks (vLLM, SGLang, TensorRT): avoid the work through prefix, response, and semantic caching; share the work through continuous batching, prefill-decode disaggregation, chunked prefill, and speculative decoding; move the work through routing to cheaper models or closer regions; and delay the work through admission control and deadline-aware queueing with priority classes.

A key metric proposed is cost per successful task, superseding cost per token or cost per request, since total cost includes GPU and model expense alongside retries, storage, failure overhead, network cost, and operational development cost. Observability functions as the input signal to control loops: telemetry feeds analysis, which informs scheduling and routing decisions, in continuous iteration. Core metrics include TTFT, utilization ratio, success per dollar, and end-to-end latency. A trade-off triangle among latency, cost, and throughput - analogous to the CAP theorem - governs system design: increased batch size improves throughput and cost but degrades tail latency; speculative decoding improves latency at increased compute cost; smaller models reduce latency and cost but lower quality and increase retry rates.

5. Discussion

These findings collectively suggest that competitive advantage in AI infrastructure is migrating away from model architecture and kernel-level efficiency toward orchestration capability. This parallels the historical evolution of cloud computing, in which coordination layers ultimately captured more value than raw virtualization. The multiplicative nature of agentic demand suggests that as autonomous, multi-step workflows proliferate, the orchestration burden will intensify further, since capacity planning premised on linear user growth becomes increasingly inadequate.

An open question concerns the degree to which control-plane logic should be built internally versus adopted from open-source projects or vendor offerings; the analysis suggests organizations should plan for the existence of such a layer regardless of its provenance. Further investigation is warranted into standardized interfaces for workflow-aware scheduling across heterogeneous hardware (Nvidia, AMD, in-house silicon) and serving runtimes, as fragmentation across these axes currently complicates uniform reliability guarantees.

6. Conclusion

This analysis demonstrates that AI inference at hyperscale is governed less by model or kernel quality than by the orchestration and control-plane infrastructure surrounding them. The divergence from classical microservice architectures - in request shape, batching requirements, state management, scaling economics, and failure modes - necessitates purpose-built scheduling across at least seven axes and reliability mechanisms situated at the control-plane level rather than the edge. Practically, organizations should adopt cost per successful task as their optimization metric, implement the four-quadrant framework across their serving stack, and anticipate the consolidation of routing, batching, caching, and scheduling into a unified inference control plane, mirroring the trajectory from VM management to Kubernetes-style orchestration in the prior cloud computing era.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub