Deep dive on LLM Inference at Scale - Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

LLM inference performance is governed by fundamental trade-offs between memory, latency, and throughput driven by KV cache growth, and understanding these fi...

By Sean Weldon

Deep Dive on LLM Inference at Scale: A First-Principles Analysis

Abstract

Large Language Model (LLM) inference has become the dominant recurring cost in production AI systems, with the inference market estimated at approximately $23 billion. Unlike training - a one-time expenditure exemplified by GPT-3's roughly $4.6 million cost - inference scales continuously with usage. This synthesis develops a first-principles account of inference cost, tracing bottlenecks to key-value (KV) cache growth and the divergent hardware characteristics of prefill and decode phases. It quantifies memory mathematics for representative models, formalizes a quality-latency-throughput trade-off triangle, and evaluates model-level optimizations (quantization, attention redesign, Flash Attention) and serving-level optimizations (paged attention, continuous batching, prefix caching, speculative decoding). Benchmark evidence shows approximately 15× throughput gains from default vLLM over naive baselines and 3-4× advantages for SGLang on agentic workloads, yielding practical guidance for engine selection.

1. Introduction

The economics of deployed language models are increasingly governed by inference rather than training cost. A useful thought experiment illustrates the scale of the problem: modeling Google-scale search traffic through LLMs implies a profit drain on the order of $36 billion unless per-query cost is driven below half a cent. This asymmetry - a one-time training cost versus a perpetually recurring inference cost - motivates the central question of this analysis: what are the fundamental constraints that make inference expensive, and which optimizations address which constraints?

Key terminology anchors the discussion. The KV cache stores key and value vectors computed during attention so they need not be recomputed at every generation step. Time to First Token (TTFT) measures latency to the first generated token, dominated by the prefill phase. Inter-token latency (ITL) measures the time between subsequent tokens, dominated by the decode phase. These two phases exhibit opposite computational profiles - compute-bound versus memory-bound - and this asymmetry is the root cause of most serving trade-offs examined below.

The central thesis is that inference performance is not a loose collection of independent engineering tricks but a consequence of a small number of first-principles constraints: KV cache growth, GPU high-bandwidth memory (HBM) limits, and attention's compute profile. Four observable pain points motivate the analysis: memory consumption growing with context size, TTFT degradation with longer contexts, throughput collapse under sequential request handling, and elevated inter-token latency during decode. The remainder of this synthesis proceeds through pipeline foundations, memory mathematics, the trade-off triangle, and a structured survey of model- and serving-level optimizations, concluding with engine selection guidance.

2. Background and Related Work

The generation pipeline proceeds from text to tokens, tokens to embeddings, embeddings through stacked transformer layers, and into an autoregressive generation loop. Approximately 95% of inference compute is consumed by the transformer layers, and within those layers, the attention mechanism is the most compute-intensive component, since every token must be projected into query, key, and value vectors with cost scaling in sequence length.

This computational structure motivates the KV cache: without caching, generating each new token would require O(N²) recomputation of key and value vectors for all preceding tokens. The KV cache converts this redundant computation into persistent memory pressure - a trade that underlies nearly all subsequent optimization strategies discussed in this work, including Paged Attention (drawing an analogy to OS virtual memory), continuous batching, and prefix caching structures such as RadixAttention.

3. Core Analysis

3.1 KV Cache Memory Mathematics and the Prefill/Decode Asymmetry

For Mistral 7B, the per-token KV footprint is calculated as 2 vectors × 128 head dimensions × 32 layers × 8 KV heads (reduced from 32 heads via grouped query attention), yielding 131 KB per token. This scales predictably: 4K context requires approximately 0.5 GB per user; 16K context requires approximately 2.1 GB per user; and 80 concurrent users at 4K context require 42 GB of KV memory alone. Since GPU memory is partitioned into fixed model weights, fixed overhead, and leftover capacity for KV cache, this creates a direct trade-off between context length served and the number of concurrent users supportable on a single GPU.

The prefill and decode phases diverge sharply in computational character. Prefill builds KV vectors for all input tokens and computes attention scores in a single pass; it is compute-bound and determines TTFT. Decode generates one token at a time, recomputing attention against all previously cached KV vectors; it is memory-bound and determines inter-token latency. This asymmetry is explained by GPU memory architecture: high-bandwidth memory (HBM) is large but comparatively slow, while shared memory (SRAM) is small but fast. Decode is bottlenecked by repeated HBM transfers per token, producing low arithmetic intensity (flops per byte transferred), whereas prefill's batched computation yields high arithmetic intensity - a distinction visualized via the roofline model.

3.2 The Quality-Latency-Throughput Trade-off Triangle

Maximum concurrent users on a GPU is bounded by available memory divided by KV size per user, itself constrained by context length. Increasing concurrent users therefore requires either more memory or reduced context length, with the latter risking quality degradation. Latency service-level objectives (SLOs) constrain both TTFT and inter-token latency as batch size grows, producing a three-way trade-off: quality, latency, and throughput, of which typically only two can be prioritized simultaneously. Premium chat applications favor low latency and high quality at the expense of throughput, while asynchronous agent workloads can favor throughput. A capacity calculator tool was demonstrated to operationalize this trade-off, fixing latency and batch-size dimensions first to determine optimal GPU selection.

3.3 Model-Level Optimizations

Quantization compresses model weights directly: for the GPT-OSS 120B model, BF16 requires 240 GB, FP8 reduces this to approximately 120 GB, and MXFP4 reduces it further to approximately 65 GB, enabling deployment on a single H100. For Mistral 7B, FP16 requires 14.6 GB, INT8 requires 7.5 GB (2× compression), and 4-bit quantization requires approximately 4.5 GB (4× compression).

Attention mechanism design forms a compression spectrum: Multi-Head Attention (MHA) uses 32 KV blocks with no compression; Grouped Query Attention (GQA) groups KV blocks; Multi-Query Attention (MQA) collapses to a single KV block, achieving high compression at some quality cost. Multi-Head Latent Attention (MLA) compresses KV into latent vectors, achieving a corrected 14× compression relative to MHA (an initial claim of 56× was revised after identifying a computational error omitting layer multiplication). DeepSeek Sparse Attention further limits computation to only the most important prior tokens rather than the full sequence. Flash Attention tiles query/key/value matrices into small blocks processed in SRAM, using online softmax to avoid repeated HBM writes, directly addressing the bandwidth bottleneck identified in Section 3.1. Linear attention and Mamba-style state space models are proposed as architectural alternatives that avoid full attention matrix multiplication entirely.

4. Technical Insights

Serving-level optimizations operate independently of model architecture and can be layered. Paged Attention resolves memory fragmentation via logical-to-physical block mapping, analogous to OS virtual memory. Continuous batching eliminates GPU idle time by admitting new requests without waiting for an entire batch to complete. Prefix caching, introduced by vLLM, saves computation across requests sharing common prefixes; however, static hashing-based implementations fail on small prompt variations, causing cache misses. SGLang's radix tree structure addresses this by collapsing branchless nodes in a prefix tree, proving especially valuable for agentic workloads with repeated, similar prompts. KV cache quantization reduces memory footprint of cached vectors while maintaining comparable throughput and latency, freeing capacity for more context or more concurrent users.

Benchmark evidence quantifies these gains: a raw HuggingFace baseline achieves approximately 51 tokens/sec, while default vLLM (combining paged attention, continuous batching, and KV caching) shows an approximately 15× throughput improvement on H100 hardware. Adding prefix caching further increases throughput and decreases TTFT while maintaining similar inter-token latency.

Speculative decoding - using a smaller draft model to propose tokens verified by a larger "referee" model - was found, based on personal testing, not particularly useful due to alignment issues between draft and teacher models. Self-speculative decoding (using an auxiliary head on the teacher model itself), the EAGLE algorithm (generating internal-layer features rather than tokens), and Medusa (parallel rather than sequential token generation) represent refinements considered superior to standard speculative decoding.

5. Discussion

These findings collectively suggest that inference optimization should be approached as a layered decision problem rather than a single technique selection. Model-level optimizations (quantization, attention redesign) reduce the fundamental resource footprint, while serving-level optimizations (paged attention, continuous batching, prefix caching) extract efficiency from a fixed footprint. Both families address the KV cache bottleneck from different angles, and their benefits compound rather than substitute for one another.

Engine benchmarking data reveals an important nuance: vLLM and SGLang show no statistically significant difference on standard non-agentic workloads (measured on ShareGPT), but SGLang performs 3-4× better on agentic, branching, multi-turn workloads due to its radix-tree-based prefix caching. This suggests that engine selection should be workload-dependent rather than universal, with TensorRT-LLM representing a distinct hardware-optimization layer applicable to Nvidia GPUs specifically. Emerging engines - Nvidia Dynamo for agentic session routing, HuggingFace for simplicity, and Stanford's MSAR for multimodal workloads - suggest continued specialization rather than convergence toward a single dominant engine.

6. Conclusion

This synthesis demonstrates that LLM inference cost is not incidental but structurally determined by KV cache growth and the compute/memory asymmetry between prefill and decode phases. Quantitative memory mathematics, the roofline model, and the quality-latency-throughput trade-off triangle provide a principled framework for reasoning about optimization choices. Practically, this implies that practitioners should first characterize their workload (chat versus agentic, latency-sensitive versus throughput-sensitive) before selecting model-level compressions and serving engines. The recommended default is vLLM for production general-purpose serving, with a switch to SGLang for agentic workloads exhibiting branching or repeated-prefix structure.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub