Why Inference Matters 12x More Than Training | YC Paper Club

Inference, not training, is the dominant cost and engineering challenge of production LLM deployment: inference compute, cost, and quality consistency now de

Abstract

Production deployment of Large Language Models (LLMs) has shifted the dominant engineering constraint from training to inference. Operational data from DeepSeek V3 indicates approximately six times more floating-point operations (flops) are spent on inference than training per weight file per year, implying inference accounts for roughly twelve times lifetime compute over a two-year model life. This synthesis consolidates findings across five areas of the inference stack: economic framing and key performance indicators, the request lifecycle at production scale, quality variance across providers serving nominally identical open weights, compiler construction via equality saturation, and KV-cache architecture in the SGLang runtime. Reported results include a 56% reduction in time to first token from hierarchical caching, 1.1-2x throughput gains from overlap scheduling, and training-free quantization matching original model quality. Implications concern margin structure, benchmark methodology, and reliability engineering for open-model serving infrastructure.

1. Introduction

Inference denotes the serving of a trained model's forward pass to end users in production; training denotes the one-time production of the weight file. Historically, research framing - exemplified by the Chinchilla scaling paradigm - has optimized architectures under ISO-train-flop comparisons, treating training compute as the primary cost variable to control for when comparing models or methods. The central thesis advanced here is that this framing is economically inverted: inference compute, cost, and quality consistency now determine margins, user experience, and competitive outcomes in the open-model ecosystem.

The quantitative basis is direct. DeepSeek V3 required approximately 2.8 million H800 GPU-hours to train, while production inference consumed roughly 43,000 H800 GPU-hours per day, yielding an annual inference-to-training flop ratio of approximately 6x. Over a plausible two-year deployment window, this implies inference accounts for roughly 12x the lifetime compute of training. This motivates a methodological correction: architecture and benchmark papers should be evaluated under ISO inference flops rather than ISO train flops alone, since the latter framing systematically undercounts the dominant cost center of a deployed model.

"I think we should really start caring about ISO inference flops."

Inference is estimated to have produced $220B of market capitalization over ten years, and DeepSeek's reported ~80% margins are attributed principally to inference-side engineering rather than model novelty. This analysis covers five domains: the performance model and KPIs governing inference economics; the request lifecycle at the scale of hundreds of trillions of tokens per month; the empirically observed quality variance across inference providers; compiler design via equality saturation for mapping models to diverse hardware; and KV-cache architecture innovations addressing hybrid attention mechanisms.

2. Background and Related Work

The governing analytical tool is roofline analysis, which bounds achievable performance by either peak flops or memory bandwidth, depending on the arithmetic intensity of a given operation. Model flop utilization (MFU) - the fraction of theoretical peak flops realized in practice - operationalizes this bound for LLM workloads. The apparent simplicity of the forward pass is misleading:

"It's just model.forward. This is it's not that hard. It's actually really, really hard."

MFU reaches approximately 50% during training and prefill (the compute-bound phase where many tokens are processed in parallel) but falls to as low as 5% during decode (the memory-bandwidth-bound phase generating one token per sequence per step). This prefill/decode asymmetry structures nearly all subsequent optimization work described below.

Relevant prior work spans three levels of the stack. At the model level: Mixture of Experts (MoE), State Space Models (SSMs) and linear attention variants, multi-head latent attention, quantization, and distillation. At the kernel and runtime level: FlashAttention, PagedAttention, and RadixAttention. At the harness level: routing, speculative decoding, recursive language models, and context compaction. Looped models - adding recurrent compute without additional weights - represent a further architectural direction trading inference flops for parameter count efficiency.

3. Core Analysis

3.1 Service-Level Metrics and the Economics of Caching

Four KPIs govern production serving: time to first token (TTFT), time per output token (TPOT), throughput per GPU, and goodput (throughput achieved within a service-level agreement). Reported figures illustrate the magnitude of caching effects: prefill throughput without cache reuse reaches 60,000 tokens/sec/GPU, while a 96% cache hit rate raises this to 1.5 million tokens/sec/GPU - a 25x improvement attributable entirely to avoiding redundant computation. Decode throughput, constrained by memory bandwidth rather than compute, remains at only 14,000 tokens/sec/GPU regardless of caching. Average TTFT is approximately 400ms, with P99 latency near 1 second. These figures explain why providers retain KV caches for extended windows (e.g., 72 hours): sustained cache hit rates directly determine unit economics.

3.2 Request Lifecycle and Scale-Up Hardware

At scale (Together AI processes approximately 400 trillion tokens/month, about one-eighth of Google's total traffic), the request lifecycle decomposes into routing, tokenization, scheduling/batching, prefill, decode, and detokenization. Routing must jointly optimize load balancing and KV-cache locality, since repeat requests sent to the same GPU avoid cache misses. Six levers are identified for accelerating inference: kernels, disaggregation (separating prefill and decode onto distinct hardware pools matched to their respective compute/memory profiles), parallelism, routing, speculative decoding, and quantization. Nvidia's GB-rack architecture extends scale-up domains from 8 to 72 GPUs at high interconnect bandwidth, enabling expert parallelism - placing individual MoE experts on separate GPUs - but introduces a reliability liability: a single GPU failure within a 72-GPU domain can take down an entire model replica absent dedicated remediation systems.

3.3 Quality Variance Across Providers

Serving the "same" open-weight model does not guarantee equivalent output quality across providers. Research cited in this analysis finds the quality gap between the best and worst provider serving an identical model can exceed the gap between open and closed models generally:

"The gap between the worst provider for a model and the best provider is actually bigger than the gap between closed and open models."

Causes include over-quantization, incorrect model versions, tool-calling/JSON parsing errors, divergent sampling techniques, and kernel- or KV-cache-precision differences. This matters disproportionately for agentic workloads: a 10% per-step error rate compounded over 15 sequential steps yields approximately 79% task failure. Monitoring approaches range from free (tool-call error logging) to cheap (log-probability drift detection) to moderately costly (distribution tests against reference weights, ~$1/test) to expensive (full benchmark evaluation). Emerging mitigations include labs publishing complete inference stacks and pre-quantized official weights, and Nvidia GPU trusted execution environments that cryptographically attest to exact weights and software serving a request.

3.4 Compiler Design via Equality Saturation

Luminal's compiler approach decomposes models into approximately 14-15 primitive linear-algebra operations, described as "PyTorch complete" rather than Turing complete - a restriction that avoids the undecidability implications of Rice's theorem and permits exhaustive reasoning about program behavior. E-graphs store all equivalent program rewrites simultaneously until "saturated," after which candidate programs are extracted, compiled, and empirically timed on real hardware:

"The best cost model is no cost model and we should actually just go and run these things on the real hardware."

Search targets a 99%-of-optimal local optimum via genetic algorithms, exploring only about 0.01% of the rewrite space, since compile-time cost amortizes across millions of subsequent inference requests.

4. Technical Insights

Several implementation-relevant findings recur across the sources. First, KV-cache structure is a primary lever for cost reduction: SGLang's radix-tree-based RadixAttention enables automatic prefix reuse across agentic and tool-calling workloads sharing system prompts, and a multi-tier hierarchical cache (GPU memory, host memory, storage) reduced TTFT by 56% in reported experiments. Second, scheduling overlap - delaying host-device synchronization so CPU batch preparation overlaps GPU execution - yields 1.1-1.5x throughput gains on Hopper hardware and up to 2x on Blackwell, with DeepSeek's DeepInfra independently adopting the same technique. Third, newer architectures combining full attention, sliding-window attention, and recurrent/SSM state layers complicate cache sharing, since cache validity depends on matching different token windows per layer type; SGLang's "Unified Radix Tree" and a dynamically-growing unified memory pool (replacing fixed-ratio pre-allocation) address this fragmentation. Fourth, quantization approaches favoring training-free methods (avoiding fine-tuning on original data, which risks degrading model behavior) rely on calibration data and scale search, verified via per-layer KL divergence, public benchmarks, and agentic evaluation loops. A trade-off worth noting: Luminal's local-optimum search strategy sacrifices global optimality for tractable compile times, which is justified only because inference request volume amortizes the one-time compilation cost.

5. Discussion

These findings collectively suggest that inference engineering has become a primary determinant of competitive differentiation in the open-model ecosystem, rather than a downstream implementation detail of model research. The observation that provider-to-provider quality variance can exceed open-versus-closed model gaps implies that model weights alone are an incomplete unit of comparison; the serving stack - quantization recipe, kernel precision, sampling implementation - is now part of the product being evaluated.

A notable gap concerns standardization: absent attestation mechanisms or published reference inference stacks, benchmark results reported for an "open model" may not generalize across providers claiming to serve it. The trend toward labs publishing complete inference stacks and pre-quantized official weights, alongside hardware-level attestation, represents a partial response to this gap, though broad adoption remains unproven as of this writing.

The shift toward ISO-inference-flop framing also has implications for research methodology: architectural choices (MoE routing, SSM layers, multi-head latent attention) that appear costly under train-flop accounting may be favorable once decode-phase memory bandwidth constraints and cache-reuse economics are incorporated.

6. Conclusion

This synthesis establishes that inference, not training, is the dominant economic and engineering constraint on production LLM deployment, with lifetime inference compute estimated at roughly 12x training compute for comparable weight files. Innovation addressing this constraint spans model architecture, kernel and runtime design, compiler construction, and harness-level orchestration, with measured gains including a 56% TTFT reduction from hierarchical caching and up to 2x throughput from overlap scheduling. Practically, this implies that model evaluation, provider selection, and architecture research should incorporate inference-phase metrics - cache hit rate, decode throughput, provider quality variance - rather than relying on training-flop comparisons alone.

Sources

• Why Inference Matters 12x More Than Training | YC Paper Club - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub