What's New in Inference Engineering - Philip Kiely, Baseten

Inference engineering continues to evolve rapidly since the publication of 'Inference Engineering' in February 2026, with new advances in quantization, KV ca...

By Sean Weldon

What's New in Inference Engineering: A Technical Synthesis

Abstract

Inference engineering - the discipline of serving trained neural networks efficiently under latency, throughput, and memory constraints - has continued to evolve rapidly since the February 2026 publication of Inference Engineering. This synthesis consolidates recent developments across three technical fronts: quantization, key-value (KV) cache management, and speculative decoding. It argues that the field has bifurcated into two optimization regimes - local inference, constrained primarily by memory capacity, and data center inference, constrained by throughput and draft-token acceptance rate - such that techniques optimal in one regime are frequently unacceptable in the other. Evidence includes Turboquant's 4-bit KV cache quantization, which halves memory but reduces tokens per second (TPS) by more than half, and Dlash diffusion-based speculative decoding, which yielded a measured >3× improvement over Eagle 3 on a single B200 serving Qwen 38B. A recurring meta-trend is the dissolution of the boundary between training and inference optimization, with faster serving generating data that improves subsequent models.

1. Introduction

The pace of change in inference optimization raises a legitimate methodological concern, articulated directly in the source material: "Why in the world would you do this? Why would you write a book about something that's changing so fast?" The implicit answer is that while specific techniques churn quickly, the underlying constraint structure - memory bandwidth, memory capacity, arithmetic intensity, and acceptance rate - remains stable and analyzable.

Inference engineering refers to the set of techniques for serving trained models efficiently, spanning quantization, caching, routing, and decoding strategies. This synthesis defines four themes: the emergence of two structurally distinct inference practices, the state of quantization following Turboquant, KV cache compaction strategies that shift compression cost between training and inference time, and speculative decoding advances centered on diffusion-based drafting. A concluding discussion addresses the increasingly porous boundary between training and serving optimization, framed as a feedback loop in which faster inference generates more data for training better models.

2. Background and Related Work

Modern autoregressive transformer serving is constrained principally by the KV cache: stored attention keys and values for prior tokens that permit incremental decoding without recomputation. The KV cache is characterized in the source material as "lossless memory" - it preserves full token-level fidelity but scales linearly with input length, distinguishing it from sublinear alternatives such as retrieval-augmented generation (RAG) or file-writing, which compress context at the cost of information loss.

Prior work addresses serving efficiency along several axes: quantization of weights and activations (e.g., NVFP4), distillation and pruning for local deployment, KV-aware routing that directs requests to workers already holding relevant cache state, PD Disaggregation (prefill-decode separation, operationalized in systems such as Nvidia Dynamo), and speculative decoding, which amortizes sequential decode cost by drafting multiple tokens verified in a single target-model forward pass.

3. Core Analysis

3.1 Two Regimes of Inference Engineering

The field has stratified into two practices with different constraint orderings. In local inference, the binding constraint is hardware capacity at batch size one; the workflow proceeds by first making the model fit - via quantization, distillation, and pruning - then restoring intelligence lost in that process. In data center inference, the workflow inverts: build correctness (e.g., a functioning VLM pipeline) is established first, followed by speed optimization through KV-aware routing, speculation, and disaggregation. This bifurcation explains why identical techniques receive opposite verdicts depending on deployment context, as demonstrated in the quantization findings below.

3.2 Quantization: The Turboquant Trade-off

Turboquant employs polar-coordinate representations to quantize the KV cache down to 4 bits, halving memory footprint and doubling effective bandwidth for KV cache movement. However, this compression requires additional computation during the forward pass at decode time, cutting TPS by more than half. The source material notes this technique gained substantial visibility ("20 million people saw Turboquant and in fact it made the memory stock macro dip for a minute"), yet the underlying trade-off is deemed unacceptable for data center production workloads, where throughput dominates. In local inference, where memory capacity - not TPS - is the binding constraint, the same technique remains valuable. Base10 continues to rely on traditional NVFP4 quantization for weights rather than KV cache, and is exploring quantization strategies across additional modalities including image and video.

3.3 KV Cache Compaction: Training-Time versus Inference-Time Compression

Given that the KV cache scales linearly and losslessly, compaction techniques attempt to reduce its footprint without discarding information wholesale. Methods such as attention matching and cartridges achieve high compression ratios but operate entirely at inference time, imposing per-request computational cost.

Base10's Still technique, developed by the post-training team, instead amortizes compression cost via training. Still employs a perceiver-style bottleneck: a fixed set of learned "loaned" query vectors cross-attends against the full KV cache to produce compact key/value representations in a single forward pass. Because the compression mechanism is learned and differentiable, it is amortized during training rather than recomputed at inference time, producing what the source material terms "a fast differentiable compressed memory." This represents a structural alternative to inference-time compaction: rather than paying a computational tax on every request, the cost is paid once during model preparation.

3.4 Speculative Decoding: From Autoregressive Drafting to Diffusion

Speculative decoding has progressed through a clear lineage: Spec (small draft model generating tokens sequentially), Medusa (additional decoder heads attached to the target model), and Eagle 3 (a billion-parameter model trained on the target model's hidden states to produce draft tokens).

The newest advance, Dlash, departs from autoregressive drafting entirely by using diffusion-based generation to produce 8-16 draft tokens per forward pass, rather than one token at a time. Dlash forward passes are individually 2-4x slower than a comparable autoregressive step, but because each pass predicts substantially more tokens with a higher acceptance rate, the net effect is a speed improvement: "a single Dlash forward pass is faster than the entire Eagle draft phase and predicts more tokens." This is achieved through attention masks that permit bidirectional drafting within a block while preserving causal consistency across the sequence. Base10 measured a greater than 3x improvement using Dlash versus Eagle on a single B200 serving Qwen 38B.

A newly released successor, DSpark, pairs a diffusion model with a sequential model to further improve acceptance rates, though production results are not yet available. Separately, continuous speculator retraining on live production prompts and responses shows 20% to 2x improvement in token acceptance rates, at the cost of additional infrastructure and data-permission complexity.

4. Technical Insights

Several implementation-relevant findings emerge from this analysis:

5. Discussion

The findings collectively suggest that inference engineering is no longer a purely serving-side discipline. The feedback loop described in the source material - faster inference generating more data, which trains better models, which enables faster inference - indicates that optimization decisions increasingly require joint consideration of training and serving pipelines. Still's training-amortized compaction and continuous speculator retraining both exemplify this convergence.

A notable gap is the absence of production validation for DSpark, suggesting that diffusion-paired sequential speculation remains an active research direction rather than a settled technique. Similarly, the regime-dependent verdict on Turboquant implies that future quantization research may need to explicitly target one deployment context rather than proposing universal solutions.

6. Conclusion

This synthesis demonstrates that recent inference engineering progress is organized around a stable set of constraints - memory capacity, bandwidth, and acceptance rate - even as specific techniques evolve rapidly. Practical takeaways include matching quantization strategy to deployment regime, considering training-time amortization for KV compaction, and adopting diffusion-based speculative decoding where aggregate throughput, not per-pass latency, is the relevant metric. As disaggregation and systemwide KV cache communication grow in importance alongside NVFP4 on emerging hardware such as Ruben, training-for-inference is likely to remain a defining theme shaping the field's next phase.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub