Are LLM Performance Benchmarks Reliable? - Ashok Chandrasekar & Jason Kramberger, Google

Common LLM benchmarking tools produce unreliable, non-reproducible results at production scale due to client-side bottlenecks, poor observability, and incons...

By Sean Weldon

Are LLM Performance Benchmarks Reliable?

Abstract

Benchmarking large language model (LLM) inference systems at production scale is frequently compromised not by the systems under test but by the measurement instruments themselves. This synthesis examines evidence presented by Ashok Chandrasekar and Jason Kramberger (Google) demonstrating that widely used benchmark harnesses suffer from client-side concurrency bottlenecks, poor observability into client execution, and inconsistent workload configuration. The analysis evaluates inference-perf, a Cloud Native Computing Foundation (CNCF) project, which addresses these failures through multi-process load generation, declarative configuration, and standardized workload catalogs. Findings show single-process harnesses saturate between 38 and 170 queries per second (QPS) and can inflate reported latency by up to 58 seconds due to client-side artifacts, while a multi-process alternative sustains 5,000 QPS with accurate reporting. Practical implications point toward client/server dual metric reporting and standardized workload catalogs as preconditions for reproducible benchmarking.

1. Introduction

The transition of LLM inference from single-node experimentation to distributed, autoscaled production deployment has outpaced the sophistication of the tooling used to measure it. Throughput figures, time-to-first-token measurements, and end-to-end latency distributions increasingly underpin decisions about accelerator procurement, replica sizing, and service-level objective (SLO) design. When the benchmarking client itself becomes the performance bottleneck, these decisions rest on artifacts rather than ground truth.

Benchmark harness refers here to the client-side software responsible for generating load against an inference server and collecting resulting metrics. A central failure mode identified across the source material is misattribution: degraded performance measured at the client is diagnosed as a server-side or inference-stack deficiency, when in fact the harness itself cannot generate or record load correctly. As one practitioner observation states, "A lot of times you diagnose it as a your server or inference stack problem but in a lot of cases it could be your benchmark harness."

This synthesis advances the thesis that unreliable LLM benchmarking at production scale stems from three interacting causes: client-side load generation limits imposed by language runtime constraints, inadequate observability into benchmark client behavior, and undocumented divergence in workload configuration across tools. The analysis proceeds by characterizing the benchmarking ecosystem and the distinct demands of production-scale evaluation, cataloguing empirically observed pitfalls, examining the architecture of inference-perf and its associated workload catalog, and distilling technical guidance for practitioners.

2. Background and Related Work

The benchmarking landscape for LLM inference spans at least four distinct segments, each optimized for different objectives. Model server frameworks such as vLLM and SGLang ship built-in Python benchmark scripts intended for rapid developer iteration rather than fleet-scale characterization. Competitive analysis suites - MLPerf, SemiAnalysis, and Artificial Analysis - target cross-vendor chip and accelerator comparison, prioritizing auditable comparability over deployment realism. Generic web load-testing tools, including Locust, k6, and Grafana-based systems, are engineered for high-scale HTTP benchmarking but lack native understanding of tokenization, streaming-token collection, or autoregressive output-length distributions.

Production-scale LLM benchmarking constitutes a fourth, comparatively underserved segment. Distributed inference frameworks such as LLMD combine online serving with batch workloads, prefill/decode disaggregation, and dynamic autoscaling - a level of architectural complexity that simple benchmark harnesses were not designed to simulate. This gap motivates the development of purpose-built tooling, exemplified by inference-perf, a project spun out of the Kubernetes working group on serving and now hosted under the CNCF.

3. Core Analysis

3.1 Structural Bottlenecks in Client-Side Load Generation

A recurring technical cause of benchmark unreliability is Python's Global Interpreter Lock (GIL), which restricts single-process Python harnesses to effectively single-CPU execution regardless of the number of cores available on the benchmarking machine. This limitation manifests concretely: a single-process harness requested to generate 200 QPS on an underpowered machine achieved only 38 QPS, without any warning or error signaling the shortfall. Migrating to a more powerful machine did not resolve the underlying architectural constraint - the same class of harness plateaued at 170 QPS, indicating that the bottleneck is structural rather than resource-dependent. Because the harness silently under-delivers load while still reporting results, practitioners may unknowingly evaluate their inference stack under substantially lower concurrency than intended, producing throughput and latency figures that understate true system capability.

3.2 Observability Failures and Latency Misattribution

Beyond load generation limits, insufficient observability into the benchmark client's own execution produces misleading latency figures. Investigation revealed that client-side thrashing while collecting streaming tokens inflated reported latency by up to 58 seconds. Absent visibility into client behavior, this delay is naturally attributed to the server or inference stack, prompting unwarranted optimization efforts on the wrong component. Validation with a multi-process harness operating at 1,000 QPS against the same simulated server produced near-zero latency, confirming that the earlier measurement was a client artifact rather than a genuine server bottleneck. This finding underscores that metric fidelity requires validating the benchmarking client's behavior in parallel with the system under test, not treating the client as a transparent, error-free instrument.

3.3 Configuration Divergence and Dataset Fidelity

A third source of unreliability arises from undocumented or inconsistent configuration choices across tools. One reported "20% better throughput" result was subsequently traced to the benchmark harness setting model temperature to zero, producing deterministic, more easily cached outputs, in contrast to real-world workloads operating near a temperature of 0.7. Similarly, the same ShareGPT dataset produced different input token counts across benchmark harnesses due to differing sampling and truncation methodologies, meaning that ostensibly identical benchmark runs were, in fact, testing different effective workloads. Additional undocumented behaviors - end-of-sequence forcing, prefix cache rates, and multi-turn conversation replay handling - further erode cross-tool comparability. As the source material emphasizes, "Your data sets do truly matter. Your workloads need to be as close to what you are intending to test as possible."

3.4 Inference-Perf Architecture and the Workload Catalog

inference-perf addresses these failures through three architectural decisions. First, it enforces reproducibility through declarative configuration, in which users specify workload and execution parameters explicitly rather than relying on tool-specific defaults. Second, its load generator employs multiple processes, bypassing the GIL constraint identified in Section 3.1; a main process queues requests according to a planned execution schedule using Poisson, constant-rate, or constant-concurrency models. Third, the tool reports both client and server metrics simultaneously, providing the observability needed to detect the artifacts described in Section 3.2. At 5,000 QPS, inference-perf maintained load and accurately reported success, in contrast to other HTTP tools such as k6, which could not sustain equivalent throughput.

Complementing the tool itself, a published workload catalog standardizes benchmark definitions across multi-turn generation, tree-of-thought reasoning, agentic generation, and batch summarization scenarios. Each catalog entry includes a natural language description alongside detailed configuration metadata, enabling cross-tool standardization that mitigates the dataset and configuration divergence documented in Section 3.3.

4. Technical Insights

Several implementation-level findings merit direct attention for practitioners designing or selecting benchmarking infrastructure. Single-process harnesses should be treated with caution above modest QPS targets (empirically observed ceilings of 38-170 QPS), regardless of host machine capability, because the constraint is architectural rather than resource-bound. Multi-process or multi-worker load generators are necessary preconditions for benchmarking at production-representative concurrency levels, such as the 5,000 QPS sustained by inference-perf.

Client-side metric collection, particularly for streaming token responses, requires independent validation against a known-latency reference server; without this, thrashing-induced delays of tens of seconds can be misattributed to the inference stack. Stochastic parameters such as temperature must match production workload characteristics (e.g., approximately 0.7 rather than 0), since deterministic settings materially and misleadingly alter throughput results. Finally, dataset preprocessing - sampling, truncation, end-of-sequence handling, and multi-turn replay logic - must be documented and standardized, as identical named datasets can yield divergent effective workloads across tools.

5. Discussion

These findings collectively indicate that benchmark reliability at production scale is not primarily a matter of statistical rigor but of instrument validity. A benchmarking client that silently fails to generate requested load, misattributes its own latency to the server, or diverges in configuration from stated intent produces results that are precise but not accurate. This distinction has practical consequences for capacity planning and accelerator procurement, where benchmark outputs directly inform capital allocation decisions.

The emergence of inference-perf as a CNCF project, alongside the LLMD framework and its Prism visualization UI, suggests an industry trend toward treating benchmarking infrastructure as a shared, standardized concern rather than a per-team or per-tool implementation detail. The published workload catalog, covering multi-turn generation, tree-of-thought, agentic generation, and batch summarization, represents an attempt to close the configuration-divergence gap identified in Section 3.3. Reported results using this pipeline - an agentic code generation workload tested on TPUs with eight replicas, scaling into hundreds of thousands of tokens per second relative to a baseline Kubernetes service - illustrate the tool's applicability at genuine production scale.

Open questions remain regarding the extent to which the workload catalog can capture the full diversity of production traffic patterns, and whether declarative configuration alone is sufficient to prevent future divergence as new serving architectures (e.g., further disaggregation strategies) emerge.

6. Conclusion

This synthesis demonstrates that unreliable LLM benchmarking results frequently originate in the measurement client rather than the inference stack under test. Structural constraints such as the Python GIL, insufficient client-side observability, and inconsistent workload configuration each independently produce misleading performance figures, and their effects compound in production-scale evaluation. inference-perf addresses these failures through multi-process load generation, declarative configuration, and dual client/server metric reporting, sustaining 5,000 QPS where comparable tools fail.

The practical takeaway for practitioners is that benchmark tooling itself requires validation before it can be trusted to validate an inference stack. Adoption of standardized workload catalogs and dual-sided metric reporting constitutes a reasonable near-term step toward reproducible, comparable LLM benchmarking across the industry.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub