Stop Shipping LLM Settings That Only Win on Paper with Rohan Pandey & Balaji Varadarajan, DigitalOcean
An automated optimization agent using Bayesian optimization can find the best VLM (vLLM) serving configuration for any given model faster than manual tuning,
By Sean WeldonStop Shipping LLM Settings That Only Win on Paper
Abstract
Serving large language models in production requires selecting among dozens of interacting engine parameters, a task conventionally performed through manual trial and error. This synthesis examines an automated optimization agent that reframes vLLM serving configuration as a black-box optimization problem, solved using Bayesian optimization (BO) implemented through the Optuna framework. The agent ingests a model's default serving recipe, evaluates candidate configurations across four representative workload regimes, scores each trial by generated goodput tokens, and outputs a tuned YAML recipe alongside an HTML report. Two documented runs demonstrate measurable gains: a GPT-OSS 120B study produced a 17.5% improvement in tokens generated over 50 trials, while a DeepSeek R1 Distill study doubled token throughput and reduced inter-token latency (ITL) by nearly half across just 9 trials. These results suggest that background-executed, sample-efficient search can substitute for manual serving-recipe tuning without sacrificing measured performance.
1. Introduction
Inference serving has become a dominant cost center in large-scale LLM deployment. Serving engines such as vLLM expose a large configuration surface - batching limits, sequence-length budgets, memory utilization targets, and parallelism settings - whose optimal values depend jointly on the model architecture, underlying hardware, and traffic pattern. Each candidate configuration can only be validated by executing a live benchmark against a loaded model, making evaluation expensive and the overall search space combinatorially large.
The central thesis examined here is that an automated optimization agent can identify a near-optimal serving configuration for an arbitrary model faster than a human engineer performing manual tuning, with the improvement verifiable through throughput and latency metrics. The agent's stated purpose is explicitly economic: to reduce the human time required to discover optimal configs and serving recipes. Running the search in the background is presented as the principal mechanism of savings, since the search consumes machine hours rather than engineer hours.
Two terms are foundational to the discussion. A serving recipe denotes the complete parameterization of the inference engine, expressed as a YAML file. Goodput denotes the volume of useful tokens generated under a given workload and serves as the agent's optimization objective. This analysis proceeds by establishing the optimization-theoretic background (§2), examining the agent's mechanism and empirical results (§3), extracting implementation guidance (§4), discussing broader implications (§5), and concluding (§6).
2. Background and Related Work
The underlying problem belongs to the class of black-box optimization: the mapping from configuration to performance has no closed form, no accessible gradient, and a costly evaluation function, since each trial requires actually running inference against the target model. Grid search and random search are conventional baselines but scale poorly once a single evaluation consumes meaningful GPU time.
Bayesian optimization addresses this regime directly. BO maintains a probabilistic model of the objective and uses it to select the next configuration to test, balancing exploration of uncertain regions against exploitation of configurations already known to perform well. As stated in the source material, BO "uses mathematical steps to minimize the search space," reducing the number of evaluations - and therefore wall-clock time - needed to locate strong parameters. The operative property is sample efficiency: BO is valuable not because it guarantees a better asymptotic optimum, but because it reaches a good optimum in comparatively few trials.
A distinction preserved explicitly in the source material clarifies a common point of confusion: "Bayesian optimization is a strategy that is used but Optuna is the tool that uses a strategy in the search loop." That is, BO is the mathematical method; Optuna is the software implementation that operationalizes it as an iterative search loop - feeding a recipe, running a trial, scoring the result, and adjusting the recipe before repeating.
3. Core Analysis
3.1 Agent Architecture and Search Loop
The agent operates on a droplet with the target model (for example, GPT-OSS 120B) already loaded into memory. A starting configuration - the default recipe shipped with the downloaded model weights - is submitted to Optuna, which drives the BO search loop: it feeds the current recipe into the optimizer, executes a trial, assesses the resulting performance, assigns a score, and adjusts the recipe before the next iteration. This cycle repeats for a specified number of trials, progressively narrowing toward an optimized configuration.
Each configuration is evaluated not against a single workload but across four distinct regimes: chat short, chat long, rag, and batch classify. Each regime carries its own ISO/OSL (input sequence length/output sequence length) token settings, which are independently configurable. This multi-regime evaluation is significant because it prevents the agent from overfitting a configuration to a single traffic pattern that would fail to generalize to other production workloads - directly addressing the thesis implied by the title, that settings which "win on paper" under one test condition may not hold up under realistic, varied usage.
3.2 Execution Workflow
Running the agent is operationally straightforward: after activating a virtual environment, the user executes a script called tuner.py, supplying four required inputs - the study name, the starting YAML configuration file, the target hardware specification, and the number of trials to run. Reported trial counts varied across runs, with one study configured for 24 trials and another for 9, indicating that trial count is a tunable budget parameter rather than a fixed constant.
Typical runs complete in a couple of hours; the GPT-OSS run specifically required four to five hours. Upon completion, the agent produces two artifacts: an HTML report identifying the best-performing trial number out of the total trials executed, and an optimized YAML file containing the tuned serving parameters ready for deployment. The entire workflow therefore requires minimal manual intervention beyond initial setup and final review of results.
3.3 Empirical Results
Two model studies illustrate the agent's measured impact. In the GPT-OSS 120B run, 50 trials were executed, and the agent identified 256 as an optimized parameter value, producing an overall 17.5% improvement in tokens generated relative to the starting configuration: "Overall we found that from the first config we had an improvement of 17.5% in terms of tokens generated."
In the DeepSeek R1 Distill run, only 9 trials were executed, with the ninth trial identified as the best. Despite the smaller trial budget, the gains were substantially larger in relative terms: token throughput doubled, and ITL was reduced by nearly half: "We found that token throughput doubled and ITL was improved by almost half aka it was reduced by almost half." The disparity between trial counts (50 versus 9) and relative improvement magnitude (17.5% versus a doubling) suggests that the achievable gain from tuning is highly model- and baseline-dependent, rather than a fixed function of search budget.
4. Technical Insights
- Sample efficiency over trial volume: The
DeepSeek R1 Distillstudy achieved larger relative gains in fewer trials (9) than theGPT-OSS 120Bstudy achieved in 50, indicating that the magnitude of improvement depends more on how far the default recipe was from optimal than on the number of trials run. - Multi-regime scoring as a generalization safeguard: Evaluating each candidate configuration across chat short, chat long, rag, and batch classify workloads, each with distinct ISO/OSL settings, guards against configurations that excel on a narrow benchmark but underperform under realistic mixed traffic - the practical concern implied by the piece's framing around settings that "only win on paper."
- Goodput as the objective function: Scoring by useful tokens generated rather than raw throughput or latency alone aligns the optimization target with production-relevant output quality, though the source material does not specify how goodput is computed when outputs are truncated or malformed.
- Operational simplicity: The four required inputs to
tuner.py- study name, starting YAML, hardware specification, and trial count - represent a minimal interface, suggesting the agent is designed for repeatable use across different models and hardware targets without bespoke integration work. - Runtime as a tunable cost trade-off: Run durations of two to five hours position the agent as a background process rather than an interactive tool, trading machine time for engineer time.
5. Discussion
The results support the thesis that automated, sample-efficient search is a viable substitute for manual serving-recipe tuning, at least for the two models examined. The magnitude of improvement - particularly the doubling of throughput for DeepSeek R1 Distill in only 9 trials - indicates that default serving recipes shipped with model weights may be substantially suboptimal for specific hardware and workload combinations, leaving meaningful performance on the table for teams that do not tune beyond defaults.
A notable gap in the source material is the absence of a direct comparison against manual tuning in terms of wall-clock time or engineer-hours saved; the claimed efficiency advantage over manual search is asserted rather than empirically benchmarked within the reported results. Similarly, no information is given on how well a configuration optimized for one hardware target or model generalizes to a different one, which would be the critical question for teams considering amortizing a single tuning run across a fleet of similar deployments.
This work fits into a broader industry trend of applying classical optimization techniques - originally developed for hyperparameter search in model training - to the adjacent problem of inference infrastructure tuning. As inference cost increasingly dominates the operating budget of deployed LLM systems, tools that systematically search engine configuration spaces, rather than relying on engineer intuition or vendor defaults, are likely to become a standard part of the deployment pipeline.
6. Conclusion
This analysis demonstrates that an agent combining Optuna-driven Bayesian optimization with multi-regime evaluation can locate vLLM serving configurations that measurably outperform default recipes, yielding a 17.5% token-generation improvement for GPT-OSS 120B and a doubling of throughput with nearly halved ITL for DeepSeek R1 Distill. The practical takeaway is that default serving configurations should not be assumed near-optimal, and that a background-run, sample-efficient search process can recover substantial performance gains with modest trial budgets. Teams deploying LLMs at scale should consider incorporating automated recipe tuning - evaluated across representative workload regimes rather than a single benchmark - as a standard pre-deployment step, with further work needed to establish how tuned configurations transfer across hardware generations and model families.
Sources
- Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean - Original Creator (YouTube)
- Analysis and summary by Sean Weldon using AI-assisted research tools
About the Author
Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.
Sean Weldon builds apps and custom software at 22Atlas.