How a Trading Firm Ships New AI Models the Same Day with Noyan Tokgozoglu, Optiver

Optiver built a highly modular, on-prem AI platform - spanning an inference gateway, agent platform, agent gateway, and user-facing surfaces - designed to let a

How a Trading Firm Ships New AI Models the Same Day: Lessons from Optiver's On-Prem Agent Platform

Abstract

This synthesis examines the architecture and operating principles of an on-premises artificial intelligence (AI) platform built by Optiver, a proprietary market-making firm, and derives generalizable principles for deploying large language model (LLM) agents in environments with immediate, quantifiable feedback. The platform is organized as a four-layer modular stack - an inference gateway, an agent platform, an agent gateway, and user-facing surfaces - built to enable same-day adoption of newly released frontier models, contingent on internal benchmarking. Findings show that unmanaged self-service provisioning produced default session contexts exceeding 200,000 tokens, that spend visibility combined with self-service alerting was sufficient to achieve decentralized cost control, and that narrowly scoped agents communicating through a shared gateway outperform monolithic agents on measurability and maintainability. Evaluation of tool calls remains an open problem due to dependence on world state at invocation time. Practical implications extend to any organization operating LLM agents at scale under tight latency-to-impact constraints.

1. Introduction

Most enterprise deployments of LLMs measure impact over months or quarters, typically through engagement proxies such as adoption rate or retention. Market making inverts this relationship entirely. Optiver trades its own capital with no external clients, executing more than 10 million trades per day, pricing over 1 million instruments, and processing petabytes of data daily across 10 or more global office locations. Because the firm's balance sheet is zeroed out at the start of each trading day, the profit-and-loss (P&L) consequence of any shipped change becomes observable by the close of that same day.

This compressed feedback loop has two architectural consequences. First, it raises the measurable value ceiling of any individual engineering decision: a well-timed capability can be attributed to revenue within hours. Second, it converts latency in capability adoption into direct opportunity cost - a missed market event, unlike a missed product feature, cannot be recovered later. The central thesis examined here is that these conditions justify a deliberately modular, vendor-agnostic, on-premises AI platform whose primary engineering goals are rapid model turnover, instrumentation at every layer, and aggressive token economy.

The analysis proceeds in four parts: the layered platform architecture and model adoption process; the cost optimization program that emerged from observed overspend; the user-facing surfaces shaped by a high-agency trader population; and the open evaluation problems surfaced during build-versus-buy decision-making.

2. Background and Related Work

The platform rests on five stated principles: move fast, stay modular, measure every layer, provide self-service with good defaults, and forward-deploy engineers alongside traders. The last principle is organizational rather than technical - AI engineers sit directly with trading desks to observe workflows firsthand, then return generalizable improvements to the shared platform rather than eliciting requirements through intermediated product cycles.

Regulatory obligations and intellectual-property sensitivity constrain the deployment topology. The entire stack, including dedicated graphics processing units (GPUs) for self-hosted open-weight models, runs on-premises. The sole exception is inference against third-party frontier models, governed by zero data retention agreements. This constraint is load-bearing for the architecture that follows: it eliminated most commercially available agent platforms from consideration, which is cited as a primary reason Optiver built its own stack rather than adopting tools such as Codex, since on-prem support for external harnesses only became available recently.

3. Core Analysis

3.1 Layered Platform Architecture and Model Adoption

The stack comprises four layers. The inference gateway serves as the single ingress for all model traffic - third-party and self-hosted - providing routing, cost tracking, budgeting, and failover. The agent platform registers skills, tools, and memory systems available to agents. The agent gateway functions as a single ledger for all agent and human orchestration, including identity management. User-facing surfaces - a web portal, agent workbench, personal assistant, and Slack client - sit atop this stack.

The stated goal is to make new frontier models available in production flows the same day they are released, contingent on passing internally curated golden-set benchmarks. Because public benchmark distributions diverge from Optiver's internal data, the firm partners directly with trading teams to build these benchmarks. Because all model traffic routes through the inference gateway, every interaction is logged and automatically fed into the evaluation system, making the platform agnostic to which model or harness underlies a given workflow. A meta harness layer sits above multiple underlying agent harnesses, letting teams configure their experience independently while connecting to team-specific knowledge bases that auto-harvest knowledge from execution traces over time. Round-the-clock regional support rotations maintain this pipeline given the firm's global footprint.

3.2 Cost Optimization Under Observed Overspend

Spend inflection points were observed coinciding with major releases - a "Claude Code moment" in December and frontier model releases over the summer - before costs were brought back under control. Investigation revealed that fresh sessions sometimes defaulted to over 200,000 tokens of context, a product of unmanaged self-service behavior in which teams dumped Model Context Protocol (MCP) tools and knowledge base content directly into system prompts.

"Sometimes people have 200k tokens worth of context by default minimum, sometimes even more. That's ridiculous, right?"

Remediation combined several mechanisms: MCP tool search to avoid loading unused tool definitions, command-line interface (CLI) wrappers around MCPs in place of direct MCP calls for greater efficiency, tuned and dynamically configured prompt caching parameters, and relocation of static content from system prompts into knowledge bases retrieved on demand. Notably, the organizational fix proved as important as the technical one: giving teams visibility into their own spend, paired with self-service alerting and limits, led teams to self-manage without centralized enforcement.

"Turns out that if you tell people how much they're spending and give them a little bit more visibility, a little bit more alerting, and give them the agency to self-manage, they actually do a pretty good job."

Open-source models were evaluated as cost-competitive alternatives for certain use cases, and model routing - dynamically sending prompts to cheaper models, or pairing a frontier model for planning with a smaller model for implementation - is under active exploration.

3.3 User-Facing Surfaces and Organizational Culture

Traders at Optiver are characterized as high-agency and will build bespoke tooling - custom development environments, ad hoc web interfaces - if the platform does not meet their needs. In response, Optiver built its own web portal with session management, chat windows, and artifacts, rather than relying on third-party harness vendors, to better integrate with internal infrastructure. Traders contribute directly via pull requests, a pattern credited with producing features such as collaborative document editing. The platform is extending toward an "agent multiplayer" capability, allowing rooms that combine a user's agents, other users' agents, and centrally deployed agents, alongside a Slack client for mobile access.

4. Technical Insights

Several implementation-level findings generalize beyond trading:

• Default context bloat is a silent cost driver. Self-service systems without guardrails will accumulate unbounded context (MCP tool lists, knowledge base dumps) into system prompts; this is detectable only through usage auditing, not static design review.
• CLI wrappers around MCPs can reduce token overhead relative to direct MCP invocation, suggesting that protocol-level tool access is not always the most efficient integration path.
• Point-in-time evaluation of tool calls is unresolved. Trading backtest databases can be snapshotted for deterministic replay, but systems such as Slack reflect live, mutable world state that cannot be reliably reconstructed after the fact. The adopted mitigation is to assign confidence values reflecting how well a given tool invocation's world state can be approximated via point-in-time snapshotting.
• Full trace-level ("hotel") logging of every tool call input and output is a prerequisite for feeding an automated evaluation pipeline; without it, the inference gateway's agnosticism to model and harness would not translate into measurable benchmarking.
• Agent scope creep produces technical debt. Expanding a single agent's responsibilities (e.g., a coding agent absorbing support for multiple repositories) by unioning functionality was found to degrade maintainability; the corrective practice is to keep agents narrowly scoped and have them communicate through the agent gateway rather than growing monolithically.

5. Discussion

The Optiver case illustrates a general principle for any organization whose decisions have a short, measurable feedback loop to outcomes: platform investment should prioritize instrumentation and reversibility over feature breadth. The firm's willingness to build rather than buy was driven less by cost than by the risk of vendor lock-in - a prior inference gateway vendor's slow roadmap response is cited explicitly as a motivating failure.

"If our infrastructure is overly invested in one particular provider, we move at the speed that they let us move."

This suggests that as frontier model releases accelerate, the opportunity cost of dependency on a single vendor's integration timeline grows commensurately, independent of industry vertical.

The unresolved evaluation problem - assessing tool calls whose correctness depends on transient world state - points to a broader gap in current agent evaluation methodology, which largely assumes replayable, static test fixtures. As agentic systems increasingly interact with live external systems rather than sandboxed environments, point-in-time confidence scoring (or comparable techniques) is likely to become a necessary, not optional, component of evaluation infrastructure.

6. Conclusion

This analysis has described a four-layer on-premises AI platform designed around the constraint that, in market making, the cost of delayed model adoption and uncontrolled token spend is measurable within a single trading day. The key contributions are organizational as much as architectural: self-service with spend visibility substituted effectively for centralized cost governance, and narrow agent scoping through a shared gateway substituted for monolithic agent design. Practically, teams operating LLM agents outside of trading can apply the same diagnostic discipline - auditing default context size, instrumenting every tool call, and treating vendor dependency as a measurable risk - regardless of whether their feedback loop is measured in hours or in quarters.

Sources

• How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub