Realtime Voice Agents with Frontier Intelligence - Bohan Li, EliseAI

Real-time voice agents can achieve Frontier-level intelligence without sacrificing speed by architecting a cascaded system - modeled after self-driving car per...

By Sean Weldon

Realtime Voice Agents with Frontier Intelligence: A Cascaded Architecture Analysis

Abstract

Real-time voice agents face a structural tension between reasoning quality and conversational responsiveness: additional inference stages that improve intelligence also introduce perceptible latency. This synthesis examines a cascaded voice-agent architecture developed by EliseAI that resolves this tension through speculation and caching rather than model compression or single-model end-to-end replacement. The architecture is organized around an explicit analogy to autonomous vehicle software stacks - perception, planning, and controls - mapped respectively onto transcription, language modeling, and text-to-speech. Three latency-hiding mechanisms are analyzed: a streaming speculative transcriber pairing a fast model (Flux) with a slower corrective model (Scribe V2), background tool-calling agents that eliminate visible inference round trips, and a prefix cache for text-to-speech that begins audio playback before generation completes. Findings indicate that substantial conversational naturalness is recoverable through harness-level engineering around frontier models, independent of underlying model architecture improvements, with direct implications for production voice-agent design.

1. Introduction

Conversational voice agents operate under a latency budget that text-based agents do not share. Human turn-taking conventions tolerate only a few hundred milliseconds of silence before a pause registers as unnatural, yet frontier-level reasoning - particularly reasoning involving multi-step tool use - typically requires several sequential model invocations per conversational turn. The central question this synthesis addresses is whether Frontier-level intelligence, defined here as reasoning quality comparable to the largest available language models including tool-augmented reasoning, can be delivered within real-time conversational constraints.

The thesis under examination, presented by Bohan Li of EliseAI, holds that this is achievable not by replacing the cascade with a single end-to-end speech model, but by aggressively optimizing each stage of a cascaded architecture - a pipeline in which audio is transcribed to text, processed by a language model, and re-synthesized as audio. The proposed optimizations are speculative and cache-based in nature: each stage begins provisional work before its input is finalized, discarding that work when later information invalidates it.

Two terms merit precise definition. Speculative execution, in this context, denotes initiating downstream computation on provisional inputs and canceling/regenerating that computation if the input is subsequently revised. Prefix caching denotes the reuse of previously computed outputs for a matching leading sequence of tokens or words. This analysis proceeds by first establishing the architectural framing (Section 2), then examining the three stage-specific optimizations in turn (Section 3), consolidating technical findings and trade-offs (Section 4), and discussing broader implications (Sections 5-6).

2. Background and Related Work

The organizing framework is an explicit analogy to the conventional software stack of self-driving vehicles, decomposed into perception, planning, and controls. Perception corresponds to transcription: raw audio is converted into structured text, analogous to converting raw sensor returns into bounding boxes or lidar point clouds. Planning corresponds to the language model, which consumes the perceptual representation and produces an intended output - the analogue of a driving trajectory. Controls corresponds to text-to-speech, converting planned textual output into an actuated audio signal, analogous to converting a trajectory into steering and throttle commands.

As stated in the source material: "Cascaded voice agents makes a lot of sense when you view it in lens of breaking it down into perception, planning, and controls." This framing is diagnostic rather than merely descriptive - it localizes latency sources to specific pipeline stages and clarifies that each stage exhibits distinct failure modes requiring distinct mitigation strategies, rather than a single global fix.

3. Core Analysis

3.1 Perception: The Streaming Speculative Transcriber

The transcription stage employs two models operating in tandem. Flux, a fast streaming transcriber, produces low-latency but occasionally imprecise text. Scribe V2, a slower batch transcriber, consumes greater context - including prior question context, such as knowledge that a question requested a name and date of birth - to correct transcription errors. Critically, the corrective layer fires only when its output disagrees with the streaming detection; when the two agree, the corrective computation is canceled, conserving compute. Final text is released to the downstream agent only after corrections settle, and punctuation-only revisions are ignored to avoid unnecessary regeneration. This design allows the system to benefit from Scribe V2's contextual accuracy without paying its latency cost on every utterance.

3.2 Planning: Reducing Language Model Round Trips

Within the language model layer, tool calling is identified as the dominant source of additional inference round trips. The architecture addresses this via background agents that execute tool calls invisibly and inject results directly into the main agent's context, such that "it appears to have made the call itself." As the source states: "One way to get rid of that is by having background agents do the tool calling for you and push the tools back into the context of the main agent so that it thinks it made the tool call, but it really didn't."

Separately, each partial transcription received from the perception stage triggers an early, speculative generation from the main agent before the user's utterance is confirmed complete. These speculative generations are canceled and regenerated when new information - for instance, a corrected name or date of birth surfaced by the background agent - arrives. The background agent additionally performs phonetic matching to correct likely mis-transcriptions of proper names, compensating for a known weakness of streaming transcription in handling low-frequency tokens.

3.3 Controls: Prefix Caching in Text-to-Speech

The text-to-speech stage aims to generate audio before the agent finishes producing the corresponding text, effectively hiding synthesis latency behind generation latency. This is achieved through a prefix cache that checks whether audio for a given sequence of words has already been generated in a prior or concurrent generation: "The prefix cache is going to be looking at the agent stream, and seeing if we already have generated audio for that sequence of words from a prior generation." To avoid excessive cache hits on generic conversational phrases, the cache withholds judgment until a threshold - approximately three words - has accumulated.

Cached audio is streamed via WebSocket to the TTS provider Cartesia alongside live generation. When a cache miss occurs, typically triggered by a unique name or unusual phrase, the system plays the cached prefix while streaming the remainder from Cartesia. Notably, Cartesia generates the full sentence with natural prosody without awareness of the cache's existence; the system suppresses the already-played portion and emits only the remaining audio, preserving prosodic continuity while eliminating redundant latency.

4. Technical Insights

Several implementation-level considerations emerge from this architecture:

A limitation implicit in this design is that all three optimizations depend on the ability to cancel and discard speculative or cached computation cheaply; systems without inexpensive cancellation semantics may not realize equivalent latency benefits.

5. Discussion

These findings suggest that meaningful gains in perceived conversational responsiveness can be achieved independently of underlying model capability improvements. Rather than waiting for lower-latency frontier models, the harness surrounding existing models is engineered to hide latency through speculation, caching, and asynchronous execution. This has implications for teams building on top of proprietary frontier models, where architecture-level latency reduction is often the only lever available since model internals are not modifiable.

A notable gap is the absence of quantitative latency figures in the source material; while mechanisms are described qualitatively, the magnitude of latency reduction attributable to each optimization (transcription correction, background tool calling, prefix caching) is not specified, limiting direct comparability with alternative architectures such as end-to-end speech-to-speech models. Future investigation might quantify the trade-off between speculative regeneration overhead and perceived latency reduction, as well as measure prefix cache hit rates in production dialogue distributions.

This work also reflects a broader industry trend of favoring cascaded, modular pipelines over end-to-end speech models in production settings, prioritizing debuggability and stage-specific optimization over architectural elegance.

6. Conclusion

This analysis has examined a cascaded voice-agent architecture that achieves frontier-level intelligence within real-time constraints through three coordinated mechanisms: a dual-model speculative transcriber, background tool-calling agents that eliminate visible round trips, and a prefix-caching text-to-speech layer. The perception-planning-controls analogy to autonomous vehicles provides a useful diagnostic lens for localizing and addressing latency at each pipeline stage independently.

The practical takeaway is that conversational naturalness in voice agents is substantially a harness engineering problem, not solely a model capability problem. As stated in the source material, "there's a lot of effort that can be done in the harness to really get a natural conversation." Teams building production voice agents may find greater near-term returns in optimizing speculative execution and caching around existing frontier models than in awaiting architectural advances in the underlying models themselves.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub