How to Build Voice Agents That Don't Fall Apart When You Interrupt with Jesse Hall, LiveKit

Building human-like real-time voice agents for financial services and other high-stakes use cases requires evaluating the entire conversational system -- not

Abstract

This synthesis examines the engineering and evaluation of real-time voice agents for high-stakes domains such as financial services, where mid-conversation corrections to payment amounts or dates carry material risk. It argues that component-level leaderboards are structurally inadequate for voice systems because errors cascade across the speech-to-text, language-model, and text-to-speech pipeline, and because reported latency metrics - time to first byte and time to first token - do not correspond to what a caller actually hears. Three corrective practices are analyzed: measuring time to first audio and time to first sentence, decoupling backend tool execution from the conversation thread via asynchronous invocation, and performing turn detection on audio prosody rather than transcript text alone. End-to-end simulated-caller benchmarking is presented as the validating methodology, with findings drawn from LiveKit's production experience, including a measured turn-detection improvement from 9.9% to 4.5% false interruption rate.

1. Introduction

Voice is plausibly the most natural human interface; as the source material observes, "we learn to speak before we learn to read." Yet the engineering requirements of a spoken agent differ qualitatively from those of a text or background assistant. Most deployed AI agents - coding assistants, research agents - tolerate latency on the order of seconds or minutes because their output is consumed asynchronously. A voice agent, by contrast, operates in a real-time conversational channel with effectively zero tolerance for awkward delay. As stated directly in the source: "A voice agent isn't just a chatbot with a speaker attached. It's a real-time system that needs to respond to a person."

The dominant failure mode is conversational collapse. Callers interrupt, change their minds, talk over the agent, or mumble; when the agent cannot keep pace, the interaction degrades and the caller presses zero to reach a human operator. In financial services the consequences are sharper: a mid-conversation revision to a payment amount or due date must be absorbed correctly, or the agent commits an incorrect transaction. This raises the practical question of how such systems should be built and, critically, how they should be evaluated prior to deployment.

The central thesis advanced here is that building human-like voice agents requires evaluating the complete conversational system rather than its constituent models, against latency metrics defined at the point of audible output, with explicit mechanisms for multitasking during slow backend operations and for audio-aware turn-taking - all validated by simulated end-to-end conversations tailored to the specific use case. Section 2 establishes architectural background; Section 3 analyzes the principal findings; Section 4 consolidates implementation-level detail; Sections 5 and 6 discuss implications and takeaways.

2. Background and Related Work

Two architectural families currently dominate production voice systems. The cascaded pipeline composes three models that must operate in unison: speech-to-text (STT) transcription, a large language model (LLM) for reasoning and planning, and text-to-speech (TTS) synthesis. Each stage contributes both latency and potential transcription error, and these errors are not easily caught downstream. Speech-to-speech real-time models collapse this pipeline into a single model operating directly on audio, removing inter-stage transcription loss, but are described as not yet mature enough to serve every use case, leaving the cascaded design the pragmatic default for many deployments.

LiveKit is cited as an open-source real-time infrastructure framework for agents that can hear, see, and speak, and it underpins ChatGPT Advanced Voice Mode. The framework extends beyond voice to video, arbitrary data streams, and physical AI/robotics applications, situating voice agents as one instance of a broader real-time agent class. A practical consequence of this ecosystem is selection complexity: hundreds of candidate models are offered by dozens of providers across the three pipeline stages. This combinatorial space is precisely what leaderboards are commonly used - and, it is argued, misused - to navigate.

3. Core Analysis

3.1 The Structural Inadequacy of Component Leaderboards

Leaderboards score individual models in isolation, not the composed conversation that is actually shipped to users. This creates two distinct failure modes. First, errors cascade across pipeline stages with no mechanism for downstream correction: if a caller says "pay on the 3rd" and STT mishears "30th," the LLM plans around the wrong date with no way to catch the error. Second, leaderboards typically evaluate models on clean studio audio, whereas real callers use speakerphones in traffic with background noise - conditions under which component rankings may not transfer. The source material reframes the design question accordingly: the relevant question is not "which model should I pick" but "which stack should I ship for my use case."

3.2 Correcting Latency Metrics: From Bytes and Tokens to Audio and Sentences

A recurring finding is that commonly reported latency metrics misrepresent caller experience. Some popular TTS providers report approximately 210ms time to first byte, but exhibit up to three-quarters of a second of silence after that first byte is emitted - as the source states plainly, "a byte does not equal audio." The corrected metric is time to first audio: the interval until the caller actually hears something.

An analogous distortion applies to LLMs. Time to first token is a text-level metric, but TTS models require a complete sentence, not a single token, before synthesis can begin. The fastest token-producing LLM is therefore not necessarily the fastest to produce a usable first sentence, and time to first sentence is identified as the correct metric for LLM evaluation in this pipeline.

3.3 Multitasking During Backend Latency

Backend operations - transaction lookups, payment status checks, dispute submissions - can take anywhere from 2 to over 40 seconds, and this latency cannot be eliminated, only managed. The source draws a direct human analogy: a person performing a lookup would not go silent but would say "give me a second" or ask another question while waiting. Voice agents should replicate this behavior.

The technical mechanism enabling this is async/await on tool calls, which decouples the conversation thread from the tool-execution thread, allowing the agent to continue speaking while a backend call remains in flight. A demonstrated hotel-booking agent showed no awkward pauses even under interruption, with tool calls saving data in the background without blocking dialogue. Where a tool call errors, the agent is expected to self-correct or escalate to a human rather than breaking down conversationally.

3.4 Turn Detection as a Second Latency Axis

Beyond raw response speed, correctly identifying when to speak versus listen is identified as a second critical evaluation axis. LiveKit's turn detector analyzes audio rhythm, pitch, and intonation rather than relying on transcript text alone. In a published benchmark on a hotel-agent test case, a leading alternative turn detector interrupted or cut off users 9.9% of the time, compared with 4.5% for LiveKit's detector. The illustrative case is a caller saying "pay $500 and 55 cents" with a pause after "500": a transcript-only detector may wrongly infer the utterance is complete, while an inflection-aware detector waits for the full statement.

This connects to a broader framing of the latency budget: the gap between a caller's last syllable and the agent's first audible response, which encompasses turn detection, LLM response preparation, and TTS generation together. Conversations are reported to remain coherent within roughly 1.5 seconds of gap, but begin to feel genuinely human-like only at 500-600ms. The governing design question is reframed accordingly: not "how fast can I go," but, as quoted directly, "how human-like of an agent can I fit within this latency window."

4. Technical Insights

Several implementation-level findings follow from the above analysis. The cascaded pipeline's three-stage structure (STT, LLM, TTS) means latency and error must be measured and mitigated at each boundary, not only in aggregate. The 210ms-byte-versus-750ms-silence disparity in TTS providers indicates that provider-reported metrics require independent verification against actual audio onset. The async/await decoupling pattern is a low-level but consequential architectural choice: it determines whether a 2-40 second backend call manifests as dead air or as managed, human-like conversational filler. Turn detection incorporating prosodic features (rhythm, pitch, intonation) represents a meaningful deviation from transcript-only turn-taking heuristics common in text-first chat architectures, and the 9.9%-to-4.5% interruption-rate reduction is a concrete, reproducible benchmark result rather than an anecdotal claim.

A further insight concerns benchmarking methodology itself: models should be swapped in and out against identical, byte-for-byte simulated input, producing a personalized benchmark per agent rather than a general-purpose leaderboard. One reported result - a Gemma 4 31B model ranking highest on LiveKit inference for a hotel-agent benchmark, while the same model was slower on a different inference provider - demonstrates that inference-layer optimization for voice workloads is separable from model choice, and that framework-level infrastructure remains model-agnostic while still materially affecting latency outcomes.

5. Discussion

The findings collectively suggest that voice agent evaluation is diverging from standard model-evaluation practice. Whereas text-based LLM evaluation can often rely on shared, static benchmarks, voice agents require conversation-level, use-case-specific simulation because the unit being shipped is not a model but an entire interactive system with compounding error pathways and conversational dynamics. The source material captures this directly: "there are as many benchmarks now as there are agents." This implies a shift in evaluation labor - from consulting a leaderboard to constructing and maintaining a bespoke simulated-caller test suite.

An open question is how this approach generalizes across domains with different risk profiles. Financial services provides a clear high-stakes case given exposure to incorrect transactions, but the same methodology - simulated callers pursuing goals, changing their minds, interrupting, mumbling, with added background noise - is presented as applicable more broadly. The described scoring approach, which checks that correct steps were taken (not merely that the task was completed), is notable for catching hallucinated function calls or unnecessary steps, a failure mode that final-outcome-only evaluation would miss. This points toward a broader industry trend of process-level, not just outcome-level, agent evaluation.

6. Conclusion

The central contribution of this analysis is a reframing of voice agent evaluation away from component leaderboards and toward complete-conversation, use-case-specific benchmarking, grounded in metrics that reflect what a caller actually experiences. Time to first audio and time to first sentence replace misleading byte- and token-level proxies; asynchronous tool execution replaces silent backend waits; and prosody-aware turn detection replaces transcript-only heuristics.

The practical takeaway is a checklist-level discipline: define the latency budget around what is actually heard, confirm critical financial details explicitly, verify identity before account access, keep the conversation alive during slow tool calls, and run simulated conversations repeatedly, converting discovered failures into regression scenarios. For practitioners building voice agents in regulated or high-stakes domains, this suggests that evaluation infrastructure - simulated callers, audio-aware scoring, personalized model comparison - is not a peripheral testing concern but a core design requirement alongside model selection itself.

Sources

• How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub