5 Voice Agent Failure Modes You'll Hit in Week One - Venky B, Plivo

'Voice AI agents that work well in demos often fail in production due to five key failure modes: latency, brittle transcription, poor data collection, unnatur...'

By Sean Weldon

5 Voice Agent Failure Modes You'll Hit in Week One

Abstract

Voice AI agents frequently exhibit strong demonstration performance yet degrade substantially in production. This synthesis identifies five recurrent failure modes in production voice agent systems: end-to-end latency, brittle speech-to-text (STT) transcription, unstructured data collection, unnormalized text-to-speech (TTS) input, and turn detection with barge-in handling. Each mode is examined against empirical latency and accuracy thresholds, underlying causal mechanisms, and targeted engineering mitigations. Principal findings include the necessity of disabling LLM reasoning modes to meet sub-550ms time-to-first-audio targets, the superiority of dynamic over static keyword boosting for transcription accuracy, and a documented improvement in data collection accuracy from 30% to 95% when voice inputs are modeled as typed schema fields rather than parsed from raw transcripts. The analysis suggests production reliability derives from layer-specific engineering interventions rather than from improved orchestration frameworks alone, with direct implications for teams building conversational voice systems at scale.

1. Introduction

Voice AI agents - systems combining speech recognition, language modeling, and speech synthesis into a real-time dialogue loop - have become a widely deployed application category. The characteristic difficulty is not construction but deployment: systems perform adequately in controlled development environments and degrade markedly once exposed to production traffic. As one practitioner observation states, agents "sound great when you're building that in your dev landscape and then the moment you take this from a proof of concept to production, things start failing."

This synthesis advances the thesis that the dev-to-production gap results not from deficient orchestration frameworks but from five distinct, layer-specific engineering problems distributed across the pipeline. Each requires targeted intervention rather than generic architectural improvement. The relevant terminology includes time to first audio (TTFA), the latency between user speech termination and agent audio onset, and time to first token (TTFT), the LLM-layer latency component that lower-bounds TTFA.

The analysis proceeds through latency and model selection, transcription brittleness, structured data collection, TTS normalization, and turn management, concluding with consolidated technical findings and broader implications for the field.

2. Background and Related Work

The standard voice agent architecture is a cascaded pipeline: an STT engine transcribes audio, the resulting text conditions an LLM, and the LLM's output is synthesized by a TTS engine. Each stage introduces additive latency and independent error modes; errors compound downstream, such that an STT substitution error can propagate into an LLM reasoning error and ultimately an incorrect spoken response.

An alternative architecture, speech-to-speech models, maps audio directly to audio without an intermediate text representation and is commonly presumed necessary for naturalistic interruption and backchannel handling. Evidence discussed in Section 3.5 suggests this presumption is not well founded. Separately, the structured data collection problem draws conceptual precedent from typed schema validation frameworks in software engineering - Python dataclasses, Pydantic, Zod, and TypeScript form field definitions - which establish field shape and constraints prior to data ingestion, inverting the conventional pattern in which structure is inferred post hoc from unstructured transcripts.

3. Core Analysis

3.1 Latency and Model Selection

TTFA is the primary user-facing metric governing perceived conversational quality. Empirically, latency below 550ms is perceived as natural; the 750ms-1.2s range is tolerated but degraded; beyond 1.2s, users disengage. Most production agents fall into the degraded band despite advertised sub-550ms targets.

A dominant contributor is LLM inference latency. Frontier models (OpenAI, Claude, Gemini) exhibit P50 TTFT of 450-500ms, with P90/P95 spiking to 1.2-1.3s - a tail latency profile incompatible with consistent sub-550ms targets. Reasoning or "thinking" modes, despite representing substantial recent advances in LLM capability, must be disabled entirely for voice use, as one practitioner notes: "all the advancements we've had in the LLM layer in the last one year, none of that even apply here."

Two mitigation paths exist. Dedicated capacity providers (Groq, Cerebras) offer fast token generation but require advance capacity booking of up to twelve months and carry substantial cost. Alternatively, self-hosted open-source models - Qwen 3.5 and Gemma 4 - can consistently achieve sub-300ms latency on owned GPU infrastructure, balancing cost, intelligence, and speed. Gemma 4 demonstrates 2.5-3x better multilingual token fertility than Qwen 3.5, reducing tokens required per word. For model sizing, 3-4B parameter Mixture of Experts (MOE) models achieve approximately 90% baseline performance out-of-box without fine-tuning, though MOE architectures are comparatively fragile under fine-tuning; domain specialization requires a minimum of 8-12B parameters. Some production systems employ heterogeneous model pairs - a smaller model for conversational flow and a larger model for tool-calling accuracy.

3.2 Transcription Brittleness

State-of-the-art STT engines report 4-6% word error rate on clean evaluation sets, but real-world noisy calls with accented speech produce double-digit error rates. Recurring failure patterns include proper nouns, domain jargon, phone numbers (missing or substituted digits), and addresses. Code-switched languages - English rendered in Hindi script, or Hindi rendered in Latin script - degrade performance across the STT, LLM, and TTS layers simultaneously.

Mitigations include dynamic keyword boosting, which conditions boosted terms on current call state rather than applying a static persistent list, improving accuracy while reducing hallucination risk. LLM-based post-processing corrects domain-specific transcription errors using contextual inference (e.g., inferring that a transcribed "E" in a phone number sequence should be "3"). A transliteration/normalization layer is recommended to produce consistent, cleaned transcripts for the LLM regardless of underlying STT engine, decoupling downstream logic from any single vendor's output idiosyncrasies. As a general design principle: "assume your transcriptions are going to be brittle."

3.3 Data Collection as a Structured UX Problem

Voice data collection is reframed as a schema definition problem rather than a transcript-parsing problem, drawing on typed field conventions from Pydantic, Zod, and TypeScript. This reframing - defining field shape and type (e.g., phone number, datetime) prior to prompting the user - enables validation and targeted correction logic unavailable when parsing occurs after the fact. This approach reportedly improved data collection accuracy from 30% to 95%.

Specific mechanisms include explicit pronunciation and spell-by-letter confirmation rules for difficult names, and datetime-typed fields combined with LLM tool-calling to resolve relative or ambiguous temporal expressions (e.g., "next week Wednesday 8," requiring am/pm disambiguation). Evaluation methodology shifts accordingly: field-level unit tests, rather than end-to-end agent test cases alone, are recommended to isolate failure sources. Notably, 95-97% accuracy was achieved without fine-tuning, through structured context and state design rather than prompt engineering alone.

3.4 TTS Output Normalization

Raw LLM output should never be routed directly to a TTS engine; a normalization layer is required as an intermediate stage. This includes stripping emojis and markdown (partially supported via flags in frameworks such as LiveKit and Pipecat), applying custom pronunciation dictionaries for proper nouns and brand terms, and reducing speech rate to 0.7x-0.8x when synthesizing entities like emails, phone numbers, and names. Normalizing messy data formats (emails, currency, dates) in-house, rather than relying on TTS engine defaults, preserves TTS-engine independence and consistency across vendor swaps. A proposed baseline diagnostic: an agent's ability to correctly pronounce a speaker's proper name and company name - "my first test is if it cannot pronounce my last name or my company's name, it's already dropping the ball."

3.5 Turn Detection, Barge-in, and Backchanneling

This dimension is treated more briefly in the source material, but one finding is significant: speech-to-speech models are commonly assumed necessary to support natural barge-in and backchannel behavior, yet this capability can be achieved within cascaded (non-end-to-end speech-to-speech) pipelines, challenging a common architectural assumption in the field.

4. Technical Insights

Several implementation-level conclusions follow from the analysis. First, latency budgets should be treated as a hard constraint governing model selection, favoring self-hosted open-source models over frontier APIs when tail latency (P90/P95) matters more than peak capability. Second, MOE models under 4B parameters are suitable defaults absent fine-tuning needs, but teams requiring domain specialization should budget for 8-12B parameter models and anticipate fragility during fine-tuning. Third, transcription pipelines should incorporate dynamic (state-conditioned) keyword boosting and LLM-based post-processing rather than relying on raw STT output. Fourth, data collection architecture should precede prompt design: defining field types and validation rules first materially outperforms post hoc parsing. Fifth, TTS input requires a dedicated normalization layer independent of any specific synthesis engine, incorporating rate control and custom pronunciation handling.

5. Discussion

The findings collectively suggest that voice agent reliability is an engineering property distributed across discrete pipeline layers, not an emergent property of orchestration frameworks. This has organizational implications: teams should allocate engineering effort proportionally across STT correction, structured data schemas, TTS normalization, and model serving infrastructure, rather than concentrating investment in a single orchestration layer or a single "better model."

A notable tension emerges between the industry trend toward more capable reasoning models and the voice-specific requirement to disable such reasoning entirely for latency compliance. This suggests that voice agent development may remain partially decoupled from general LLM capability improvements, with self-hosted smaller models retaining a durable advantage in this domain. The finding regarding speech-to-speech models' non-necessity for barge-in handling likewise suggests that cascaded architectures remain viable and potentially preferable given their modularity and debuggability, an area meriting further empirical investigation.

6. Conclusion

This synthesis identifies five distinct, empirically grounded failure modes affecting production voice agents and demonstrates that each is addressable through targeted, layer-specific engineering rather than generic framework improvements. The practical takeaway is that production-grade voice agents require deliberate investment in latency-optimized model selection, robust transcription correction, schema-driven data collection, dedicated TTS normalization, and careful turn-management design. Teams building conversational voice systems should treat these as independent engineering workstreams, each with measurable success criteria, rather than assuming general-purpose orchestration tooling will resolve them by default.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub