'"My name is... my name is...": A Linguistic Map for Voice Agents - Midam Kim, ServiceNow'

Voice AI failures can be systematically diagnosed and addressed through a linguistic framework that treats human-bot communication as a joint activity requir...

By Sean Weldon

"My name is... my name is...": A Linguistic Map for Voice Agents

Abstract

Voice-based conversational agents commonly fail through cascading breakdowns that are diagnosed as isolated engineering defects rather than systemic communication failures. This synthesis, drawing on work presented by Midam Kim of ServiceNow, argues that voice AI failures are better understood as breakdowns in linguistic orchestration - the coordinated alignment of sound, word, interaction, and mental model layers that humans manage automatically as part of joint activity communication. A two-channel, four-component diagnostic framework is presented and applied to a representative failure trajectory in which a single mispronounced name cascades into escalation to a human agent. Because transient linguistic signals vanish while the user's mental model persists and accumulates, the mental model is identified as the primary optimization target. Practical interventions, benchmarking approaches including ServiceNow's Eva Bench, and longitudinal adaptation challenges are discussed, with implications for both technical design and business outcomes.

1. Introduction

Voice AI systems are increasingly deployed in customer-facing workflows, yet the diagnostic vocabulary used to explain their failures has not kept pace with the complexity of the interactions they attempt to support. When a voice agent underperforms, post-mortems typically isolate a single culprit - a misrecognized phoneme, a synthesis error, an ill-timed interruption - and remediation proceeds component-by-component. This approach treats communication as a pipeline of independent transformations, which is analytically convenient but linguistically incomplete, since it obscures how failures in one layer of communication propagate into others.

The central thesis of this analysis is that voice AI failures can be systematically diagnosed and addressed through a linguistic framework that treats human-bot communication as a joint activity requiring alignment across four interdependent layers: sound, word, interaction, and mental model. Under this view, no single component is independently sufficient for success; alignment must be achieved simultaneously across all four, in both the listening and speaking directions of the exchange.

Several terms are defined at the outset. ASR (automatic speech recognition) and STT (speech-to-text) refer to the transcription of user speech into text. TTS (text-to-speech) refers to the synthesis of agent speech from text. Turn detection refers to the mechanism by which an agent determines whether a user has finished speaking. A mental model, in this context, denotes the user's evolving internal representation of what the agent knows, intends, and is capable of doing.

This synthesis proceeds by first establishing the theoretical grounding in joint-activity accounts of communication, then presenting the four-component framework and applying it to a documented failure trajectory. It subsequently translates the framework into concrete technical interventions, discusses business implications and diagnostic benchmarking, and closes with longitudinal considerations regarding adaptation and linguistic drift.

2. Background and Related Work

The theoretical foundation for this framework rests on treating human communication as a joint activity rather than as unilateral message transmission. Under this account, conversation is a continuous back-and-forth exchange of sounds and words in which all participants concurrently process incoming signal and update their internal models of one another. As stated in the source material, "human communication is a joint activity, the thing that we're doing right now." Meaning is not delivered intact from speaker to listener; it is jointly constructed, monitored, and repaired in real time.

This framing carries a direct design implication for voice AI. Because humans have evolved as communicators over thousands of years, the conversational norms following from joint activity - timely turn-taking, cooperative repair strategies, inference of unstated intent - constitute deeply entrenched expectations. Users do not suspend these expectations when speaking to a machine; they apply the same evaluative criteria to a bot that they would apply to a human interlocutor. This establishes the intellectual context for the four-component linguistic framework developed in Section 3, and explains why isolated component-level fixes are insufficient: they address the mechanics of communication without addressing its joint, expectation-laden character.

3. Core Analysis

3.1 The Two-Channel, Four-Component Framework

The framework organizes voice agent behavior into two channels - listening and speaking - each decomposed into four components: sounds, words, interaction, and mental model. On the listening channel, the relevant diagnostic questions are whether the bot recognizes speech accurately (sounds), understands the words produced (words), waits for appropriate timing (interaction), and correctly infers user intention (mental model). On the speaking channel, the parallel questions concern correct pronunciation (sounds), selection of understandable vocabulary (words), appropriately timed responses (interaction), and provision of the information the user actually needs (mental model).

Critically, these components are interdependent rather than modular. A failure in one component propagates into the others, meaning that all four must be aligned simultaneously for the interaction to succeed. This interdependency distinguishes voice interaction from text-based chat: whereas chat interfaces preserve a persistent transcript, spoken sounds, words, and interactional turns vanish the moment they are produced. Only the user's mental model persists and accumulates across the call timeline. As the source material states, "sounds, words, interactions vanish the moment they're spoken, but the mental model proceeds and grows over the timeline." This asymmetry identifies the mental model as the ultimate target for user satisfaction, since it is the only component that outlives the moment of utterance.

3.2 Cascading Failure: A Case Analysis

The documented failure trajectory illustrates how a single low-level error cascades through every layer of the framework. The interaction began with an ASR failure at the sound level: the bot misheard "M" versus "N" while the user was spelling the name "Midam." This sound-level error propagated to the word level when the TTS system applied English-centric reading rules to a non-English name, rendering "Midam" as "Madam" - prompting the user's corrective utterance, "My name is Midam, not Madam."

The breakdown then extended into the interaction layer. The bot requested an "account number" without adequate framing, leaving the user uncertain about the request's purpose - a mental-model failure on the speaking channel, since the bot failed to provide information the user needed to proceed. When the user attempted to supply a spoken alphanumeric account number, STT failed to parse the string correctly, causing the bot to cut the user off mid-turn - an interaction-layer failure compounding the earlier word-level error. Rather than attempting a clarification strategy analogous to human conversational repair, the bot simply asked the user to repeat the information, demonstrating a failure to track or update its mental model of the exchange. The cumulative effect of these compounding failures was user frustration sufficient to trigger escalation to a human agent, illustrating that each layer's failure amplifies rather than merely adds to prior failures.

4. Technical Insights

Several actionable technical findings follow from this analysis. First, ASR/STT model selection and configuration, combined with post-processing, directly determine sound- and word-level accuracy; the M/N confusion and alphanumeric parsing failure both originate at this layer and require targeted post-processing strategies for names and structured strings such as account numbers. Second, TTS model selection and pre-processing must account for non-English or non-standard names, since default English-centric grapheme-to-phoneme rules produce systematic mispronunciations, as demonstrated by the "Midam"/"Madam" case. Third, curating a shared vocabulary between bot and user reduces word-level misunderstandings by aligning terminology expectations on both sides of the exchange.

Fourth, turn-detection, latency, and turn-taking mechanisms require deliberate engineering attention, as failures here directly caused the bot to cut off the user mid-utterance - a failure mode with no analog in text-based systems. Fifth, emotion detection and handling constitute a necessary capability for tracking user state as frustration accumulates across turns. Sixth, dynamic context retention across the call - adapting to different user types (e.g., children) and emotional states - is necessary to maintain an accurate mental model of the user's evolving needs. A key trade-off is that improvements at the sound and word layers alone are insufficient if interaction-layer and mental-model-layer mechanisms, such as clarification strategies, remain unimplemented; the case analysis shows that the bot's failure to attempt clarification, rather than any single recognition error, was the proximate cause of escalation.

5. Discussion

The equivalence asserted in the source material - that "business implications are linguistic implications and vice versa in this voice AI scene" - merits synthesis. Poor linguistic orchestration produces measurable business consequences: user frustration, task failure, escalation to live agents, call abandonment, and silent failures that evade detection entirely. This equivalence implies that voice AI quality assurance cannot be separated from linguistic diagnosis, motivating the development of end-to-end benchmarks such as ServiceNow's Eva Bench, which is designed to diagnose voice agent status across the full framework rather than at isolated component level.

A significant gap concerns longitudinal adaptation. Human speakers naturally adjust to conversational partners over repeated interactions, raising the open question of whether voice agent systems are architected to adapt as users adapt to them over time. A related and largely unaddressed challenge is that language itself changes on the timescale of six months to a year, raising the question of whether voice agents - and the models and vocabularies underlying them - can keep pace with linguistic drift. These questions point toward future work in continual learning and dynamic vocabulary curation for deployed voice systems, extending beyond static benchmarking toward ongoing adaptation.

6. Conclusion

This synthesis has presented a linguistic framework that reframes voice AI failures as breakdowns in orchestration across sound, word, interaction, and mental model layers, rather than as isolated engineering defects. The case analysis demonstrates that a single phoneme-level error can cascade through recognition, synthesis, turn-taking, and cognition to produce complete task failure and escalation. Because transient signals vanish while the user's mental model persists, that model is identified as the primary target for optimization and evaluation.

The practical takeaway for AI researchers and engineers is that component-level metrics - ASR word error rate, TTS naturalness scores, latency figures - are necessary but insufficient; systems must be evaluated for cross-layer alignment using end-to-end benchmarks such as Eva Bench. Future work should extend this framework toward longitudinal adaptation, addressing both user-specific adaptation over repeated interactions and broader linguistic drift over time.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub