Tolan: Voice-First AI Companion - Paula Dozsa, Tolan

Building a voice-first AI companion (Tolin) requires fundamentally rethinking LLM engineering around latency, volatility, and memory rather than treating voi...

By Sean Weldon

Tolan: Voice-First AI Companion - Engineering Analysis

Abstract

Voice-first conversational systems impose engineering constraints that differ qualitatively from text-based chat interfaces. This synthesis examines the architecture of Tolin, a voice-first artificial intelligence (AI) companion with over four million logged hours of conversation, and derives four design principles: designing for conversational volatility, treating latency as a first-class product property, implementing memory as a retrieval system rather than a transcript, and reassembling context per turn rather than resisting contextual drift. Reported evidence indicates that a 500-millisecond latency regression degraded all measured product metrics, that a turn-taking classifier reduced severe early response aborts by more than half at a 60-millisecond cost, and that stakes-based model routing sent approximately one-third of turns to a smaller model with negligible retention impact. A parallel finding concerns internal development practice: an orchestrated fleet of coding agents, wired through the Model Context Protocol (MCP), coincided with a crash-free rate increase from 99.6% to 99.9% and a greater-than-50% reduction in runtime errors.

1. Introduction

Most contemporary large language model (LLM) applications inherit assumptions from text chat: turns are slow, users compose deliberately, and conversational context accumulates in a stable, append-only fashion. Voice-first interaction violates each of these assumptions. Round-trip response budgets fall below two seconds, speech arrives disfluent and fragmented, and users change topic mid-sentence while engaged in other activities such as cooking, walking, or falling asleep.

The central thesis advanced here is that voice cannot be retrofitted onto a text-oriented LLM stack. Instead, the engineering problem must be reformulated around three axes - latency, volatility, and memory - with architectural decisions following from those axes rather than from chat conventions. A secondary thesis holds that agentic AI systems can be applied internally to the production of such products, improving both delivery speed and software reliability.

Key terms established here include conversational volatility (unpredictable interruption, topic-shifting, and disfluency in spoken input), time to first token (TTFT) (the latency interval between request submission and initial model output), and context reassembly (the per-turn reconstruction of the model's input window rather than incremental accumulation). Sections 2 and 3 establish problem context and the four derived principles. Section 4 distills implementation-level trade-offs. Sections 5 and 6 discuss broader implications and conclusions.

2. Background and Related Work

The design space is bounded by a multi-stage inference pipeline in which each stage contributes measurable delay: end-of-utterance detection, speech transcription, TTFT, token generation, text-to-speech (TTS) first byte, and audio playback. Because these stages compose serially, perceived responsiveness is the sum of independently optimizable components - a structure that motivates per-stage instrumentation rather than aggregate benchmarking.

Prior practice in retrieval-augmented generation supplies the conceptual basis for the memory architecture, while the notion of a persona card - a compact, structured representation of character identity - supplies the basis for identity stability across turns. The internal development methodology draws on agentic coding frameworks and MCP-based tool integration, connecting coding agents to issue tracking (Linear), error monitoring (Sentry), and observability (DataDog) systems.

3. Core Analysis

3.1 Conversational Volatility as a Design Target

Production observation indicates that spontaneous speech is structurally noisy. Users interrupt the system, emit minimal acknowledgments such as "yeah," and employ profanity carrying affective signal. A naive normalization pipeline that strips such tokens discards information; the reported approach instead treats them as meaningful input requiring special handling.

"Speech is messy. Most LLM apps assume that you'll have a clean and stable conversation and we have to build for the opposite."

The engineering response reframes the objective from minimizing interruptions outright to minimizing bad early interruptions - cases where the system aborts a response prematurely on a false signal. A smart turn-taking system was built to read speech patterns and distinguish genuine interruptions from false ones, cutting the worst early aborts by more than half at a cost of approximately 60 milliseconds of additional latency - a deliberate trade of speed for perceived conversational competence.

3.2 Latency as the Product

Each pipeline stage is measured independently rather than as an aggregate, since TTFT alone is often the largest single contributor, at approximately one second. This granularity enabled a targeted intervention: switching to GPT-5.1 on the responses API cut time to speech by more than 0.7 seconds. A tiered model fleet routes turns between frontier and smaller/faster models, governed by a tone router - a lightweight classifier that reads emotional state per turn. Critically, routing decisions are based on conversational stakes rather than cost minimization; crisis or therapist-register turns are always routed to the best available model.

"Latency isn't just a number you check at the end, it's actually the product." "We route based on stakes not on cost."

The frontier model costs approximately five times as much per turn as the smaller alternative, yet routing roughly one-third of turns to the smaller model produced almost no measurable effect on retention - indicating that stakes-based routing preserves user experience while materially reducing inference cost on low-stakes turns.

3.3 Memory as Retrieval, Not Transcript

Retaining full conversation history as an append-only transcript does not scale and is associated with hallucination and degraded output quality. The reported architecture instead extracts facts, preferences, and emotional signals, embeds them, and stores them in a vector database with sub-50-millisecond lookup latency.

"We see memory as a retrieval system."

A nightly compression process merges duplicate memories, clusters related entries, resolves contradictions, and discards noise. Notably, retrieval is not limited to matching the last user message: the system generates internal questions about the user and the relationship, using these as retrieval queries. Memory is partitioned into stable (cached, high-recall-frequency) and unstable (live-tail, low-latency) blocks, allowing frequently accessed facts to be served efficiently while recent, still-evolving information remains available without contaminating the cached layer.

3.4 Rebuilding Context Rather Than Fighting Drift

Reusing cached context across turns is characterized as a structural trap in volatile voice conversation, since drift accumulates faster than incremental updates can correct. The reported solution reassembles the context window from scratch on every turn: a recent-message summary, the persona card, retrieved memories, tone guidance, and real-time application state.

"You should rebuild context and not fight drift."

Identity stability is preserved through an alien character design that permits user projection without a real-world reference to anchor comparisons, combined with a parallel tone-monitoring system that adjusts delivery to emotional cues while preserving core identity across hundreds of turns.

4. Technical Insights

Several implementation-level findings carry direct applicability beyond this system. First, latency must be treated as a decomposed, per-stage metric; a drift from 2.0 to 2.5 seconds round-trip degraded every measured product metric, indicating that voice-oriented products may have latency thresholds with nonlinear effects on user behavior rather than gradual degradation. Second, model routing decisions benefit from being framed around conversational stakes rather than raw cost, since a substantial fraction of turns tolerate cheaper models without retention loss - suggesting headroom for cost optimization in similarly structured products. Third, memory systems built as retrieval infrastructure (embeddings, vector search, nightly consolidation) appear more robust at scale than transcript-based context accumulation, though this trades off some interpretability of "what the model knows" at any given moment. Fourth, context reassembly per turn imposes computational overhead relative to context caching but appears necessary given the volatility of spoken interaction; this represents an explicit design trade-off rather than a default architectural choice. Finally, the internal agent fleet - implementation agents building to pixel-perfect snapshots, review agents enforcing standards, a PR shepherd, and a triage bot integrated via MCP - suggests that codebases can be structured to serve as agent-readable documentation, reducing reliance on separate specification files.

5. Discussion

These findings collectively suggest that voice-first AI products cannot be evaluated using text-chat benchmarks or architectures without material performance cost. The reported metrics - crash-free rate rising from 99.6% to 99.9%, runtime errors dropping over 50%, and highly engaged user share doubling coincident with AI-authored code becoming dominant - raise a broader question about causal attribution: multiple product and engineering changes occurred concurrently, and the synthesis cannot isolate the agent fleet's contribution from other simultaneous improvements. Similarly, the near-zero retention impact of routing one-third of turns to smaller models is a single-product finding whose generalizability across domains, populations, and stakes-classification schemes remains untested.

The multi-judge evaluation framework - assessing archetype fidelity, model mechanics, code standards, a skeptical persona lens, and safety - represents a notable methodological contribution for evaluating character-driven AI systems, though its reported use (a single new character shipped in one day using 7 million tokens and 4.5 hours of compute) constitutes one case rather than a validated benchmark. These practices align with broader industry movement toward agentic software development and stakes-aware inference routing, but independent replication would strengthen confidence in the reported effect sizes.

6. Conclusion

This analysis identifies four architectural principles for voice-first AI systems - volatility-aware turn-taking, latency as a measured product property, memory as retrieval rather than transcript, and per-turn context reassembly - each supported by specific, quantified production outcomes. The practical takeaway for engineering teams building conversational voice products is that text-chat architectural assumptions (stable context, tolerant latency budgets, transcript-based memory) do not transfer and must be deliberately re-engineered. The secondary finding, that agentic coding fleets wired via MCP coincided with substantial reliability improvements, suggests a viable path for scaling engineering velocity in parallel with product complexity, warranting further controlled investigation of causal contribution in future work.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub