How Does Voice AI Work? ASR, LLMs and the Latency Budget

In human conversation, the average pause between one speaker finishing and another starting is roughly 200 milliseconds. While humans tolerate slightly longer pauses when talking to software over a telephone line, a silence exceeding 1.5 seconds creates awkwardness, prompts users to repeat themselves, and leads to conversation breakdown.

Building an enterprise voice AI agent is fundamentally an exercise in real-time latency budget management. Every turn requires streaming voice audio over cellular networks, converting acoustic signals to text, reasoning over intent, executing backend business logic, and synthesizing natural speech - all within roughly one second.

This guide breaks down the core architecture of enterprise voice AI, showing where the milliseconds go and how technical teams build resilient, sub-second conversational systems.

The Pipeline at a Glance

Most enterprise voice AI deployments rely on a cascaded (or staged) architecture. In this design, caller audio enters through a media gateway and passes sequentially through three specialized systems before streaming back to the caller.

1. Speech-to-Text (ASR): Converts incoming audio streams into text transcripts in real time.
2. LLM Reasoning & Function Calling: Processes text, determines intent, references knowledge bases, and executes API calls to backend systems.
3. Text-to-Speech (TTS): Converts generated text back into synthesized, natural audio streams.

The staged pipeline vs native speech-to-speech

While emerging speech-to-speech (S2S) models process audio directly without an intermediate text step - promising lower latency and native tone preservation - the staged pipeline remains the enterprise standard.

Staged architectures allow engineering teams to inspect intermediate text transcripts, swap individual components independently, enforce strict safety guardrails, and audit LLM function calls before audio is synthesized.

ASR: Turning speech into text

Automatic Speech Recognition (ASR) is the entry door for voice AI. To minimize delay, enterprise ASR operates on a continuous audio stream, returning partial transcription hypotheses while the caller is still speaking.

The end-of-turn challenge

The most critical task for an ASR engine is end-of-turn detection (endpointing) - deciding exactly when a caller has finished speaking.

Legacy systems rely on simple silence timers (e.g., waiting for 500ms of absolute silence). However, static timers present a tradeoff:

• Too aggressive (400-600ms): The system cuts callers off mid-thought during natural pauses.
• Too cautious (800-1200ms): The system introduces noticeable silence before starting to process a reply.

Modern enterprise ASR replaces static timers with semantic VAD (Voice Activity Detection) models. 

These acoustic models analyze grammar, pitch contours, and sentence structure to detect whether a pause represents a completed thought or a mid-sentence hesitation.

The LLM: Understanding, Reasoning, and Acting

Once text is transcribed, it is passed to a Large Language Model (LLM).

In a voice application, the metric that matters most is time-to-first-token (TTFT) - the wall-clock time required for the model to output its first word.

Because speech synthesis can begin as soon as the first clause is generated, low TTFT is essential for maintaining natural conversation flow.

Managing function-calling delays

When an AI agent executes a backend action such as checking a database for an order status or validating an account number, the required API lookup introduces additional latency.

To prevent long silences during API execution, well-engineered voice architectures utilize two strategies:

1. Speculative fillers: The LLM streams an immediate, natural bridge phrase (e.g., "Let me check that account for you...") while the backend API call executes in parallel.
2. Dual-model topology: A fast, lightweight model handles dialogue turn-taking, while a larger, specialized model handles complex background reasoning and tool calls.

TTS: Turning Text Back into Speech

The final stage converts text tokens from the LLM back into spoken audio. 

The primary performance metric here is time-to-first-audio (TTFA) or time-to-first-byte (TTFB) - the duration from receiving text tokens to emitting the first playable audio frame.

Modern streaming TTS engines process text on a sentence or clause boundary, allowing audio playback to begin while the LLM is still generating subsequent words.

Where the Time Goes: A Realistic Latency Budget

To understand where time is spent during a voice conversation, consider a representative latency budget for a well-optimized, staged enterprise voice agent:

Note: Mouth-to-ear latency represents total round-trip time experienced by the caller. Platform-only latency metrics quoted by vendors often exclude mobile network transport and audio buffer delays. Target latency for live production should aim for ~1.3 seconds, with an upper acceptable threshold of 1.5 seconds. 

Technical levers for reducing latency

• Co-locate services: Host ASR, LLM, TTS, and orchestration engines within the same regional cloud data centers to eliminate multi-region network hops.
• Stream end-to-end: Never wait for complete sentences or full responses; stream audio bytes and text tokens across every boundary.

• Eliminate audio resampling: Telephony networks deliver audio at 8kHz or 16kHz. Avoid unnecessary upsampling or downsampling conversions during pipeline processing.

What This Means for Indian Enterprise Deployments

Deploying voice AI across Indian enterprise environments introduces specific architectural realities:

• Cellular network jitter: Callers often connect over variable 3G/4G connections. Audio packet loss requires adaptive jitter buffers that handle network instability without inflating total latency.
• Code-switching complexity: Mixed-language speech (e.g., Hinglish) increases ASR processing time if models rely on multi-pass translation steps instead of native multilingual acoustic models.

• Evaluating P95 latency: Average or median latency metrics hide performance spikes. Enterprise evaluation must focus on 95th-percentile (p95) latency under peak call volume to ensure performance remains stable during traffic spikes.

Haptik addresses these requirements by deploying voice infrastructure within local data centers, minimizing network hops and ensuring low-latency routing across regional telecom networks.

How to Test Latency Before You Buy

When evaluating enterprise voice AI vendors, use this verification checklist:

• Demand mouth-to-ear metrics: Ask for p95 mouth-to-ear latency figures measured from live production calls, rather than isolated model inference numbers.
• Supply production audio files: Test candidate platforms using real call recordings containing background noise, regional accents, and mid-sentence pauses.

• Test under peak load: Evaluate system responsiveness under simulated concurrent traffic rather than single-call test environments.

How Haptik Approaches Architecture

Haptik designs its conversational voice platform around operational performance and real-world deployment needs:

• Engineered for complex networks: Designed specifically to process Indic languages, code-switched speech, and compressed audio over regional cellular networks.
• Outcome-oriented infrastructure: Architectural decisions are evaluated based on verified issue resolution rather than isolated speed metrics.

• Tailored enterprise tuning: Forward-deployed engineering teams work directly with enterprise clients across 500+ deployments to tune pipeline performance to specific operational traffic needs.

The Bottom Line

Voice AI operates as an integrated system, and overall performance is determined by the slowest component in the pipeline. Enterprise buyers should evaluate how vendors manage latency budgets across speech recognition, LLM reasoning, system integrations, and audio synthesis - judging platforms on p95 mouth-to-ear performance rather than idealized sales demos.

FAQs