The real-time voice AI stack for agents: An architecture guide

A production voice agent chains transport, speech recognition, a language model, speech synthesis, and orchestration into a single real-time exchange, and each layer draws from the same shared clock. It has about a second to start talking before the exchange feels broken.

Perceived willingness begins to drop after 600 ms and steps down significantly from 700 to 800 ms. Transport, recognition, endpointing, the LLM's first token, TTS first byte, and the return network hop each take a slice of that budget. This guide breaks down what each layer costs and where you can cut to keep your voice AI stack inside it.

Every layer in a voice AI stack spends part of the per-turn budget. These are the trade-offs that matter most as you build.

• You risk degraded production calls when total turn latency crosses the acceptable range.
• Carrier transport costs more latency than WebRTC transport before any model runs.
• Streaming can materially reduce time-to-first-audio.
• Endpointing is a highly tunable layer in the stack.
• Tool calls can materially increase LLM latency.

Bundled platforms reduce integration work, so reach for them when you want fewer integrations to manage. Frameworks trade that convenience for more media and deployment control, and each provider's strengths vary by layer. Scan the HIPAA and self-hosted columns first if compliance is a hard requirement, since those rule providers out fast. From there, compare concurrency model and pricing to see which fits your traffic pattern.

Test latency, concurrency, compliance, and failure handling on your carrier path and audio before you commit to a provider. With the providers mapped, the next question is what actually makes up that stack and why voice pushes it harder than any other interface.

A voice AI stack is the chain of perception, reasoning, action, and orchestration layers that turn spoken audio into a spoken response. The same chain runs in any agent, but voice's turn-taking timing turns generous headroom into a hard deadline. Here's the general architecture, then why voice breaks it in ways text never does.

Input becomes structured signal, reasoning picks the action, and orchestration sequences the rest. Voice imposes a tighter reasoning deadline than text.

The mode of human turn transitions falls between 0 and 200 ms in all ten languages in Stivers et al.'s cross-linguistic study. Deployed assistants miss badly, with a 2025 corpus study of 1,077 Alexa turn transitions measuring an average floor-transfer offset of 1,366 ms. Invariant turn timing, the authors noted, makes the assistant sound mechanical and unnatural rather than conversational.

Nothing your model does matters until the audio finishes its trip from mouth to server, and phone calls pay the biggest price for it. SIP or PSTN transport adds 200 to 400 ms before processing starts, while WebRTC connections that skip a carrier cost less.

• SIP and PSTN overhead: For network planning, ITU-T G.114 treats one-way delay below 150 ms as essentially transparent and 400 ms as a ceiling you shouldn't exceed.
• WebRTC and browser transport: Traffic that never touches a carrier costs less latency over UDP, which avoids TCP's head-of-line blocking.
• Failure modes when transport is an afterthought: Test the carrier path your callers use.

Speech recognition is the first layer your audio touches, and where you start the clock changes your whole latency budget. Budget from transcript finalization instead of the first partial result.

Deepgram documentation places the total client side transcript latency at between 200 and 500 ms for Speech Recognition, so you should measure against the finalized transcript. From there, this section covers typical streaming speeds, where noisy audio and domain vocabulary trip up accuracy, and the tools that help.

A 2026 enterprise tutorial measured Deepgram streaming STT at a 337 ms P50 in a self-hosted pipeline and 402 ms P50 when paired with a cloud LLM. Across both setups, latency ranged from a 184 ms floor to a 601 ms worst case.

Getting the words right the first time protects your latency budget, since misheard terms trigger retries, corrections, or a fallback pass that erase whatever time you saved upstream. Test the terms your callers actually use with domain vocabulary.

On AfriSpeech-MultiBench, GPT-4o Transcribe averaged a 38.94% WER across 100-plus African English accents. Streaming itself costs accuracy: a multi-provider study measured 10.9% WER streaming versus 9.37% batch (p<0.01). Keyterm Prompting lets you bias up to 100 terms per session, inside a 500-token limit.

A healthcare provider on the Five9 platform doubled user authentication rates after switching to Deepgram for alphanumeric inputs. CallTrackingMetrics runs Deepgram inside its AWS VPC, citing faster transcription and lower latency in addition to higher accuracy.

Set the cutoff by balancing response latency against the risk of cutting callers off mid-sentence. You can tune Endpointing around that tradeoff.

Flux cuts agent response latency by 200–600 ms versus a traditional STT-plus-VAD chain and detects end of turn at about 260 ms p50 by default. Eager mode starts LLM generation before confirmation. Deepgram documents saving hundreds of milliseconds at the cost of 50–70% more LLM calls.

A 2026 turn-taking study measured a 60.0% early-interruption rate for an acoustic VAD baseline at 320 ms. A 2025 semantic VAD study measured 0.962 overall accuracy at a 500 ms threshold once semantic detection was layered on top.

On its own, acoustic VAD caught none of the still-talking cases in the same test. Deepgram reports Flux reduces false interruptions by about 30%. Dead air shows up on the other side of that same tradeoff, when eot_timeout_ms is set too high and the agent waits out the full silence window before responding.

Flux emits Start of Turn and End of Turn events from the recognition model itself; Nova-3 doesn't. Pipecat's open Smart Turn v3 provides framework-level turn detection.

Streaming means your TTS can start speaking the moment the LLM emits its first tokens, so a slow full response barely matters if those first tokens arrive fast. A 2026 benchmark study puts the median target at 500 ms time-to-first-token for real-time voice interaction, since that's the delay callers actually feel.

Deepgram's production pipeline put GPT-4.1-mini at 457 ms P50 with a 784 ms maximum. Self-hosted inference runs faster: a benchmark of Llama-3.1-8B-Instruct on TensorRT-LLM averaged 162.67 ms. You can allocate 200–400 ms here when planning your pipeline budget.

Once a turn reaches outside the model, tool calls can account for 30 to 80% of latency before the first token of the final answer renders. Much of that cost comes from orchestrators running tool execution and LLM calls sequentially instead of overlapping them, wasting parallelism that's already available in the request.

Your choice of provider changes baseline TTFT as much as your choice of model does, so weigh that before you wire one in. The Voice Agent API accepts BYO LLM configurations for OpenAI platform (open_ai), Anthropic models (anthropic), Google Cloud (google), NVIDIA models (nvidia), Groq models (groq), and AWS Bedrock (aws_bedrock).

Nearly all of a text-to-speech turn's latency budget goes toward producing that first byte of audio, since everything the model generates after that streams while the caller is already listening.

Deepgram's Speak v2 announcement says the endpoint "begins responding in as low as 80 milliseconds." That's a best case, not an average or an SLA, so measure your own P95 for Speech Synthesis.

Your model, endpoint, network path, and measurement boundary all affect time-to-first-audio. Cartesia's current Sonic model claims sub-90ms model latency, while the ElevenLabs platform puts its Flash models at about 75ms model inference.

Both figures measure the model alone. ElevenLabs' own latency documentation adds 20 to 200 ms for network round-trip, plus a common 500 ms player buffer, before audio actually reaches the caller.

Request-shaped TTS forces manual sentence chunking and reconnection hacks when your LLM streams fragments; Flux TTS is turn-based instead and holds voice state across turns. When your client sends an Interrupt message, Flux TTS returns text_spoken and text_remaining, so you know what the caller heard and where to resume.

Detecting the barge-in itself still happens upstream, in your STT/VAD or the Voice Agent API, not inside Flux TTS. Entity errors are the other common failure mode, where a model mangles drug names, account numbers, or other alphanumerics that the caller actually needs to hear correctly. Deepgram built Flux TTS specifically to communicate that kind of information accurately.

Speak v2's WebSocket and REST endpoints carry Flux TTS at GA. Cartesia offers Sonic models, and ElevenLabs offers Flash v2.5.

Orchestration holds the budget together by running transport, recognition, the LLM, and synthesis in overlap instead of in sequence, so each layer's savings compound rather than stack. Here's what it actually coordinates, where it breaks under production load, and how platforms handle it for you.

• What the orchestration layer actually coordinates: A single WebSocket at https://agent.deepgram.com/v1/agent/converse carries the loop, wants a KeepAlive every 8 seconds when idle, and closes at 2 hours.
• Failure modes at production call volume: Concurrency caps can bite before compute, at 10 call slots by default on Vapi and 50 instances per Pipecat Cloud deployment.
• Orchestration and voice agent platforms: One runtime covers STT, LLM calls, and TTS in Deepgram's Voice Agent API. See the current pricing page for rates.

There's no single best voice AI stack, only the best combination for whichever priority matters most to you, whether that's speed, control, or compliance. Here's the shortlist by layer, how to match tools to what matters most, and when a bundled runtime beats stitching point solutions together.

Use this default layer-by-layer shortlist:

• Transport: WebRTC where you control the client; co-located telephony for phone traffic.
• Recognition: Flux for conversational turns; Nova-3 for Keyterm Prompting and formatting.
• Orchestration: the Voice Agent API bundled; Pipecat or LiveKit for framework control.

If compliance leads, favor pipelines that keep text at every stage for redaction and logging. Vapi's security portal says HIPAA requires Enterprise and a signed BAA. LiveKit reports SOC 2 Type II and lists its HIPAA-eligible services.

When a turn breaks, terminal errors like FAILED_TO_THINK and FAILED_TO_SPEAK identify the failing stage.

Architecture decides whether each millisecond buys auditability, overlap, or raw speed, and the four common patterns trade these off differently. Cascading pipelines spend on auditability, streaming reclaims milliseconds through overlap, speech-to-speech spends everything on raw speed, and hybrid setups mix patterns within a single call.

Cascading pipelines cost you upfront latency, but they buy something end-to-end voice models can't match, which is control at every stage. LiveKit's taxonomy credits cascaded pipelines with the modularity and compliance tooling that a single fused model can't give you. That matters more than raw speed once you're handling regulated calls.

Compared with sequential waiting, the same enterprise tutorial's own pipeline example dropped time-to-first-audio from about 1,600 ms to 900 ms. It did that by overlapping STT, the LLM's first sentence, and TTS instead of waiting for each stage to finish. The overlap itself, not any single fast component, is the actual mechanism behind every streaming claim in this guide.

Single-model systems skip the text hop: Kyutai's Moshi reports about 200 ms response latency in practice. An independent benchmark measured OpenAI Realtime at roughly 232 ms median and 320 ms average end-to-end. Speed costs control, and some speed-focused S2S models require cloud APIs.

When you need S2S speed but your own TTS voice, half-cascade setups restore the choice. The same tutorial measured Qwen3-Omni's Thinker paired with external TTS at roughly 755 ms, close to the 702 ms cloud-only S2S path. That's a small price for keeping your own TTS voice.

Build your production latency budget by allocating milliseconds to each layer before you write code, then instrument every stage so real calls can be measured against the plan and tightened where they run over. The subsections below walk through a sample allocation, where to cut first when you're over budget, and how to validate the whole thing on your own audio.

Use component measurements already captured in your traces to build the sample budget. Account for overlap between streaming stages instead of adding every measurement as a sequential wait.

Tighten endpointing first, where eager end-of-turn handling trades extra LLM calls for earlier starts. Swap the LLM for a smaller model last; it can't buy back transport time.

Measure user stop-speaking to first audio byte. Instrument every stage and judge real call audio at P95, because medians hide the turns that make callers hang up. Deepgram's Voice Agent API emits a per-stage LatencyReport covering STT, first token, tool calls, and TTS, so you can attribute overages to a specific layer instead of guessing from an end-to-end number.

Try the per-layer prices against your traffic. Create a free account and put your $200 free credits toward measuring your hardest calls.

Target a P95 turn latency under about 1,000 ms, and treat anything beyond a couple of seconds as the point where callers start to disengage. Set percentile SLOs, not averages, and calibrate against user-study research like this IEEE study on perceived response delay in voice interfaces, then confirm the specific thresholds against your own caller feedback.

Budget for the tool's own execution time plus extra LLM inference for tool selection and post-tool response generation, which can meaningfully extend a turn if run sequentially.

An AsyncLM benchmark shows that overlapping function-call generation and execution asynchronously can cut task completion latency by 1.6x to 5.4x versus synchronous tool calling. Set a separate function timeout so one slow dependency doesn't consume the rest of your turn budget.

Yes, but only through a warm handoff between prefilled instances rather than a live pattern swap inside one session. The closest documented pattern is OpenAI's warm model-instance handoff, where a second instance prefilled with session context runs in parallel and cuts over when ready. Preserve the same session context when transferring control between instances.

Budget 200–400 ms for SIP and PSTN legs versus roughly sub-150 ms for WebRTC that skips the carrier, per ITU-T G.114, then confirm the split with real carrier-path measurements. Test packet loss, jitter-buffer sizing, geographic routing, and carrier failover because each can change the production result.

Use synthetic-call canaries and clock-skew detection before opening traffic. Define trace-sampling, timeout, alert, and rollback thresholds as launch gates. Deepgram's Voice Agent API emits a per-stage LatencyReport covering STT, first token, tool calls, and TTS.