The voice agent architecture guide: STT, LLM, and TTS pipeline design in 2026

Design a production voice agent pipeline. Budget STT, LLM, and TTS latency, handle interruptions, and pick your architecture.

Budget latency across STT, LLM, and TTS stages while carrying interruption state. Choose an AI voice agent architecture for production call volume. Consider a bundled API when it can reduce inter-service hops that a composed stack can't avoid.

Callers start noticing lag near 800 ms of total round-trip time, and one 2025 observational study of nine voice-assistant users measured a 1,366 ms mean response delay. Upstream STT and LLM stages spend most of the budget before synthesis ever starts, and you often don't find out where it went until a call already sounds broken.

This guide breaks that budget into its STT, LLM, and TTS stages, then walks through the interruption handling and architecture trade-offs that follow.

Key takeaways

Lock down these five decisions before you write a line of pipeline code:

• In a local NVIDIA H100 study, LLM generation dominated ASR compute.
• Flux TTS interleaves text and audio generation, so response length doesn't raise first-audio latency.
• On barge-in, /v2/speak returns text_spoken and text_remaining. The returned fields split what the caller heard from what never played.
• Starting LLM prefill on Flux STT's eager end-of-turn signal trims latency at the cost of additional speculative LLM calls.
• A single-WebSocket bundled API eliminates the cross-process turn-promotion race and reduces inter-service hops.

Provider comparison at a glance

A bundled API wins when fewer handoffs matter more than component freedom. Reach for a composed stack instead when you need independent models, deployment controls, or per-stage timing.

How to use the table

Evaluate both options across six production decision points: connection count, interruption reconciliation, model choice, timing visibility, failure surfaces, and compliance isolation.

Why Voice Agent Latency Breaks Down Across the Pipeline

STT finalization and LLM time-to-first-token add to TTS time-to-first-audio, so every response accumulates three budgets that compound. Acceptable numbers at each stage can still sum to a broken conversation, which is why voice agent architecture work starts with a budget rather than a model choice.

STT delay comes from endpointing, not compute

In the H100 study, cited above, most recognition delay occurs outside raw model compute. The wait around it dominates. Softcery's latency breakdown puts VAD and endpointing at 150–400+ ms because you have to confirm the caller actually finished. Benchmark numbers bundle both costs.

LLM generation dominates the latency budget

Generation dominated the model side in the H100 study. Remote APIs stretch that further. VerticalAPI's April 2026 benchmark measured OpenAI GPT-4o at roughly 820 ms p50 time-to-first-token from cloud regions, which consumes most of a one-second naturalness budget on its own. Speculative prefill on early transcripts can cut the critical path.

Synthesis is the smallest slice of the latency budget

When the upstream stages behave, synthesis is the smallest slice. Flux TTS interleaves text and audio generation, so response length doesn't raise first-audio latency, and Deepgram's launch documentation puts first audio as low as 80 ms.

Transport still bites. Softcery's budget adds 30–80 ms of ingress jitter buffering on the way in, and each extra service hop carries its own network cost.

How STT, LLM, and TTS components interact in a voice agent pipeline

The pipeline loses the most time as audio becomes a transcript and the transcript becomes a prompt. The token-to-speech handoff adds another delay. Handoff design decides whether those stages overlap or run in sequence, and that difference separates real-time pacing from awkward silence.

Streaming audio and partial transcripts

Maintain two transcript buffers during live recognition: a replaceable hypothesis you can act on early and an append-only committed record you trust. Deepgram's Streaming Audio exposes orthogonal signals for both. is_final: false marks a replaceable interim, is_final: true marks a finalized segment, and speech_final: true reports a detected pause.

Append each finalized segment and treat speech_final as the flush trigger. Deepgram's docs warn: "Long utterances may have multiple is_final: true responses before speech_final: true is returned."

Turn detection and LLM handoff

Gradium's semantic VAD write-up points out that a short mid-sentence pause can sound identical to an end-of-turn pause. Deepgram's Flux STT models the turn itself instead: an EagerEndOfTurn event signals moderate confidence the caller finished, TurnResumed cancels it, and EndOfTurn commits.

Starting LLM prefill on the eager signal trims 100–200 ms off the critical path, at a documented cost of 50–70% more LLM calls. The trade buys perceived responsiveness with cheap tokens, and it usually makes sense for customer-facing agents.

Streaming text to Flux TTS

Forward tokens to the synthesis socket as your LLM emits them, then send Flush to mark the turn complete. You don't repeat per-turn styling. Flux TTS strips markup such as W3C SSML and style tags automatically and emits a Warning with code INPUT_MARKUP_STRIPPED; expressive delivery comes from the model's own conversational state.

Azure pushes that styling work back onto you every turn. Microsoft Azure's SSML documentation states that a missing or invalid mstts:express-as style value reverts to neutral delivery, with no error to flag the regression.

Designing for interruption and state handling

Treat barge-in as a pipeline-wide design decision, because the API contract you pick determines how much of that tracking you own. Bug reports across different stacks describe the same unsolved problem: after a caller cuts in, a composed pipeline can't know exactly what audio was heard without hand-built tracking.

Client-side interrupt signaling

You detect the cut-in in your STT or VAD layer, because Flux TTS state carries no caller audio. The endpoints return different answers. On the legacy /v1/speak endpoint (Aura-2), the Clear message stops audio, but the Cleared confirmation carries only a type and a sequence_id. You must track playback position and decide what to discard or regenerate.

Flux TTS text spoken and text remaining

Send Interrupt with a playback_offset on /v2/speak. The SpeechInterrupted response reports audio_played_ms, text_spoken, and text_remaining. These fields separate the exact fragment the caller heard from the text that never played.

The API returns that boundary, so your client never has to reconstruct it. By contrast, if you use Pipecat, watch for a reported interruption frame overtaking already-released text frames. One turn committed 'The pool opens' while the caller heard 'The pool opens at six.' The orphaned words leaked into the next assistant message as 'at six. We close at nine.'

Resuming context mid-turn

Prosody survives the barge-in. Deepgram's context documentation states that Interrupt leaves model state intact, that state persists through Flush and barge-in, and that only a new connection resets it.

The reset applies only to the model's own prior generations; your application still owns user audio, LLM reasoning, and the conversation history: append text_spoken, then drop or regenerate text_remaining.

Plan the connection lifecycle around that reset rule. Reconnecting starts prosody from scratch. Sessions on Deepgram's Voice Agent API carry a 2-hour session limit, warn 5 minutes before closing, and expect a KeepAlive every 8 seconds.

Choosing bundled vs composed architecture

Compose the stack when you need self-hosted components or an existing LLM contract. Composition also provides per-stage observability, but you must accept the seams.

Otherwise a bundled voice agent architecture, one connection carrying one latency budget, removes more failure modes than component flexibility returns. Identify which trade-offs matter in your deployment.

When bundling reduces failure points

If you use LiveKit Agents, watch for a reported race condition between caller-speech detection and agent-output pausing. A second failure mode shows up across provider integrations, Deepgram's among them.

The stream transcribes fine for a few turns, then stops emitting with no close frame and no error. You may need a watchdog-and-failover layer. Deepgram's Voice Agent API runs the whole loop over a single WebSocket, which means one connection lifecycle to monitor and no cross-process promotion race to lose.

When a composed stack makes sense

Keep the seams when you already hold an existing OpenAI contract or custom-trained model weights that a bundle can't absorb. A self-hosted Llama deployment is another reason. Per-stage observability is another honest reason; Pipecat's metrics framework tracks time-to-first-byte per stage.

Regulated workloads can favor a composed stack when you need greater per-component isolation, but Deepgram also documents a self-hosted Voice Agent API. Deepgram's self-hosted deployments send no audio or transcripts back to Deepgram, only license validation and usage metadata.

Vendor-reported case studies describe CallTrackingMetrics running Deepgram inside its own AWS VPC. Elerian AI reports an optional on-premises ASR deployment for banking customers.

Evaluating your total latency budget

A 2025 survey of spoken language models puts the average human reply gap at approximately 200 ms, a baseline none of the pipelines measured above approaches. Twilio's engineering guidance budgets 350 ms for STT, 375 ms for LLM first token, and 100 ms for TTS first byte and targets a 1,115 ms median mouth-to-ear gap.

Work backward from the noticeability threshold above. Subtract transport costs, including jitter, and endpointing delay before allocating what's left to models. If the composed total won't fit, remove hops before shopping for a faster model.

Building your AI voice agent architecture

Start from the caller's ear and work backward: transport, synthesis first audio, LLM first token, endpointing. The stage figures above give your voice agent architecture defensible targets at every hop. Compose when the seams buy control you'll provably use; bundle when the seams themselves are the bigger risk.

Evaluation checklist

You'll catch most integration surprises with four checks against production-like traffic:

• Measure p95 per stage, not means; scope differences explain most benchmark disagreements.
• Interrupt the agent mid-sentence and verify the conversation history matches the audio that actually played.
• Confirm session limits, reconnection behavior, and which state a reconnect destroys.
• Price the composed stack against bundled rates before contract renewal, not after.

Get started with Deepgram

Every budget in this guide is easier to trust once you've measured it against your own audio, not a vendor's demo reel. Create a free account, and put your $200 free credits toward Flux STT streaming, Flux TTS sessions, or the bundled Voice Agent API on your own hardest calls.

FAQ

How many concurrent Flux TTS connections can one project run?

Your project limit depends on plan, region, and project under Deepgram's rate limits. Check the project-specific ceiling before load testing or production rollout.

What is AI voice agent architecture?

AI voice agent architecture is the design of the full pipeline connecting speech recognition, language model reasoning, and speech synthesis into one real-time conversation loop. It covers how these components share state, handle interruptions, and hit combined latency targets rather than optimizing each piece in isolation. You can build it as a bundled API or compose separate services. Either way, the architecture decision determines how much of the caller experience you engineer yourself versus inherit from a vendor.

For an alternative to a separate STT, LLM, and TTS pipeline, review our guide to enterprise speech-to-speech models.

How do Deepgram's endpointing defaults affect turn handling?

Use 300–500 ms of silence for conversational turns. Streaming endpointing defaults to 10 ms of silence before returning speech_final. For utterance-level segmentation, the utterance_end_ms setting accepts 1,000–5,000 ms and needs interim results turned on.

What compliance coverage does Deepgram document?

For qualifying covered entities, Deepgram's compliance documentation covers HIPAA-aligned deployments and BAAs through sales and enterprise agreements, plus SOC 2 Type II certification and PCI compliance with yearly review. Treat the BAA as a procurement dependency rather than a self-serve setting. HHS guidance treats any cloud service processing ePHI as a business associate requiring a BAA.

Can you swap the LLM or TTS inside the Voice Agent API?

Yes. Deepgram's bundled-versus-assembled breakdown documents both options. Before rollout, test partial failures at each substituted component boundary, including timeout propagation, retry ownership, and fallback behavior.