Voice AI latency: why pauses read as hesitation

Meta Title: Voice AI latency: why pauses read as hesitation

Meta-description: Learn how voice AI latency affects user perception, why silence reads as incompetence, and how to set latency budgets by deployment context in 2026.

URL Slug: voice-ai-latency-why-pauses-read-as-hesitation

Voice AI latency: why pauses read as hesitation

In a Deepgram survey of 40 developers building voice agents, 61% reported noticeable to major flow issues. Flow breaks when a reply arrives nearly a second after the caller stops talking, which reads as hesitation even when every word is right.

The person on the call draws their own conclusion from your delay. Voice AI latency becomes a perception problem before it becomes a metrics problem. And the meaning gets worse as the silence grows.

End-of-turn detection, the language model, and speech synthesis each consume part of the delay you can afford, so no single component owns the total. Latency budgets differ by deployment context: phone IVR, in-app assistant, drive-through ordering, and live captioning each need a separate target.

In this guide, you'll see what a caller reads into a pause, and when your agent should start talking. You'll also see why the silence before the agent speaks is the number to track. Then you'll set a target for your own deployment, whether that's a phone line, an app, a drive-through, or live captions.

• In human conversation, transitions between speakers average under a quarter-second. When your agent takes a second or longer, callers read the wait as uncertainty rather than processing.
• When your agent replies before the caller finishes, it cuts them off and returns an incomplete transcript.
• Flux STT end-of-turn detection consumes only part of the budget; LLM plus TTS time sits on top of it.
• Conventional pipelines take 1,000 to 1,500 milliseconds per turn, at least four times longer than a human reply.

A caller who waits too long for a reply reads hesitation first, then uncertainty, then the sense that nobody is listening. Voice AI latency gets graded on the same social scale as a human reply. Voice AI latency shapes the caller's read the same way a human pause does.

The typical pause between speakers in human conversation lasts a fraction of a second. In a 2023 cross-linguistic study, Hoogland and colleagues measured a Dutch mean of 199 milliseconds and an English mean of 242 milliseconds. Callers bring that baseline to every call, so they judge your agent's delay against a norm under a quarter-second rather than against your latency target.

A 2024 pause-perception study found that longer inter-turn pauses make a responder seem less knowledgeable and less confident. Replicating an earlier Polish experiment with Chinese listeners produced the same result, though listeners cut non-native speakers more slack on willingness to help than on competence. Your agent gets the same treatment, and callers blame the pause on the agent rather than on a hard question.

Bridging the wait with an "um" makes the delay feel shorter without making it shorter. A 2025 VR study found that natural conversational fillers improved perceived response time, especially under high-delay conditions, but the underlying processing time remained. Shorten your agent's real delay first, then use fillers to smooth over what's left.

Your agent should reply the moment the caller's turn has actually ended, and not one beat before. The band between interrupting and stalling is narrower than most pipelines assume, and both mistakes cost you the caller's confidence.

Conventional pipelines land between 1,000 and 1,500 milliseconds, according to a 2024 architecture estimate. Those turns run at least four times longer than the human baseline measured earlier. Deepgram's own Flux TTS launch post draws the practical line even earlier, putting the point where callers start noticing lag at roughly 800 milliseconds of total round-trip time.

A reply that arrives before the caller finished couldn't have accounted for the ending, so it reads as an agent that stopped listening. End-of-turn detection that fires too early cuts the user off while they're still speaking and hands your pipeline an incomplete transcript. An early start costs you the rest of the caller's sentence, plus a repeat turn to recover it. Aim for the shortest delay that still respects the user's turn.

Turn detection has to decide whether the caller actually finished before anything downstream can react correctly. Sakuma and colleagues report that half of human response timings are below 350 milliseconds, while end-of-utterance detection and speech recognition generally take longer. Where that prediction lives, in a speech-to-speech model or a cascaded pipeline, shapes everything downstream.

The caller only ever perceives one number, the silence between their last word and your agent's first sound. Everything else you measure is a distribution they never experience.

Your caller hears their own last word, then silence, then your agent's voice. They stop timing at that first sound, so synthesis time, token throughput, and transcript finalization never reach them. Budget to first audio and treat full-response time as a separate engineering number.

A median tells you about the middle call and hides the slowest tenth, which is where callers hang up. Published Flux STT launch figures put p90 end-of-turn detection at 1 second and p95 at 1.5 seconds. Detection alone accounts for that, so LLM and synthesis time add to the silence on top. Track your p90.

Some turns need a fast reply and others need an accurate one. A yes-or-no confirmation can be short, while an account-number readback needs the length accuracy requires. Synthesis works the same way, since audio can start before the full sentence exists. Set your target by the turn rather than by the call.

A latency budget is the silence you allow before your agent replies, and no single number fits every deployment. One target everywhere either overspends on engineering or loses calls.

A driver shouting over engine noise and a viewer reading captions grade different clocks, so set the target by what the listener is doing and hold it on your slowest calls.

Each row sets its own target, and the sections below explain where each number comes from.

An automated phone menu answers inbound calls with no screen involved, so silence is the only signal the caller gets, and the table above assigns telephony its tightest budget. Telephony also spends part of that budget on network transmission before your pipeline sees the audio, so the caller's clock starts before your agent does. Deepgram's migration guide reports approximately 260 milliseconds p50 for Flux STT end-of-turn detection at default settings, before LLM and TTS time are added.

A voice feature inside an app the user is already looking at is the only case here where you can show the wait.

An immediate progress indicator gives the user feedback that a screenless call can't provide. Don't count on it to buy latency headroom, since the VR study cited earlier found wait indicators didn't measurably improve perceived response time.

Your agent sits on a speaker post outside, where engine noise and a queue set the terms, and repeats cost more than milliseconds. In a 2025 mystery-shop study, 34% of AI drive-through customers had to repeat their orders, versus 22% at human-staffed locations.

The table target is an extrapolation from cascaded-pipeline results. Accuracy under noise protects this budget better than shaving first-byte time, because every re-prompt spends whole turns.

Captions appear on screen while an event is still happening, so judge delay against the rule's own standard rather than a stopwatch. The FCC's caption quality rule sets no millisecond ceiling. Instead it requires a caption to land alongside the speech it belongs to, and to stay with the spoken words as closely as the program type allows.

Accuracy outranks speed on this row. Captions can run behind and still comply when live programming makes tighter synchronization impractical, though accuracy alone doesn't determine compliance.

Pick one number per deployment, the p90 silence from end of user speech to first agent audio, and treat it as the spec. Your voice AI latency is whatever that number says, which is why a healthy median can't excuse a slow tail.

Deepgram's Voice Agent Quality Index treats the turn as the unit of quality and gives interruptions 40% of its score.

In its own test runs, agents at or above 3 seconds of latency scored in the mid-50s even with few interruptions, which Deepgram attributes to callers judging the agent incompetent during the silence. Per-turn measurement surfaces both failure edges, while request-level timing hides them.

You can trade compute for detection speed, and Flux STT exposes the setting. Deepgram's guidance puts eager end-of-turn processing at roughly 100 to 200 milliseconds shaved off total latency, at the cost of 50 to 70% more LLM calls.

Whether that trade pays depends on your row in the table; a phone IVR fighting the noticing threshold probably takes it, while a captioning pipeline has no reason to.

Match your deployment to a table row and treat that row as your spec. Set its p90 target, run a week of production traffic against it, and only then touch model settings.

Test against real audio rather than logs. Flux STT, Deepgram's conversational speech recognition model, emits turn events you can timestamp instead of inferring end of speech from a transcript. Create a free account and put your $200 free credits against your hardest calls.

Not by itself. Trigger the tone from microphone activity rather than completed reasoning, and suppress it when the first audio is already ready. Log tone-to-first-audio separately so status feedback doesn't hide a slow pipeline.

Stream the response. A 2026 measured pipeline reached 755-millisecond time-to-first-audio with an LLM time-to-first-token of 296 milliseconds. Buffer enough partial text to prevent choppy synthesis, then track time-to-first-token and first audio separately.

Set one target for prerecorded output and a separate one for live. Prerecorded captions can be checked and corrected before release, so hold them to a tight sync target you verify during review. Live captions air once, so pick a words-behind-audio ceiling your pipeline can actually hold. Log that number per program, then review the breaches rather than the averages.

Vary the delay while holding content constant. Inject fixed delays across a range, keep the wording identical, and have participants rate the same task at each step. Randomize the order so nobody learns the pattern. Tolerance shifts with what users think the agent is doing, so test your actual task rather than a generic prompt.

Yes, through repeats. Accented or disfluent speech raises the error rate, and each correction costs another turn, so the callers your model handles worst also wait the longest. Build accent-stratified test sets, track repeat requests by cohort, and define a human fallback after repeated recognition failures.