Turn detection in voice agents: silence timeouts vs VAD vs semantic models

Every production voice agent platform needs a rule for deciding when a caller has finished speaking, a job known as end of turn detection. Three mechanisms do that job. A fixed silence timer and a voice activity detector both infer turn ends from quiet, while a fused model reads acoustic and semantic signals together.

Your choice sets two production numbers at once, how often your agent cuts callers off and how long it waits before every response. In this guide, you'll walk through the failure mode of each mechanism and how to measure your own false interruption rate against your current baseline.

For platform teams embedding voice agents in enterprise products:

• Fixed silence timers fire on mid-turn pauses because pause and gap durations overlap heavily.
• A VAD detects when sound stops but can't determine whether the thought is complete; noise and hesitation both trigger it.
• Models fusing acoustic and semantic signals predict completion instead of waiting for silence.
• No industry standard defines a bad false interruption rate; build a baseline and compare against it.
• Eager end-of-turn events start the LLM earlier at the cost of discarded drafts.

Each mechanism trades interruptions against latency, and only the fused model predicts completion instead of waiting for silence. The sections below cover how each one fails.

The configuration surface row is the one to read twice, because it predicts how much of your team's time each mechanism will consume after launch.

A VAD scores each audio frame for speech presence, then applies a silence window. A fused model uses semantic information and acoustic cues such as prosody and timing to estimate the probability that the turn is complete.

The timer can't tell a pause from a finish, so it interrupts people who think out loud and delays everyone else. The VAD misreads hesitation silence as completion and noise as speech. The fused model trades those failures for compute cost and a tuning surface: confidence thresholds you have to set and then verify.

Silence timeouts cut people off because a thinking pause and a finished turn last about the same time, so the timer expires mid-thought. What follows quantifies that overlap, shows which pauses fool a timer, and names the one job it still does well.

A 500 ms timer catches more of the pauses you want to ignore than the turn ends you want to act on. Across two conversation corpora, that threshold captured 56 to 60% of within-turn pauses but only 47 to 51% of real turn transitions. Stretching the timer to a full second keeps the same pattern and makes every response slower.

Pause length depends on who's calling. One study measured 686 ms of average pause for Mandarin-English bilinguals against 546 ms for native English speakers, with a wide spread inside both groups. Both averages sit above a 500 ms timer, so a caller who's still thinking gets cut off. One threshold can't fit a caller base that diverse.

For deployed voice agents, keep this mechanism only as a backstop. Deepgram's Flux STT documentation describes a silence fallback built for exactly this role. Setting eot_silence_threshold_ms forces a turn end once the model stays uncertain past its 5000 ms default. The fallback guarantees your agent eventually responds; the model handles every normal exchange long before it fires.

Falling pitch and falling loudness mark a finished turn, though no single cue settles it. A turn end shows up when several of them line up at once.

Speakers drop their pitch and lower their voices as they approach the end of a turn. A corpus study of task-oriented dialogue measured both patterns, and found that the more turn-yielding signals a speaker gives at once, the likelier the listener is to jump in. It also turned up two results that cut against intuition. Speakers speed up rather than slow down before a turn end, and drawn-out final syllables turn up more often mid-turn than at the finish.

A voice activity detector scores short slices of audio for whether someone is talking, then waits for a set stretch of quiet before calling the turn over. Silero, the one most teams reach for, works exactly this way. Nothing in that sequence asks whether the sentence finished.

Purpose-built turn detectors do better than a VAD and still hit a ceiling. One open-source detector listens to the last eight seconds of a turn. It scores that audio for unfinished thoughts, filler words, and intonation, and its accuracy varies by language. Sound alone only goes so far. Research on ambiguous Chinese sentences found prosody accounted for about 15% of how complete a sentence seemed.

A filler or a half-finished word means the speaker is still going, though a silence timer can't tell. Whether the words so far form a complete thought is something a transcript helps settle and audio can't.

People pause to think, and they do it at the worst possible moment for a machine. Pauses land mid-thought often enough that silence on its own can't tell you a turn ended. A caller who says "because" and stops has finished nothing, yet the audio after that word is indistinguishable from trailing silence.

The words are what tell you a thought closed, and only a transcript carries them. Research on predicting utterance ends shows a model can call the ending before the speaker stops when it reads words and audio together. But a detector fed by a separate transcription stream inherits that stream's delay and its mistakes.

Newer systems put both jobs in one model rather than running a turn detector after a transcription stream. Deepgram's Flux STT launch post states the test plainly, that the model knows "because…" or "uh, sorry…" isn't a complete thought, while "Thanks so much." clearly is.

Deepgram reports the fused model cut response latency by 200 to 600 ms against pipeline approaches and cut false interruptions by about 30%, both its own figures. Separate research on fusing audio with language found the combination beat an audio-only model by 22.6% but a text-only model by just 3.67%. The conversational state write-up covers the architecture in depth.

Building your own baseline is the only option, because no ITU-T, ETSI, or ISO standard defines an acceptable rate for a voice agent. The method matters more than the number you land on.

Every time your agent's speech starts before the caller's turn ended, log one false interruption. Exclude backchannels ("mm-hmm") and background noise from the numerator, and write your definition down. Published benchmarks define it differently, so their numbers can't be pooled.

The TurnBench leaderboard, for instance, disqualifies systems above a 0.15 false positive rate, but the bar applies only to that benchmark. Deepgram's evaluation methodology for end-of-turn detection models covers corpus-level benchmarking; the steps here cover your deployed platform.

Decide how many turns you need before you open a single call. Use stratified random sampling by language, then distribute the sample across deployed hours and enterprise-customer queues. Because turns within one call correlate, plan at the conversation level and adjust the sample size for clustering.

Only your own prior measurement is a valid comparison point. Measure the current configuration, change one parameter, re-measure on the same sampling frame, and test the difference between the two proportions. Two patterns count as bad whatever your aggregate says.

The first is a rate that climbs after a change you made to bring it down. The second is a subgroup sitting well above your overall figure, whether that's one language, one tenant, or one queue. Segment before you call any number acceptable. A rate that looks fine next to a published benchmark can still be terrible for your platform's callers.

Eager mode starts the LLM on a medium-confidence guess instead of waiting for confirmation, buying back latency callers notice. You pay in discarded drafts, so it fits cheap, fast models only.

Eager mode adds two signals. EagerEndOfTurn means the caller is probably done, and TurnResumed means they kept going. Deepgram's eager end-of-turn docs put the threshold between 0.3 and 0.5.

At those settings the eager signal arrives 150 to 250 ms early, at the price of 50 to 70% more LLM calls. Start drafting on the eager signal, hold the spoken reply until the turn is confirmed, and throw the draft away if the caller resumes.

Every time the caller resumes, you've paid for a draft nobody hears. Nobody publishes a typical discard rate for this. The closest public figure comes from research on early endpoint prediction, which put wasted computation at 28.4% of the theoretical maximum.

The study work committed to an ending up to 2.56 seconds before the caller stopped and cut average response time by 505 ms.

Turn it on for short, transactional turns across customer deployments, such as order status and authentication, when your LLM is cheap and fast. Routing is another suitable use case. Keep it off for dictation-style workloads or expensive models. And instrument the discard rate from day one, so you know what fraction of eager drafts you're paying to throw away.

Start from defaults and tune only after you've measured, because every threshold trades interruptions against latency.

Run the Flux STT quickstart with the default turn threshold and change nothing else on the first pass. Leave the silence fallback alone and keep eager mode off. Raising the threshold cuts interruptions but makes affected turns wait longer, so measure before you touch it.

After any threshold change for a customer, run the false interruption measurement again on the same sample and watch response times alongside it. Break the numbers out by customer and by language if your callers switch languages mid-call. If eager mode is on, track how many drafts you throw away against your LLM bill.

Tuning goes faster with your own audio in the loop. Start with Deepgram and spend the $200 in free credits on your hardest production calls.

No. Barge-in detection listens for the caller cutting in while the agent talks, and turn detection decides whether the caller has finished. The errors cost different things. A false barge-in cuts off valid agent speech, and a missed one leaves the caller ignored.

A turn detector trained on one language does a poor job on another. Testing across languages found models built for a single language predicted turns badly outside it. Flattening the pitch of the audio cost up to two points of accuracy in Japanese and Mandarin, and left English roughly unchanged.

No. Raise it only as a controlled test with rollback criteria. A longer threshold does cut interruptions, but it delays every response and gives mid-thought pauses more room to read as turn ends. Measure both effects on the same sample before and after, and roll back if completion accuracy falls outside your baseline range.

Track endpoint latency by channel type and tag calls with music or overlapping speech. Silero's maintainers document false alarms on string instruments such as piano and guitar.

Endpoint Anticipation can commit seconds before the actual stop. It starts LLM and TTS work on partial context, so every downstream call has to be cancellable.