Backchannels vs interruptions: Handling listener signals in voice agents

An energy-threshold detector, scored against 30 hours of hand-annotated human conversation, fired on 45% of the backchannels and non-speech noise it heard. In casual talk, where backchannels are densest, that rate climbed to 52%.

None of those triggers was a bid to speak, and every one of them would stop a voice agent mid-sentence. Turn detection separates a listener's "keep going" from a caller taking the floor. An energy threshold can't make that distinction.

Your transcript already carries most of it. In this guide you'll find which tokens to look for, how long to wait after one, and what to log when your policy guesses wrong.

Four things decide whether your agent holds its ground or stops talking:

• A backchannel is a continuer that tells the speaker to keep going rather than a signal that the listener wants the floor.
• Energy-based barge-in can't separate the two, because both arrive as speech energy while your agent is talking.
• The specific backchanneling tokens (uh-huh, mhmm) land in your transcript once filler words are on; handling them is a lookup.
• The negative forms (uh-uh, nuh-uh, mm-mm) are the exception. They're two-syllable corrections your agent should yield to.

A backchannel is a short listener signal that means "keep going," not a bid to take the floor. It's nodding out loud, the audio equivalent of a "..." while the other person keeps talking. The recognisable examples are mm-hmm, uh-huh, right, yeah, and okay.

Victor Yngve coined "back channel" in 1970 for a secondary channel that carries short listener messages while the primary speaker keeps the turn. Conversation analysts refined the affirmative subset into "continuers."

Mm-hmm, uh-huh, and mhmm are the pure listener signals. Right, yeah, and okay do the same job, but they double as agreement tokens and sometimes as full replies. Depending on prosody and context, yeah and okay can signal agreement, disagreement, or surprise, and that ambiguity is why they're harder to handle than mm-hmm.

Your VAD stops on a backchannel because a continuer and a floor bid look identical to it. Both arrive as voiced energy mid-playback, and energy is the only feature it measures.

Frame-level voice activity detection makes one binary call, speech or not-speech. It reports that the caller spoke and never why, so "uh-huh" and "wait, that's the wrong address" both cross the threshold as voiced energy.

Stop on a continuer and the agent halts for a caller who wanted more. Hold through a correction and the agent talks over a caller saying no. Lowering barge-in sensitivity fixes the first error and worsens the second; raising it does the reverse. Base rates tilt the damage. In 30 hours of recorded conversation, listeners backchanneled nearly seven times more often than they cut the speaker off. An energy detector therefore makes far more false stops than real catches.

The rate undersells the damage because the caller experiences every false stop as a full recovery cycle. The agent halts, apologizes, restarts, and loses its place, which costs seconds and makes it sound twitchy.

Turn filler words on and, per Deepgram's filler-word documentation, the Speech-to-Text API returns seven vocalizations with consistent spellings. Five of them are listener signals; uh and um are hesitation fillers. Handling backchannels becomes a string lookup rather than a modeling problem.

Deepgram's filler words documentation names uh, um, mhmm, mm-mm, uh-uh, uh-huh, and nuh-uh. The parameter filler_words sits at false by default, and with it off the stripping behavior covers uh and um. Sample a few hundred of your own calls to confirm the forms you get back before branching on exact matches. Deepgram documents the feature for its Nova-family general models, so confirm coverage for the model you actually run.

The two sets call for opposite actions. Mhmm and uh-huh tell you to keep talking, and uh-uh, nuh-uh, and mm-mm tell you to stop. Recent work on conversational feedback keeps "Disagree" and "No" as classes separate from Continue, so the negatives aren't a milder version of the affirmatives. Spellings differ across sources, and no single reference catalogues all three negative forms together.

Bundling uh-uh, nuh-uh, and mm-mm into one negative class is an engineering convenience.

The negative forms run two syllables with a hard break in the middle, where the affirmatives stay breathy and smooth. They also arrive far less often than mhmm and uh-huh, so this branch fires seldom. A caller answering a payment-amount readback with "nuh-uh" has objected, and holding through that is the worse error.

A turn-detection model reads the words and the acoustics together, so "uh-huh" and "wait, that's the wrong address" land on different sides of the decision. A frame-level VAD has only energy to go on. Flux STT works that way, fusing turn detection into the same model that produces your transcript. The technical companion describes it as carrying acoustic and semantic context into a joint turn decision rather than a binary speech-versus-silence flag.

"Right" and "okay" transcribe as regular words with no marker separating them from the start of a sentence, so a token lookup can't classify them. The list is English-only, and inventories don't transfer. Backchannel models trained on two languages perform poorly on an unseen third, because Japanese, English, and Chinese each use their own forms.

Listeners hear the split in pitch direction, both on the token itself and on the speaker's last unit before it. A rising contour on "okay" reads as a bid for the floor, and a falling one reads as a continuer.

Listeners get the difference from how a token is said, not just which token it is. "Yeah...", "yeah?" and "yeah!" are one word carrying three different readings.

Pitch numbers are a weak stand-in for that. Tested against listener judgments of English feedback clips, measurements like mean pitch, pitch range, and pitch slope agreed 36 to 49% of the time. Fuller acoustic representations of the same clips reached 60 to 77%.

You can't read pitch off the API. Deepgram's streaming speech-to-text documentation lists no F0 or contour field, so a design that branches on contour has nothing to branch on. Use transcript-level proxies instead.

Use token identity first. Word-level timestamps show its position within the turn and whether speech resumes after it. For turn detection itself, lean on Flux STT rather than reconstructing prosody from what the API returns.

Four checks in order settle hold or yield, and the order is what does the work. Get it wrong and you either cut off a caller or talk through a correction, and those two errors pull in opposite directions.

Wire your handler to turn-level events rather than raw VAD frames. Bind your handler to the speech-start event named in Deepgram's Flux state reference, which documents the states Flux STT moves through. Trigger on a short vocalization during playback and ignore an isolated energy spike.

Check the transcribed text against the seven-token list. First confirm that the model you're running in the agent path returns those tokens. The turn events in Step One and the filler-word tokens here can come from different models. Classify a match as affirmative (mhmm, uh-huh) or negative (mm-mm, uh-uh, nuh-uh). Send unmatched text, including ordinary words and full sentences, to standard barge-in handling.

Wait briefly after the token to see whether the caller adds more words. "Uh-huh, but actually the address is wrong" is a floor bid that opens with a continuer; a standalone "uh-huh" followed by silence isn't. Word end-timestamps plus a short wait window give you this distinction, the practical substitute for the pitch contour you can't retrieve.

Apply the policy in this order:

1. Standalone affirmative token: hold. Keep speaking; the caller asked you to.
2. Any negative token: yield immediately. Treat it as a correction, standalone or not.
3. Affirmative token with continued speech: yield. The caller is taking the floor.
4. No match: yield. This is ordinary barge-in.

Treat that ordering as an application-layer design choice rather than a validated ranking. The hold-or-yield decision belongs to your application layer.

No published industry baseline exists for how often agents stop themselves by mistake, so define the ratio and compute it from your own logs. The working definition is the share of cases where the agent stops talking in response to a sound that was never a bid for the floor.

Production traffic has no test cases, so adapt the ratio. For the denominator, count every event where your agent stopped its own speech in response to caller audio. For the numerator, count the subset where the caller produced no full turn within a fixed window after the stop. A caller who stays quiet was backchanneling rather than bidding for the floor. Two seconds is a defensible starting point; nothing in the literature fixes it. Write that window down so two engineers reading the same logs compute the same number.

Log four fields per event. Was the agent speaking when caller audio arrived, what did the transcript say, what did your policy decide, and what did the caller do next. The last field gives you the numerator. Record a full turn when one follows; if none does, record silence or a repeated token.

False interruptions and missed corrections move in opposite directions under single-threshold tuning. Waiting a little longer before yielding cuts false stops but delays every real barge-in, and tightening your endpointing threshold does the reverse. So track a missed-correction count alongside it, triggered by negative tokens the agent held through. Use your own previous deployment as the comparison for the miss side.

You already have a policy, so start by measuring the one your stack shipped with. Whatever its barge-in behavior does today is implicit and unmeasured, and logging the decision has to come before changing it.

Turn on filler_words, add the four log fields, and run a normal week of traffic. Compute false-interruption rate and the missed-correction count from that baseline before touching a threshold or a hold rule.

Split the policy by what a mistake costs at each step. Disclosures and payment confirmations should yield, because a missed "nuh-uh" costs a wrong charge while an unnecessary stop costs a second of awkwardness. Flux STT lets you change thresholds mid-stream, so you can loosen turn detection while collecting an account number and tighten it afterward. Open-ended support flows can hold through affirmative tokens, since callers do most of their backchanneling while explaining a problem at length.

Replay a day of your own production audio with filler words on and count how many "uh-huh"s your current policy would've stopped for. Create a free account with Deepgram and put your $200 free credits toward that experiment.

A backchannel leaves the floor with the current speaker, and an interruption takes it away. So mark a backchannel when the speaker keeps the turn and an interruption when the listener starts a full turn. Because the audio can be near-identical, label the intent instead of relying on vocalization length alone.

No. filler_words defaults to false, so you have to turn it on to get the token strings back. A migration audit should separate configuration coverage from caller behavior. Existing transcripts recorded with the setting off won't provide a usable baseline. Confirm filler_words is on in the model path producing your transcript, then compare only new traffic.

No. Duration and sensitivity knobs only trade one error type for the other, so ignoring backchannels reliably takes a model that reads context.

The evidence leans yes. A 2024 EMNLP Findings study found significantly higher engagement when an agent backchanneled. But there's a ceiling. Past a moderate rate, more feedback made agents seem less attentive rather than more.

Yes, in both the forms themselves and how often callers use them. Don't translate the English token list. Give each language to a fluent QA owner and have them validate the forms, frequencies, and placement before release.