Paralanguage in voice AI: a complete guide to paralinguistic cues

Voice AI systems log what a caller said and drop how they said it. A transcript records the words a speaker chose and discards the delivery. Gone are the pitch rise, the two-second pause, and the breathy tail.

Paralinguistic cues cover that remainder, and they fall into five families. Those families are prosody, voice quality, disfluency, timing and silence, and non-speech vocalization.

This guide gives each family a unit, a measurement scale, and a verdict on whether you can get it out of a production transcription API. Cues without units can't be logged.

Each of the five cue families sits at a different stage of production readiness.

• Disfluency and timing ship today. Turn on filler transcription, keep word timestamps, and add speaker attribution.
• Prosody needs a parallel audio path. Run openSMILE alongside transcription and log the eGeMAPS parameters.
• Voice quality has fixed units but no agreed method for mapping measurements to labels like "breathy."
• Non-speech vocalization has no production answer beyond one provider's laughter tags. Sighs and crying go undocumented everywhere.
• Delete the audio and you lose prosody and voice quality permanently.

Anything in the voice that carries meaning without being a word counts as paralanguage. Paralinguistic cues are the measurable pieces of that signal, and this article treats gesture and facial expression as outside the boundary.

Filled pauses like "uh" are transcribable lexical tokens and hesitation signals. They sit on the line between content and delivery. Text-derived judgments aren't acoustic measurements. Use this rule to tell them apart. A sound swap that changes which word you said is phonemic, while the same word delivered breathier or louder is paralinguistic.

A cue you can't name precisely is a cue you can't measure or compare across sources. The list below defines the terms used throughout this guide.

• Paralanguage: the set of vocal signals that carry meaning without being lexical content.
• Paralinguistic cue: a single measurable signal within paralanguage, such as a pitch rise.
• Prosody: the cue family covering pitch, loudness, and tempo across an utterance.
• Voice quality: the cue family covering breathiness, creak, and strain.
• Disfluency: a break in fluent delivery consisting of a reparandum, an optional interregnum, and a repair.
• Non-speech vocalization: a vocal sound that isn't a word, including laughter, sighs, and coughs.

The first two terms name the category itself. The rest name the families this guide works through one at a time, each with its own units and its own verdict.

Prosody carries structure and emphasis before any word is parsed, and general-transcription APIs don't surface it. Prosodic measurement runs on a standard feature inventory called eGeMAPS, which fixes a name, a definition, and a unit for each parameter.

Intonation, emphasis, and pace organize the utterance before recognition assigns words. The units below come from the eGeMAPS parameter set, built to give voice research a common measurement baseline.

Prosody interacts with recognition accuracy. On IEMOCAP, Li et al. (2023) measured a Word Error Rate of 38.9% for happy speech against 36.3% for neutral. They noted that neutral wasn't the best-recognized category. Related work on the synthesis side is covered in the accuracy tax of emotional voices in TTS.

Every cue in the table above is detectable today, though not from the transcript. Pitch, loudness, and tempo come out of a feature extractor like openSMILE run over the raw audio. Only words per minute is computable from a transcription response, using word-level timestamps.

General-transcription APIs don't surface prosody parameters. The near-exception is Microsoft Azure and its Pronunciation Assessment, whose ProsodyScore covers stress, intonation, speaking speed, and rhythm in the en-US locale only.

Breathiness, creak, and strain all have fixed units you can measure, though a 2025 systematic review found no agreed approach to measuring breathiness in non-pathological speakers.

These measurements describe delivery without changing lexical identity. The Praat manual carries the operational definitions of jitter, shimmer, and HNR, along with MDVP pathology thresholds it flags as provisional. H1–H2 is the harmonic difference parameter in eGeMAPS.

Log all four as separate fields, because no single measurement identifies a breathy or strained voice on its own.

All four measurements are extractable today from raw audio using standard toolkits. Turning them into labels is the weak link: a 2024 preprint built on those four features reached 61.59% accuracy for breathiness and 52.74% for creakiness.

Models miss both the parameters and the labels. No general-transcription API surfaces voice-quality output, and no cross-domain standard maps the measurements to terms like "breathy" or "strained."

Disfluency events share one structure whether they're a lone "um" or an abandoned sentence. Each event divides into a reparandum and repair, with an optional interregnum between them and a split at the interruption point.

The reparandum is the stretch the speaker replaces; the repair resumes fluency. Shriberg's taxonomy sorts these events by observable surface patterns, not cause.

Clark and Fox Tree proposed in 2002 that uh marks an expected minor delay and um an expected major one. A 2025 turn-taking review notes acoustic data showing no reliable duration difference between the two.

Filler words are the mainstream case, excluded by default and retained on request. Deepgram's filler_words parameter on a Nova-family model keeps seven hesitation tokens, and two other providers in the table below document the same behavior.

The remaining two providers don't document filler-word retention at all. Across all five, repetitions, substitutions, false starts, insertions, and articulation errors receive no labels. Extracting them requires post-processing the transcript against a disfluency parser.

Timing and silence are the easiest paralinguistic cues to measure and the hardest to label. No agreed cut-off separates a hesitation from a completed turn, so endpointing thresholds stay an application decision.

Human conversation puts longer silences inside turns than between them, which is why a fixed silence timeout misfires. Production dialogue systems commonly close a turn after about one second of silence. Thresholds shift with task: a 2025 study found 200 ms best for proficiency in both task types and 350 ms best for perceived fluency in dialogue.

Silence, overlap, and system latency all carry temporal information. A multilingual turn-taking study measures gap duration at transitions and pause length during turn holds.

Every row is a duration, so one timestamped transcript with speaker attribution supports the whole table.

Models detect every timing cue in the table, because word-level start and end times are ubiquitous. Those timestamps give you same-channel silence directly, and diarization tells you whether a given silence is a pause or a gap.

What's missing is the label, not the number. All five providers return raw durations rather than marking a silence as a pause, a gap, or a hesitation, so classification is left to application logic.

Laughter, sighs, and crying are the one family production APIs leave almost entirely to separate classifiers. The strongest laughter detector in a 2024 benchmark reached an F1 of 0.784 on the AMI meeting corpus.

You can assign start and end times to affective sounds, along with a label. A 2025 foundation model for non-verbal vocalization treats laughter, crying, sighs, and coughs as its target categories.

Unlike the other families, none of these arrive labeled from a transcription API, so each row needs a detector you supply.

Only laughter, and only as one of a set of audio event tags requested through prompting. The closest active work sits in research foundation models built for vocal bursts rather than in transcription APIs.

Only AssemblyAI documents laughter, as one tag among more than fifty audio events. No provider documents sigh or cry detection. Deepgram's end-of-speech detection documentation says the feature ignores non-speech audio like door knocking, a ringing phone, or street noise.

Across providers, production coverage clusters around fillers and timing. Three cue families remain absent from general-transcription output.

Every provider returns timing, and none returns voice quality. Fillers are the only column where they split, while prosody and non-speech sounds each appear once as a limited exception.

A "No" means the provider doesn't document that cue in general-transcription output, not that the underlying audio lacks the signal.

Start with the cues you already have. Turn on filler transcription, keep word-level timestamps, and add speaker labels so same-speaker pauses stay distinguishable from between-speaker gaps.

Then decide about the audio itself before you delete it. Prosody and voice quality can only be measured from the waveform, so a text-only retention policy closes those two families permanently. Storing audio alongside transcripts keeps the option open even if you don't build the extraction path yet.

Run one hour of your own calls through a transcription API and count what comes back. Create a free account and put your $200 free credits toward finding out which of the five families your pipeline can actually log today.

Not reliably. The 2002 proposal that uh signals a minor delay and um a major one is contested by acoustic work finding no consistent difference in the pauses that follow each.

Acoustic units transfer; annotation categories don't. The ToBI annotation system, which symbolically describes prosodic content in American English, was never designed to cover other languages. Adaptations like GToBI and K-ToBI exist for that reason.

Measurement returns an H1–H2 value in dB; interpretation maps it to a phonation type for your target domain. Log both separately so you can revise the classification without re-extracting the acoustic feature.

Multiple acoustic measures contribute, and no agreed method combines them across domains.

Only when the output preserves timestamps. Check diarization before naming each interval: same-speaker silence is a pause, while silence between different speakers is a gap.