Paralanguage in voice AI: a complete guide to paralinguistic cues

A guide to paralinguistic cues in voice AI: units, provider coverage, and what you can log from a transcript.

Voice AI systems log what a caller said and drop how they said it. A transcript records the words a speaker chose and discards the delivery. Gone are the pitch rise, the two-second pause, and the breathy tail.

Paralinguistic cues cover that remainder, and they fall into five families. Those families are prosody, voice quality, disfluency, timing and silence, and non-speech vocalization.

This guide gives each family a unit, a measurement scale, and a verdict on whether you can get it out of a production transcription API. Cues without units can't be logged.

Key takeaways

Each of the five cue families sits at a different stage of production readiness.

• Disfluency and timing ship today. Turn on filler transcription, keep word timestamps, and add speaker attribution.
• Prosody needs a parallel audio path. Run openSMILE alongside transcription and log the eGeMAPS parameters.
• Voice quality has fixed units but no agreed method for mapping measurements to labels like "breathy."
• Non-speech vocalization has no production answer beyond one provider's laughter tags. Sighs and crying go undocumented everywhere.
• Delete the audio and you lose prosody and voice quality permanently.

What counts as paralinguistic

Anything in the voice that carries meaning without being a word counts as paralanguage. Paralinguistic cues are the measurable pieces of that signal, and this article treats gesture and facial expression as outside the boundary.

Inside the words and around them

Filled pauses like "uh" are transcribable lexical tokens and hesitation signals. They sit on the line between content and delivery. Text-derived judgments aren't acoustic measurements. Use this rule to tell them apart. A sound swap that changes which word you said is phonemic, while the same word delivered breathier or louder is paralinguistic.

Paralinguistic cues: the terms in one place

A cue you can't name precisely is a cue you can't measure or compare across sources. The list below defines the terms used throughout this guide.

• Paralanguage: the set of vocal signals that carry meaning without being lexical content.
• Paralinguistic cue: a single measurable signal within paralanguage, such as a pitch rise.
• Prosody: the cue family covering pitch, loudness, and tempo across an utterance.
• Voice quality: the cue family covering breathiness, creak, and strain.
• Disfluency: a break in fluent delivery consisting of a reparandum, an optional interregnum, and a repair.
• Non-speech vocalization: a vocal sound that isn't a word, including laughter, sighs, and coughs.

The first two terms name the category itself. The rest name the families this guide works through one at a time, each with its own units and its own verdict.

Prosody: pitch, loudness, tempo

Prosody carries structure and emphasis before any word is parsed, and general-transcription APIs don't surface it. Prosodic measurement runs on a standard feature inventory called eGeMAPS, which fixes a name, a definition, and a unit for each parameter.

Signature table

Intonation, emphasis, and pace organize the utterance before recognition assigns words. The units below come from the eGeMAPS parameter set, built to give voice research a common measurement baseline.

Prosody interacts with recognition accuracy. On IEMOCAP, Li et al. (2023) measured a Word Error Rate of 38.9% for happy speech against 36.3% for neutral. They noted that neutral wasn't the best-recognized category. Related work on the synthesis side is covered in the accuracy tax of emotional voices in TTS.

What current models detect

Every cue in the table above is detectable today, though not from the transcript. Pitch, loudness, and tempo come out of a feature extractor like openSMILE run over the raw audio. Only words per minute is computable from a transcription response, using word-level timestamps.

What they miss

General-transcription APIs don't surface prosody parameters. The near-exception is Microsoft Azure and its Pronunciation Assessment, whose ProsodyScore covers stress, intonation, speaking speed, and rhythm in the en-US locale only.

Voice quality: breathiness, creak, strain

Breathiness, creak, and strain all have fixed units you can measure, though a 2025 systematic review found no agreed approach to measuring breathiness in non-pathological speakers.

Signature table

These measurements describe delivery without changing lexical identity. The Praat manual carries the operational definitions of jitter, shimmer, and HNR, along with MDVP pathology thresholds it flags as provisional. H1–H2 is the harmonic difference parameter in eGeMAPS.

Log all four as separate fields, because no single measurement identifies a breathy or strained voice on its own.

What current models detect

All four measurements are extractable today from raw audio using standard toolkits. Turning them into labels is the weak link: a 2024 preprint built on those four features reached 61.59% accuracy for breathiness and 52.74% for creakiness.

What they miss

Models miss both the parameters and the labels. No general-transcription API surfaces voice-quality output, and no cross-domain standard maps the measurements to terms like "breathy" or "strained."

Disfluency: fillers, repairs, false starts

Disfluency events share one structure whether they're a lone "um" or an abandoned sentence. Each event divides into a reparandum and repair, with an optional interregnum between them and a split at the interruption point.

Signature table

The reparandum is the stretch the speaker replaces; the repair resumes fluency. Shriberg's taxonomy sorts these events by observable surface patterns, not cause.

Clark and Fox Tree proposed in 2002 that uh marks an expected minor delay and um an expected major one. A 2025 turn-taking review notes acoustic data showing no reliable duration difference between the two.

What current models detect

Filler words are the mainstream case, excluded by default and retained on request. Deepgram's filler_words parameter on a Nova-family model keeps seven hesitation tokens, and two other providers in the table below document the same behavior.

What they miss

The remaining two providers don't document filler-word retention at all. Across all five, repetitions, substitutions, false starts, insertions, and articulation errors receive no labels. Extracting them requires post-processing the transcript against a disfluency parser.

Timing and silence

Timing and silence are the easiest paralinguistic cues to measure and the hardest to label. No agreed cut-off separates a hesitation from a completed turn, so endpointing thresholds stay an application decision.

Human conversation puts longer silences inside turns than between them, which is why a fixed silence timeout misfires. Production dialogue systems commonly close a turn after about one second of silence. Thresholds shift with task: a 2025 study found 200 ms best for proficiency in both task types and 350 ms best for perceived fluency in dialogue.

Signature table

Silence, overlap, and system latency all carry temporal information. A multilingual turn-taking study measures gap duration at transitions and pause length during turn holds.

Every row is a duration, so one timestamped transcript with speaker attribution supports the whole table.

What current models detect

Models detect every timing cue in the table, because word-level start and end times are ubiquitous. Those timestamps give you same-channel silence directly, and diarization tells you whether a given silence is a pause or a gap.

What they miss

What's missing is the label, not the number. All five providers return raw durations rather than marking a silence as a pause, a gap, or a hesitation, so classification is left to application logic.

Laughter, sighs, and crying

Laughter, sighs, and crying are the one family production APIs leave almost entirely to separate classifiers. The strongest laughter detector in a 2024 benchmark reached an F1 of 0.784 on the AMI meeting corpus.

Signature table

You can assign start and end times to affective sounds, along with a label. A 2025 foundation model for non-verbal vocalization treats laughter, crying, sighs, and coughs as its target categories.

Unlike the other families, none of these arrive labeled from a transcription API, so each row needs a detector you supply.

What current models detect

Only laughter, and only as one of a set of audio event tags requested through prompting. The closest active work sits in research foundation models built for vocal bursts rather than in transcription APIs.

What they miss

Only AssemblyAI documents laughter, as one tag among more than fifty audio events. No provider documents sigh or cry detection. Deepgram's end-of-speech detection documentation says the feature ignores non-speech audio like door knocking, a ringing phone, or street noise.

Which paralinguistic cues today's models can and can't detect

Across providers, production coverage clusters around fillers and timing. Three cue families remain absent from general-transcription output.

Provider coverage

Every provider returns timing, and none returns voice quality. Fillers are the only column where they split, while prosody and non-speech sounds each appear once as a limited exception.

A "No" means the provider doesn't document that cue in general-transcription output, not that the underlying audio lacks the signal.

Logging paralanguage on your own audio

Start with the cues you already have. Turn on filler transcription, keep word-level timestamps, and add speaker labels so same-speaker pauses stay distinguishable from between-speaker gaps.

Then decide about the audio itself before you delete it. Prosody and voice quality can only be measured from the waveform, so a text-only retention policy closes those two families permanently. Storing audio alongside transcripts keeps the option open even if you don't build the extraction path yet.

Run one hour of your own calls through a transcription API and count what comes back. Create a free account and put your $200 free credits toward finding out which of the five families your pipeline can actually log today.

FAQ

Do "uh" and "um" signal different things?

Not reliably. The 2002 proposal that uh signals a minor delay and um a major one is contested by acoustic work finding no consistent difference in the pauses that follow each.

Are paralinguistic cues consistent across languages?

Acoustic units transfer; annotation categories don't. The ToBI annotation system, which symbolically describes prosodic content in American English, was never designed to cover other languages. Adaptations like GToBI and K-ToBI exist for that reason.

What's the difference between detecting a cue and interpreting it?

Measurement returns an H1–H2 value in dB; interpretation maps it to a phonation type for your target domain. Log both separately so you can revise the classification without re-extracting the acoustic feature.

Why is breathiness harder to measure than pitch?

Multiple acoustic measures contribute, and no agreed method combines them across domains.

Can a text transcript recover timing after the fact?

Only when the output preserves timestamps. Check diarization before naming each interval: same-speaker silence is a pause, while silence between different speakers is a gap.