TTS with emotion: emphasis and control in live conversations

TTS with emotion in a live voice agent has three control points. You can put it in markup the LLM writes inline, in a parameter on the synthesis request, or in state the model carries across turns.

Voiceover tooling assumes an editor reviews every line before it ships. In a live call, the text arrives from an LLM mid-turn and nobody touches it before synthesis. Deepgram's Flux TTS, its conversation-native Text-to-Speech model, is the running example here. Deepgram's documentation is specific about what those expressive controls do and don't guarantee. This guide covers what each control point costs in latency, tokens, and word accuracy, and which one belongs in a regulated readback.

Flux TTS gives you session-level expressivity and persistent context, but non-default settings increase word-level risk:

• TTS with emotion in Flux TTS lives in a beta connection control.
• Use the tuned default for accuracy-sensitive flows and audition alternatives before shipping.
• Cross-turn state carries delivery forward within the active connection.
• Streaming one sentence at a time flattens delivery, so let the server handle chunking.
• Pitch and duration carry most of an emphasis cue, so one scalar setting lands differently across voices.

TTS with emotion built for narration produces flat or erratic delivery once an LLM writes the script mid-call. A style preset assumes a person listens to every line before it ships, and a live voice agent has no such person.

Text reaches the synthesizer as the LLM produces it, one turn at a time, so there's no review step where a person could tune a line. A slider that sets "cheerful" on line 12 of a script has nowhere to attach, because line 12 doesn't exist until the customer stops talking. You have to get expressive delivery from the LLM output or the request layer. Some speech models can also supply it through their own state.

A model fed one sentence at a time can't shape emotion across a whole turn, since chunk size decides how much context it gets. LiveKit reports the same effect, with the TTS losing the emotional through-line of a response. Batching sentences into larger chunks before synthesis stabilized emotion and pitch for each turn. Flux TTS moves that boundary server-side. Deepgram's Flux TTS quickstart states that Speak messages don't require manual chunking, since the server manages prosody and sentence boundaries. You stream text as the LLM produces it and let the model decide where a phrase ends.

When every generated token affects cost, inline tags create two expenses. You pay once in prompt weight and again when a broken tag gets read aloud. Markup instructions add prompt weight before speaking. Injecting formatting instructions into the prompt is a documented way to do this, and researchers behind LPML confirm it generalizes beyond math tasks. These failures get stranger once you're streaming. A tag that gets cut off partway through can sound like the model glitching until you check what was actually sent. So when TTS with emotion lives in the text itself, you own the job of holding incomplete tags until they're complete.

Prosody control comes from one of three places: markup the LLM writes inline, a parameter you set on the connection, or state the model carries across turns. Markup can shift every turn but costs tokens and a sanitizer. A connection parameter is cheap and predictable, though fixed for the session. Model state needs no markup tokens or separate control field. It depends on the active WebSocket session and can't be steered directly.

The LLM controls delivery here by writing emotion tags directly into the text it generates. One approach has the model emit a single marker dialect that gets translated for whichever provider is active, since providers don't share a format. The translation layer also has to normalize tags, because models drift outside whatever set they're given, instead of passing them straight to the synthesizer. Choose this route when you need turn-by-turn range, a laugh on one line and a hesitation on the next, and can afford the prompt weight and buffering.

This connection-level control is Flux TTS's expressivity setting, an integer from -2 (calm) to 2 (animated), default 0, with no LLM tokens or markup involved. The model has been generally available since August 2026, but Deepgram's documentation still labels this setting beta. It's set once per connection on the streaming WebSocket, so whatever you open with holds until the socket closes. It applies only to Flux TTS on /v2/speak, not to Aura-2 on /v1/speak. Speed is a separate parameter, and Deepgram recommends auditioning the two together because large combined values "may sound less natural." Deepgram lists pause and pronunciation controls as coming soon for Flux TTS, while Aura-2 already ships pronunciation control. Pick this route when you want one consistent register across a call type.

This state carries forward automatically, so prosody and pacing continue from the prior turn's output. It's the default behavior of /v2/speak rather than a request-level setting, confirmed by Flux TTS's cross-turn context documentation. The state includes only the model's own prior generations, not the user's audio, text, or your LLM's reasoning. Ending a turn with Flush keeps it, and only a new WebSocket connection resets it. An Interrupt stops synthesis and clears the buffer without resetting the state. Deepgram's documentation still lists interruption feedback on barge-in as planned, so don't build on the feedback payload yet. On the Voice Agent API, you get this behavior by setting agent.speak.provider.version to "v2". The model can preserve coherence with turn five, but it offers no direct way to make turn six sadder. Reach for this when you want a call to hold its register without re-declaring it turn after turn.

Emphasis moves through pitch, phoneme duration, and spectral energy shifting together, not through a single gain knob. Two voices given the same setting land differently because each starts from a different baseline for these cues.

Emphasizing a syllable draws on one burst of vocal effort, which raises pitch, adds loudness, and lengthens the sound at once. Researchers behind EE-TTS confirm the payoff. Predicting where emphasis belongs from a sentence's syntax and semantics, then rendering it acoustically, lifted naturalness scores by roughly half a point over baseline models. A single expressivity value can't isolate one cue. Turning it up moves pitch, loudness, and timing as one unit, not as separate controls.

One word like "happy" covers many different performances, so one label can't capture them all. The same emotion label can vary widely in expression, since acting performances differ. A label like "sad" collapses lonely and hurt into one style. Scaling that label up or down with a single factor is just as hard to tune, and adjusting it often destabilizes audio quality.

Fixing a word's sound is separate from shaping emphasis or emotion. For drug names and specialized terms, Aura-2 gives you lexicon control, accepting inline IPA objects such as {"word": "dupilumab", "pronounce": "duːˈpɪljuːmæb"} in English and Spanish. Flux TTS is built for consequential terms like account numbers and alphanumerics, but it doesn't take a lexicon override yet. A flow needing exact IPA control has a model choice to make first. Even the right expressivity value won't fix a mispronounced word.

If your agent reads back account numbers or dosages, keep expressivity at the default and pick the register through the voice instead. In those flows, TTS with emotion comes from voice selection rather than the parameter.

The register should track the call's risk. Healthcare and financial flows carry a high cost per misread word, so they stay at the default. IVR contact center flows carry the same exposure. Aura-2 has no expressivity dial at all, so healthcare and financial deployments default to it for high-volume calls. For new real-time work, Flux TTS is the current model, so read that as a register choice rather than a model recommendation. Outbound sales and appointment reminders can tolerate a warmer register, so a warmer expressivity value may earn its place there. Even then, listen to it on your voice and your scripts first.

Non-default expressivity values raise the risk of hallucinations and pronunciation errors, according to Deepgram's Voice Agent configuration page, which also labels the parameter beta. Hallucination here includes repetitions and words the model adds or drops. The expressivity documentation also states that the tuned default is the only value validated for production. For an IVR reading a confirmation code, one repeated digit is a failed call, so a livelier voice is a bad trade. For a sales opener, the same risk is a judgment call you can make with your own ears.

Before production, audition the exact integer you plan to ship, on the exact voice, in Deepgram's online TTS playground. Use your real scripts, including the dull scripts that read back numbers. Retest and revalidate the chosen value whenever the model updates or before finalizing it for production, since a beta parameter's behavior may shift between releases.

Start from the accuracy floor you have to hold and the kind of conversation you're running, then pick the control point and validate the exact setting. Working backward from whichever demo sounded best is how a beta value ends up in a regulated flow.

The required accuracy and conversation type determine the control point.

• Regulated readbacks (healthcare, banking, IVR confirmations): expressivity at the default, cross-turn state for consistency, register chosen through the voice.
• Support and scheduling: default value first; audition a positive value only if listeners flag flatness on real transcripts.
• Outbound and conversational sales: inline markup if your provider accepts it and you can own the sanitizer, or a non-default value after auditioning.

Four checks catch most expressive-setting surprises. First, confirm the parameter reaches the model. The Voice Agent API defaults to v1 (Aura) when version is omitted, which is easy to miss in a config diff. Second, check that reconnection logic preserves the intended delivery after a dropped socket. Third, run your number-heavy scripts at the value you plan to ship and count dropped or repeated tokens. Fourth, put a re-listen on the calendar for each model update.

Then hear it on your own scripts. Create a free account with Deepgram and spend your $200 free credits on the calls that read back numbers.

You can keep separate sockets open for planned expressivity values and route complete turns to the matching connection. Each socket maintains its own cross-turn history, so switching sockets also switches delivery context. Avoid moving a partial turn between them.

Expressivity controls the emotional register of delivery, from calm to animated. Speed controls how fast the words are spoken. Test them independently. Hold expressivity at 0 while testing speed, then hold speed constant while testing expressivity. This shows whether listeners are reacting to pace or delivery register.

Yes, if you're buffering inline markup, since incomplete tags add latency while your sanitizer waits for them to close. A connection parameter like expressivity adds none. Measure latency from the first LLM token to synthesis.

Each voice has its own baseline for pitch, loudness, and timing, so the same setting lands differently. Test with identical neutral and expressive lines plus a number-heavy readback, scored independently so theatrical samples don't mask pronunciation errors.

Two connections solve this. Classify the text before synthesis, route the regulated span through one held at the default value. Use the other for the warmer closing, but only if that register change is worth losing the first socket's cross-turn history. If terminology needs an IPA override, route that span through Aura-2.