Build A Voice Agent With Pipecat And Deepgram Flux STT And Flux TTS

Last week, we launched Flux TTS, the second model in the Flux family and the first conversation-native text-to-speech model built for real-time voice agents. Where Flux STT reimagined speech recognition for live conversation, Flux TTS does the same for the speaking half: it holds the whole conversation in memory as it speaks instead of starting fresh on every line, so tone, pacing, and pronunciation stay consistent from the first turn to the tenth. No SSML, no style tags.

That launch post covers why we built it. This one is about wiring it up — Flux STT on listen, Flux TTS on speak, your LLM of choice in the middle. Both models are generally available now, and Flux TTS is free to build with through September 12, 2026.

Check out the Pipecat docs here

Building a production voice agent means choosing an orchestration framework and wiring in speech models for both directions of the conversation. A Pipecat voice agent can run Deepgram Flux for speech-to-text and Flux TTS for the voice.

A standard Pipecat quickstart needs a Silero VAD, a smart-turn analyzer, an STT service, and a third-party TTS. Flux folds transcription and turn detection into one model, so two of those components disappear. You configure fewer services and thresholds, with fewer hand-offs to debug between parts that were never designed to agree with each other.

Both halves of the stack are production-ready today. Flux STT and Flux TTS are generally available, and both support self-hosted and on-prem deployment alongside the cloud API. This guide builds a real-time voice agent that replaces two separate pipeline components with a single speech model.

Building on that, the Flux stack removes two components from the standard Pipecat pipeline before you write any application code.

• Flux handles transcription and turn detection in one model; Pipecat auto-requests ExternalUserTurnStrategies, so you skip manual turn-strategy configuration.
• Silero VAD becomes optional with Flux; keep it only if you want STT metrics.
• Deepgram's latency docs report lower agent response latency versus STT-plus-VAD pipelines.
• Pipecat includes DeepgramFluxTTSService; the default voice is flux-alexis-en.
• Flux STT and Flux TTS are both generally available, and both deploy in the cloud, self-hosted, or on-prem.

Flux combines transcription with model-native turn detection, which makes the VAD and the separate turn analyzer optional. Most quickstarts put VAD and turn-analysis logic outside the transcription service, so STT handles the words while something else handles the turn boundary.

Originally built inside Daily as internal tooling for real-time conversational AI, Pipecat is an open-source Python framework for voice and multimodal agents. Its vendor-neutral architecture lets you choose providers across the pipeline, including text-to-speech, and Daily itself is only an optional transport.

At its core, it handles orchestration across services, moving audio frames between your transport, your STT service, your LLM, and your TTS. In its default setup, it also runs a SileroVADAnalyzer plus a local smart-turn model to decide when the user has finished speaking. Those are the two pieces Flux makes redundant.

Deepgram's latency docs report lower agent response latency than traditional STT-plus-VAD setups, because the model detects turn endings earlier and more accurately. Deepgram's launch post positions Flux as conversational speech recognition built to understand when a speaker is done.

It reduces false interruptions while keeping latency low. If you set eager_eot_threshold, EagerEndOfTurn can fire before EndOfTurn at the cost of more LLM calls. Instead of a speech_final flag inferred from silence, Flux emits structured turn events from the same model that produces the transcript.

The architecture changes because the thing deciding whether you're done talking has actually understood what you said.

Once the LLM produces text, something has to say it without blowing the latency budget. Flux TTS is Deepgram's Speak v2 synthesis model, exposed in the framework as DeepgramFluxTTSService. Voices use the flux-{voice}-{lang} naming pattern.

The cascade route described here gives you component-level control over the LLM and turn thresholds while exposing per-service metrics.

Three API keys and one CLI command take you from an empty directory to a scaffolded agent. Deepgram's official Pipecat integration guide uses the same setup path.

Before you run the CLI commands below, make sure you have Python 3.11+ and uv installed. Then install the CLI and scaffold:

uv tool install "pipecat-ai[cli]"
pipecat init
cd pipecat-bot
pipecat create --name pipecat-deepgram --bot-type web --transport daily --mode cascade \
  --stt deepgram_flux_stt --llm openai_llm --tts deepgram_tts --no-deploy-to-cloud

The create command wires Flux STT, an OpenAI LLM, and Deepgram TTS into a cascade over a Daily WebRTC transport. Finish with cd pipecat-deepgram/server and uv sync to install dependencies. It's a rare thing in voice AI: a scaffold command that actually leaves you with a working demo instead of a pile of stub files.

Both Flux STT and Flux TTS read the same DEEPGRAM_API_KEY, created in your Deepgram dashboard. You'll also need an OpenAI API key for the LLM and a Daily API key for the scaffolded WebRTC transport.

Managing three keys for a first run is simpler than it might seem at first. Thankfully, though, each one maps to exactly one service, so a 401 tells you immediately which key to check. The Daily key belongs to the transport. For more information, check out the pipecat integration docs here.

With all three keys in hand, copy the template with cp .env.example .env, then fill in four values:

• DEEPGRAM_API_KEY: your Deepgram key
• DEEPGRAM_VOICE_ID: the scaffold's variable for the legacy
• DeepgramTTSService, using the older Aura-2-era voice format; leave a value in it, because an empty string may return a 400 error
• OPENAI_API_KEY: from your OpenAI dashboard
• DAILY_API_KEY: from your Daily dashboard

With your project scaffolded and credentials in place, the STT side of the pipeline needs just one change. Swap in one constructor and the external turn-control pieces drop out.

Replacing a generic STT service with DeepgramFluxSTTService

Inside bot.py, the scaffold already uses the Flux service because you passed --stt deepgram_flux_stt. If you're migrating an existing bot, the swap looks like this:

from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

stt = DeepgramFluxSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    settings=DeepgramFluxSTTService.Settings(
        min_confidence=0.3,
    ),
)

Use the settings= object, not the flat model and params constructor arguments. Both were deprecated in 0.0.105, with removal targeted at 2.0.0, so code written against them will break on that major version. Defaults are sensible: model flux-general-en, and server-side end-of-turn detection keeps the pipeline simpler.

Handling turn events: start, end, and eager end of turn

Four handlers give you hooks into the conversation: on_start_of_turn, on_end_of_turn, on_eager_end_of_turn, and on_turn_resumed. StartOfTurn is your barge-in signal; Deepgram's Flux API reference describes describes it as more reliable than external VAD and guaranteed to carry a non-empty transcript.

The eager pair only fires when you set eager_eot_threshold. Use it only when reducing response latency is worth the extra LLM work. TurnResumed then tells you the user kept talking.

The service source shows that the framework doesn't yet ship built-in gating to start LLM or TTS work early on EagerEndOfTurn or cancel it on TurnResumed. Eager handling is yours to build for now.

Skipping external VAD and turn detectors

You can drop the Silero import for turn control. The framework's service docs say Flux "automatically requests ExternalUserTurnStrategies at start," so you don't configure turn strategies by hand.

The same docs mark VAD as optional when Flux drives turn detection; include a SileroVADAnalyzer only if you want the STT metrics it feeds. Per the v1.6.0 changelog, STT services with server-side end-of-turn detection recommend ExternalUserTurnStrategies automatically, and your own setting still wins if you set one.

Adding Flux TTS and an LLM to complete the loop

The v1.6.0 release notes say the framework ships a dedicated DeepgramFluxTTSService, distinct from the older DeepgramTTSService that targets legacy Aura-2 voices. Check which one you're importing.

Configuring DeepgramFluxTTSService and voice selection

The voice catalog is English-only for now, all in the flux-{voice}-{lang} format per Deepgram's voice docs. Configure the service like this:

from pipecat.services.deepgram.flux.tts import DeepgramFluxTTSService

tts = DeepgramFluxTTSService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    settings=DeepgramFluxTTSService.Settings(voice="flux-alexis-en")
)

Here, flux-alexis-en is the default voice, and the settings= object is where the Flux voice lives; DEEPGRAM_VOICE_ID can be used to override the voice from your .env settings otherwise it uses the default.

The service streams from the Speak v2 WebSocket at https://api.deepgram.com/v2/speak, which produces raw audio only. The Flux TTS quickstart lists linear16 (the default), mulaw, and alaw encodings, and the streaming endpoint rejects batch-only parameters like container.

Connecting an LLM for response generation

You already wired in OpenAI with the --llm openai_llm flag, so there's nothing to add for a first run. The framework's modular service architecture lets you swap LLM providers without changing pipeline code, so your Flux configuration survives a later model change.

At runtime, Flux emits a final transcript when it detects the end of the user's turn, the LLM streams tokens, and Flux TTS speaks them as they arrive.

Testing the full cascade locally

Run uv run python bot.py --transport daily to start the agent. Interrupt it mid-sentence while you test, since should_interrupt defaults to True and the bot should stop the moment you speak.

It's a small, satisfying moment when you talk over the bot and it actually shuts up. One catch: standard interruption_strategies such as MinWordsInterruptionStrategydon't work as expected with Flux, because the service pushes an InterruptionTaskFrameupstream instead of letting the transport control interruption (issue #2988).

For production, pick a transport, decide whether Flux should run in your own infrastructure, and watch turn detection under real traffic. It's also the moment to re-check whether a cascade or speech-to-speech architecture fits your call volume.

If Daily's WebRTC infrastructure doesn't fit, alternatives include: FastAPI WebSocket, LiveKit WebRTC, Vonage WebRTC, and the keyless peer-to-peer SmallWebRTCTransport.

Telephony runs through WebSocket serializers for Twilio, Telnyx, Plivo, Exotel, Genesys, and Vonage. For managed hosting, Pipecat Cloud reached general availability on January 8, 2026, and Daily states that anything you run there can be self-hosted exactly the same way.

Audio never has to leave your network. Deepgram documents self-hosted Flux with two variants, flux-general-en and flux-general-multi, per the deployment environments docs.

The variants run in private infrastructure, including VPCs and dedicated cloud or bare-metal deployments. Self-hosting requires a Deepgram Enterprise Plan. For AWS-native teams, Flux is a supported model family on Amazon SageMaker, where the container runs network-isolated in your own AWS VPC with no connection to the Deepgram Cloud.

Capacity planning runs off the engine_flux_max_streams and engine_flux_used_streams metrics the self-hosted Engine exposes, and the SageMaker container tunes concurrency with flux.max_streams=25.

Flux TTS follows the same path. It deploys in the cloud, self-hosted, or on-prem, with the same model and the same benchmarks across all three, so the speaking half of the pipeline can run under the same HIPAA, data-residency, and enterprise compliance rules as the listening half.

Use enable_metrics=True in PipelineParams to collect per-service TTFB and processing time, plus time-to-first-audio for TTS (metrics docs). Add a UserBotLatencyObserver to measure how long the user waits between going quiet and hearing the bot respond, which is the number callers actually feel.

On the Deepgram side, track percentiles rather than averages. A mean hides the tail, and the tail is where turn detection fails. Deepgram recommends tracking percentile latency at p50, p95, and p99.

It also recommends watching how often EagerEndOfTurn resolves to TurnResumed versus EndOfTurn to judge your eager threshold. Use the Flux baseline as a starting point, then compare it with your own traffic.

Test turn detection against your own callers' audio. Create a free account, grab your $200 free credits, and test Flux with your own callers' audio.

Check out the Pipecat docs here

Choose Flux when turn detection is part of the speech model; Nova-3 fits transcription-heavy workflows where your pipeline already owns turn boundaries. Deepgram's models overview recommends Flux for real-time agents and interactive, turn-based experiences.

It recommends Nova-3 for meetings and captioning, along with workflows that involve multi-speaker or noisy audio. Flux streams over /v2/listen; Nova-3 uses /v1/listen. For migrations, treat the endpoint change as a pipeline behavior change. Flux is built for real-time voice agents while keeping word error rate low.

Yes. Deepgram's Flux Multilingual announcement confirms partner integrations with Twilio, Vapi, LiveKit, Pipecat, and Jambonz, so the same model travels with you if you later switch frameworks.

For turn control, yes, but test the handoff before deleting every VAD-related line. Log on_start_of_turn, on_end_of_turn, and on_turn_resumed, then compare those events with the actual audio.

If the bot starts listening, stops listening, and handles resumed speech correctly, Flux is owning turn management. Keep Silero only when you need VAD-derived STT latency measurements; maintainer guidance in issue #4279 confirms that role.

Check out the Pipecat docs here

Treat voice IDs as configuration, not code constants. The catalog will keep growing — more languages and voice cloning are on the roadmap — so keep the selected voice in an environment variable or config file and pass it into DeepgramFluxTTSService.Settings(voice=...).

Use flux-alexis-en as your local default, and make an unsupported voice fail loudly during startup rather than mid-call.

Yes, on both sides of the pipeline. Treat the STT side as a capacity-planning project before you ship: choose a supported GPU, avoid NVIDIA T4, and keep Flux on its own Engine node. If the container starts but streams fail under load, check Engine stream capacity before tuning turn thresholds. Flux TTS deploys in the cloud, self-hosted, or on-prem as well, so both halves of the agent can run inside the same compliance boundary.