---
title: "Build a voice agent with Pipecat and Deepgram Flux"
description: "Learn how to build a voice agent using Pipecat with Deepgram Flux STT and Flux TTS. Skip VAD and turn detection setup with this step-by-step guide."
canonical: "https://deepgram.com/blog/build-voice-agent-pipecat-deepgram-flux-stt-tts"
authors:
  - "Deepgram"
date: "2026-08-13"
category: "AI Engineering & Research"
---

# Build a voice agent with Pipecat and Deepgram Flux STT and Flux TTS

**Last week, we launched Flux TTS, the second model in the Flux family and the first conversation-native text-to-speech model built for real-time voice agents.** Where Flux STT reimagined speech recognition for live conversation, Flux TTS does the same for the speaking half: it holds the whole conversation in memory as it speaks instead of starting fresh on every line, so tone, pacing, and pronunciation stay consistent from the first turn to the tenth. No SSML, no style tags.

That launch post covers why we built it. This one is about wiring it up — Flux STT on listen, Flux TTS on speak, your LLM of choice in the middle. Both models are generally available now, and Flux TTS is free to build with through September 12, 2026.

[Check out the Pipecat docs here](https://docs.pipecat.ai/api-reference/server/services/tts/deepgram#deepgram-text-to-speech)

### Introduction

Building a production voice agent means choosing an orchestration framework and wiring in speech models for both directions of the conversation. A [Pipecat](https://docs.pipecat.ai/pipecat/get-started/introduction) voice agent can run [Deepgram Flux](https://developers.deepgram.com/docs/flux/quickstart) for [speech-to-text](https://deepgram.com/product/speech-to-text) and [Flux TTS](https://developers.deepgram.com/docs/flux-tts/overview) for the voice.

A standard Pipecat quickstart needs a [Silero VAD](https://docs.pipecat.ai/api-reference/server/utilities/audio/silero-vad-analyzer), a smart-turn analyzer, an STT service, and a third-party TTS. Flux folds transcription and turn detection into one model, so two of those components disappear. You configure fewer services and thresholds, with fewer hand-offs to debug between parts that were never designed to agree with each other.

Both halves of the stack are production-ready today. Flux STT and Flux TTS are generally available, and both support self-hosted and on-prem deployment alongside the cloud API. This guide builds a real-time voice agent that replaces two separate pipeline components with a single speech model.

## Key takeaways

Building on that, the Flux stack removes two components from the standard Pipecat pipeline before you write any application code.

- Flux handles transcription and turn detection in one model; Pipecat auto-requests `ExternalUserTurnStrategies`, so you skip manual turn-strategy configuration.
- Silero VAD becomes optional with Flux; keep it only if you want STT metrics.
- Deepgram's latency docs report lower agent response latency versus STT-plus-VAD pipelines.
- Pipecat includes `DeepgramFluxTTSService`; the default voice is `flux-alexis-en`.
- Flux STT and Flux TTS are both generally available, and both deploy in the cloud, self-hosted, or on-prem.

## Why Pipecat and Deepgram Flux fit together for production voice agents

Flux combines transcription with model-native turn detection, which makes the VAD and the separate turn analyzer optional. Most quickstarts put VAD and turn-analysis logic outside the transcription service, so STT handles the words while something else handles the turn boundary.

### What Pipecat handles in the pipeline

Originally built inside [Daily](https://www.daily.co/blog/pipecat-cloud-is-now-generally-available/) as internal tooling for real-time conversational AI, Pipecat is an open-source Python framework for voice and multimodal agents. Its vendor-neutral architecture lets you choose providers across the pipeline, including [text-to-speech](https://deepgram.com/product/text-to-speech), and Daily itself is only an optional transport.

At its core, it handles [orchestration across services](https://docs.pipecat.ai/api-reference/server/services/supported-services), moving audio frames between your transport, your STT service, your LLM, and your TTS. In its [default setup](https://docs.pipecat.ai/pipecat/learn/speech-input), it also runs a `SileroVADAnalyzer` plus a local smart-turn model to decide when the user has finished speaking. Those are the two pieces Flux makes redundant.

### What Flux adds that a standard STT service doesn't

Deepgram's [latency docs](https://developers.deepgram.com/docs/measuring-streaming-latency) report lower agent response latency than traditional `STT-plus-VAD` setups, because the model detects turn endings earlier and more accurately. Deepgram's [launch post](https://deepgram.com/learn/introducing-flux-conversational-speech-recognition) positions Flux as conversational speech recognition built to understand when a speaker is done.

It reduces false interruptions while keeping latency low. If you set `eager_eot_threshold`, `EagerEndOfTurn` can fire before `EndOfTurn` at the cost of more LLM calls. Instead of a `speech_final` flag inferred from silence, Flux emits structured turn events from the same model that produces the transcript.

The architecture changes because the thing deciding whether you're done talking has actually understood what you said.

### Where Flux TTS fits as the voice

Once the LLM produces text, something has to say it without blowing the latency budget. Flux TTS is Deepgram's Speak v2 synthesis model, exposed in the framework as `DeepgramFluxTTSService`. Voices use the `flux-{voice}-{lang}` naming pattern.

The cascade route described here gives you component-level control over the LLM and turn thresholds while exposing per-service metrics.

## Setting up your Pipecat project and Deepgram credentials

Three API keys and one CLI command take you from an empty directory to a scaffolded agent. Deepgram's official Pipecat integration guide uses the same setup path.

### Installing the Pipecat CLI and scaffolding a project

Before you run the CLI commands below, make sure you have [Python 3.11+ and](https://developers.deepgram.com/docs/pipecat-integration) `uv` installed. Then install the CLI and scaffold:

```bash
uv tool install "pipecat-ai[cli]"
pipecat init
cd pipecat-bot
pipecat create --name pipecat-deepgram --bot-type web --transport daily --mode cascade \
  --stt deepgram_flux_stt --llm openai_llm --tts deepgram_tts --no-deploy-to-cloud

```

The `create` command wires Flux STT, an [OpenAI LLM](https://platform.openai.com/api-keys), and Deepgram TTS into a cascade over a Daily [WebRTC transport](https://docs.pipecat.ai/api-reference/server/services/transport/daily). Finish with `cd pipecat-deepgram/server` and `uv sync` to install dependencies. It's a rare thing in voice AI: a scaffold command that actually leaves you with a working demo instead of a pile of stub files.

### Getting your Deepgram API key

Both Flux STT and Flux TTS read the same `DEEPGRAM_API_KEY`, created in your Deepgram dashboard. You'll also need an OpenAI API key for the LLM and a Daily API key for the scaffolded WebRTC transport.

Managing three keys for a first run is simpler than it might seem at first. Thankfully, though, each one maps to exactly one service, so a 401 tells you immediately which key to check. The Daily key belongs to the transport. For more information, check out the pipecat integration docs [here](https://developers.deepgram.com/docs/pipecat-integration).

### Configuring environment variables

With all three keys in hand, copy the template with `cp .env.example .env`, then fill in four values:

- `DEEPGRAM_API_KEY`: your Deepgram key
- `DEEPGRAM_VOICE_ID`: the scaffold's variable for the legacy
- `DeepgramTTSService`, using the older Aura-2-era voice format; leave a value in it, because an empty string may return a 400 error
- `OPENAI_API_KEY`: from your OpenAI dashboard
- `DAILY_API_KEY`: from your Daily dashboard

### Wiring Flux speech-to-text into the pipeline

With your project scaffolded and credentials in place, the STT side of the pipeline needs just one change. Swap in one constructor and the external turn-control pieces drop out.

#### Replacing a generic STT service with DeepgramFluxSTTService

Inside `bot.py`, the scaffold already uses the Flux service because you passed `--stt deepgram_flux_stt`. If you're migrating an existing bot, the swap looks like this:

```python
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

stt = DeepgramFluxSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    settings=DeepgramFluxSTTService.Settings(
        min_confidence=0.3,
    ),
)

```

Use the `settings=` object, not the flat `model` and `params` constructor arguments. Both were [deprecated in 0.0.105](https://reference-server.pipecat.ai/en/stable/api/pipecat.services.deepgram.flux.stt.html), with removal targeted at 2.0.0, so code written against them will break on that major version. Defaults are sensible: model `flux-general-en`, and server-side end-of-turn detection keeps the pipeline simpler.

#### Handling turn events: start, end, and eager end of turn

Four handlers give you hooks into the conversation: `on_start_of_turn`, `on_end_of_turn`, `on_eager_end_of_turn`, and `on_turn_resumed`. `StartOfTurn` is your barge-in signal; Deepgram's [Flux API reference](https://developers.deepgram.com/reference/speech-to-text/listen-flux) describes describes it as more reliable than external VAD and guaranteed to carry a non-empty transcript.

The eager pair only fires when you set `eager_eot_threshold`. Use it only when reducing response latency is worth the extra LLM work. `TurnResumed` then tells you the user kept talking.

The [service source](https://github.com/pipecat-ai/pipecat/blob/dc97ffc9/src/pipecat/services/deepgram/flux/stt.py) shows that the framework doesn't yet ship built-in gating to start LLM or TTS work early on `EagerEndOfTurn` or cancel it on `TurnResumed`. Eager handling is yours to build for now.

#### Skipping external VAD and turn detectors

You can drop the Silero import for turn control. The framework's [service docs](https://docs.pipecat.ai/api-reference/server/services/stt/deepgram) say Flux "automatically requests ExternalUserTurnStrategies at start," so you don't configure turn strategies by hand.

The same docs mark VAD as optional when Flux drives turn detection; include a `SileroVADAnalyzer` only if you want the STT metrics it feeds. Per the [v1.6.0 changelog](https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md), STT services with server-side end-of-turn detection recommend `ExternalUserTurnStrategies` automatically, and your own setting still wins if you set one.

#### Adding Flux TTS and an LLM to complete the loop

The [v1.6.0 release notes](https://github.com/pipecat-ai/pipecat/releases/tag/v1.6.0) say the framework ships a dedicated `DeepgramFluxTTSService`, distinct from the older `DeepgramTTSService` that targets legacy Aura-2 voices. Check which one you're importing.

#### Configuring DeepgramFluxTTSService and voice selection

The voice catalog is English-only for now, all in the `flux-{voice}-{lang}` format per Deepgram's [voice docs](https://developers.deepgram.com/docs/flux-tts/voices). Configure the service like this:

```python
from pipecat.services.deepgram.flux.tts import DeepgramFluxTTSService

tts = DeepgramFluxTTSService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    settings=DeepgramFluxTTSService.Settings(voice="flux-alexis-en")
)

```

Here, `flux-alexis-en` is the default voice, and the `settings= object` is where the Flux voice lives; `DEEPGRAM_VOICE_ID` can be used to override the voice from your `.env` settings otherwise it uses the default.

The service streams from the Speak v2 WebSocket at `https://api.deepgram.com/v2/speak`, which produces raw audio only. The [Flux TTS quickstart](https://developers.deepgram.com/docs/flux-tts/quickstart) lists `linear16` (the default), `mulaw`, and `alaw` encodings, and the streaming endpoint rejects batch-only parameters like `container`.

#### Connecting an LLM for response generation

You already wired in OpenAI with the `--llm openai_llm` flag, so there's nothing to add for a first run. The framework's modular service architecture lets you swap [LLM providers](https://developers.deepgram.com/docs/voice-agent-llm-models) without changing pipeline code, so your Flux configuration survives a later model change.

At runtime, Flux emits a final transcript when it detects the end of the user's turn, the LLM streams tokens, and Flux TTS speaks them as they arrive.

#### Testing the full cascade locally

Run `uv run python bot.py --transport daily` to start the agent. Interrupt it mid-sentence while you test, since `should_interrupt` defaults to `True` and the bot should stop the moment you speak.

It's a small, satisfying moment when you talk over the bot and it actually shuts up. One catch: standard `interruption_strategies` such as `MinWordsInterruptionStrategy`don't work as expected with Flux, because the service pushes an `InterruptionTaskFrame`upstream instead of letting the transport control interruption ([issue #2988](https://github.com/pipecat-ai/pipecat/issues/2988)).

### Deploying and scaling your Pipecat voice agent

For production, pick a transport, decide whether Flux should run in your own infrastructure, and watch turn detection under real traffic. It's also the moment to re-check whether a cascade or speech-to-speech architecture fits your call volume.

### Choosing a transport for production

If Daily's WebRTC infrastructure doesn't fit, alternatives include: [FastAPI WebSocket](https://docs.pipecat.ai/api-reference/server/services/transport/fastapi-websocket), [LiveKit WebRTC](https://docs.pipecat.ai/api-reference/server/services/transport/livekit), [Vonage WebRTC](https://docs.pipecat.ai/api-reference/server/services/transport/vonage), and the [keyless peer-to-peer](https://docs.pipecat.ai/api-reference/server/services/transport/small-webrtc) SmallWebRTCTransport.

Telephony runs through [WebSocket serializers](https://github.com/pipecat-ai/pipecat/blob/06de5cb2/README.md) for [Twilio](https://www.twilio.com/), Telnyx, Plivo, Exotel, [Genesys](https://www.genesys.com/), and Vonage. For managed hosting, Pipecat Cloud reached general availability on January 8, 2026, and Daily states that anything you run there can be self-hosted exactly the same way.

### Self-hosted and VPC options for compliance-sensitive teams

Audio never has to leave your network. Deepgram documents [self-hosted Flux](https://developers.deepgram.com/docs/flux-self-hosted) with two variants, `flux-general-en` and `flux-general-multi`, per the [deployment environments docs](https://developers.deepgram.com/docs/self-hosted-deployment-environments).

The variants run in private infrastructure, including VPCs and dedicated cloud or bare-metal deployments. Self-hosting requires a [Deepgram Enterprise Plan](https://deepgram.com/enterprise). For [AWS-native teams](https://aws.amazon.com/), Flux is a supported model family on [Amazon SageMaker](https://developers.deepgram.com/docs/deploy-amazon-sagemaker), where the container runs network-isolated in your own AWS VPC with no connection to the Deepgram Cloud.

Capacity planning runs off the `engine_flux_max_streams` and `engine_flux_used_streams` metrics the self-hosted Engine exposes, and the SageMaker container tunes concurrency with `flux.max_streams=25`.

Flux TTS follows the same path. It deploys in the cloud, self-hosted, or on-prem, with the same model and the same benchmarks across all three, so the speaking half of the pipeline can run under the same HIPAA, data-residency, and enterprise compliance rules as the listening half.

### Monitoring latency and turn detection in production

Use `enable_metrics=True` in PipelineParams to collect per-service TTFB and processing time, plus time-to-first-audio for TTS ([metrics docs](https://docs.pipecat.ai/pipecat/fundamentals/metrics)). Add a `UserBotLatencyObserver` to measure how long the user waits between going quiet and hearing the bot respond, which is the number callers actually feel.

On the Deepgram side, [track percentiles](https://developers.deepgram.com/docs/flux/voice-agent-eager-eot) rather than averages. A mean hides the tail, and the tail is where turn detection fails. Deepgram recommends tracking percentile latency at p50, p95, and p99.

It also recommends watching how often `EagerEndOfTurn` resolves to `TurnResumed` versus `EndOfTurn` to judge your eager threshold. Use the Flux baseline as a starting point, then compare it with your own traffic.

Test turn detection against your own callers' audio. [Create a free account](https://console.deepgram.com/signup), grab your $200 free credits, and test Flux with your own callers' audio.

[Check out the Pipecat docs here](https://docs.pipecat.ai/api-reference/server/services/tts/deepgram#deepgram-text-to-speech)

## FAQ

### What's the difference between Deepgram Flux and Nova-3 for voice agents?

Choose Flux when turn detection is part of the speech model; Nova-3 fits transcription-heavy workflows where your pipeline already owns turn boundaries. Deepgram's [models overview](https://developers.deepgram.com/docs/models-languages-overview) recommends Flux for real-time agents and interactive, turn-based experiences.

It recommends Nova-3 for meetings and captioning, along with workflows that involve multi-speaker or noisy audio. Flux streams over `/v2/listen`; Nova-3 uses `/v1/listen`. For migrations, treat the endpoint change as a pipeline behavior change. Flux is built for real-time voice agents while keeping [word error rate](https://deepgram.com/learn/what-is-word-error-rate) low.

### Does Pipecat support Deepgram Flux out of the box?

Yes. Deepgram's [Flux Multilingual announcement](https://deepgram.com/learn/introducing-flux-multilingual) confirms partner integrations with Twilio, Vapi, LiveKit, Pipecat, and Jambonz, so the same model travels with you if you later switch frameworks.

### Can I skip external VAD entirely when using Flux in Pipecat?

For turn control, yes, but test the handoff before deleting every VAD-related line. Log `on_start_of_turn`, `on_end_of_turn`, and `on_turn_resumed`, then compare those events with the actual audio.

If the bot starts listening, stops listening, and handles resumed speech correctly, Flux is owning turn management. Keep Silero only when you need VAD-derived STT latency measurements; maintainer guidance in [issue #4279](https://github.com/pipecat-ai/pipecat/issues/4279) confirms that role.

[Check out the Pipecat docs here](https://docs.pipecat.ai/api-reference/server/services/tts/deepgram#deepgram-text-to-speech)

### What Flux TTS voice options work with the Flux TTS service?

Treat voice IDs as configuration, not code constants. The catalog will keep growing — more languages and voice cloning are on the roadmap — so keep the selected voice in an environment variable or config file and pass it into `DeepgramFluxTTSService.Settings(voice=...)`.

Use `flux-alexis-en` as your local default, and make an unsupported voice fail loudly during startup rather than mid-call.

### Can Flux run in a self-hosted or VPC environment?

Yes, on both sides of the pipeline. Treat the STT side as a capacity-planning project before you ship: choose a supported GPU, avoid NVIDIA T4, and keep Flux on its own Engine node. If the container starts but streams fail under load, check Engine stream capacity before tuning turn thresholds. Flux TTS deploys in the cloud, self-hosted, or on-prem as well, so both halves of the agent can run inside the same compliance boundary.

## Related articles

- Anam solves turn-taking for unscripted avatar conversation with Flux STT (AI Agents)
- Audio, decision, composition: Five open-source Jev projects (AI Engineering & Research)
- Introducing Nova-3 Medical multilingual: Medical speech recognition in 10 languages (Announcements)
- AI voice agent to human handoffs: How to warm transfer calls without losing context (AI Engineering & Research)