Connecting TTS to Twilio: The Complete Guide to Audio Encoding, Streaming, and Production Gotchas

This guide covers exactly what you need to connect an external text-to-speech provider to Twilio Media Streams pipelines: the correct audio encoding parameters, the JSON message structure for outbound audio, and the barge-in abort logic that keeps your voice agent responsive after interruptions. The encoding constraints, container headers, and stream lifecycle management on the outbound side introduce failure modes that only surface during live calls.

In production, those failure modes have direct business costs. If you don't abort synthesis on barge-in, you generate audio that never plays while the caller has already moved on—and because Deepgram TTS is usage-billed by characters, that wasted synthesis adds up fast. Twilio's own latency benchmarks peg the complete mouth-to-ear pipeline at roughly 1.1 seconds. Every millisecond lost to encoding mismatches or blocked queues pushes your agent further from natural conversational timing.

Here's what makes or breaks Twilio Media Streams TTS:

• Generate raw 8kHz mu-law: encoding=mu-law&sample_rate=8000&container=none.
• Use <Connect><Stream> for bidirectional audio; <Start> can't play audio back.
• Send base64 frames of exactly 160 bytes (20ms) and include the correct streamSid.
• Serialize sequenceNumber, chunk, and timestamp as strings.
• On barge-in, send Twilio clear, tell Deepgram to stop, and drop late audio.

Audio encoding mismatches are the most common cause of silent failures in Twilio Media Streams TTS pipelines—you often get static or silence instead of a clear error message.

The Twilio Media Streams WebSocket spec defines a narrow audio format: audio/x-mu-law at 8,000 Hz, mono, base64-encoded, sent in fixed 20ms intervals. This isn't configurable. Your pipeline has to produce audio that matches this format byte-for-byte.

The outbound message payload must contain raw audio samples only. No container headers, no metadata, no WAV wrappers. Twilio interprets every byte in the payload as audio data, so any non-audio bytes get played as noise.

Most TTS APIs default to output formats that don't match telephony: higher sample rates, different codecs, and containers like WAV or MP3. If you do the transcoding yourself, you add latency and another failure point.

The practical impact is familiar: your WebSocket accepts messages, your logs look fine, and the caller hears crackling noise or dead air because the audio format was wrong.

When a TTS API returns audio with a WAV container, the header bytes contain metadata like sample rate, bit depth, and channel count. If you base64-encode and send those bytes to Twilio, the first fraction of every utterance plays as a pop or click. Short phrases may sound entirely garbled.

This is where container=none becomes critical. If your provider can't emit raw samples, you'll need to strip headers (or transcode) before you base64-encode anything.

The fastest path is to request Twilio-compatible audio directly from Deepgram, so you don't resample or strip headers in your own stack.

Three query parameters produce Twilio-compatible output from Deepgram's text-to-speech API:

• encoding=mu-law: mu-law encoding, the telephony standard
• sample_rate=8000: 8kHz sample rate
• container=none: raw audio output without WAV headers

The complete request looks like this:

POST https://api.deepgram.com/v1/speak?model=aura-asteria-en&encoding=mu-law&sample_rate=8000&container=none

All three parameters are required. Miss container=none and you send header bytes. Miss encoding=mu-law and you send the wrong codec. Miss sample_rate=8000 and Twilio interprets your audio at the wrong speed.

Deepgram supports both REST and WebSocket synthesis. For live telephony, WebSocket is usually the right choice—you can start playing audio while synthesis is still happening. For voicemail drops, agent greetings, or other non-interactive audio, buffered output is simpler: validate the full audio before sending anything into a call.

Deepgram's TTS latency docs report about 277ms time-to-first-byte on persistent connections. The practical takeaway: create the Deepgram WebSocket when the call starts, not when the LLM produces the first response.

To inject TTS audio into a live Twilio call, you need a bidirectional stream and a single place in your codebase that owns framing and cancellation.

Twilio has two streaming modes:

• <Start><Stream> is unidirectional (call audio to your server). You can't inject audio back.
• <Connect><Stream> is bidirectional (call audio to your server and your audio back into the call).

If you pick <Start> for a voice agent, you'll build a great speech-to-text pipeline and then wonder why your TTS never plays.

<Response>
    <Connect>
        <Stream url="wss://your-server.com/bidirectional" />
    </Connect>
    <Say>The voice bot session has ended</Say>
</Response>

When Twilio opens the bidirectional WebSocket, the start event includes a streamSid that uniquely identifies the media stream. Every outbound media message must include the correct streamSid, or Twilio drops it.

In production, you'll typically manage two WebSocket connections per call: one to Twilio and one to Deepgram. Treat outbound audio as a single writer that owns 20ms chunk sizing, sequence numbers, and cancellation. Below is a minimal Node.js/TypeScript pseudocode example that accumulates raw mu-law bytes and emits Twilio media messages only when it has a full telephony frame.

// Twilio expects 20ms of 8kHz mu-law per outbound message.
// At 8,000 samples/sec and 1 byte/sample (mu-law), that is 160 bytes per frame.
const TWILIO_FRAME_BYTES = 160;

type TwilioSend = (msg: unknown) => void;

class TwilioOutboundMux {
  private sequenceNumber = 1;
  private chunk = 1;
  private startedAtMs = Date.now();
  private buffer = Buffer.alloc(0);

  constructor(private send: TwilioSend, public readonly streamSid: string) {}

  pushmu-law(bytes: Buffer) {
    this.buffer = Buffer.concat([this.buffer, bytes]);

    while (this.buffer.length >= TWILIO_FRAME_BYTES) {
      const frame = this.buffer.subarray(0, TWILIO_FRAME_BYTES);
      this.buffer = this.buffer.subarray(TWILIO_FRAME_BYTES);

      this.send({
        event: "media",
        sequenceNumber: String(this.sequenceNumber++),
        media: {
          track: "outbound",
          chunk: String(this.chunk++),
          timestamp: String(Date.now() - this.startedAtMs),
          payload: frame.toString("base64"),
        },
        streamSid: this.streamSid,
      });
    }
  }

  reset() {
    this.buffer = Buffer.alloc(0);
  }
}

function handleBargeIn(
  twilioWs: { send: (s: string) => void },
  deepgramWs: { send: (s: string) => void },
  mux: TwilioOutboundMux
) {
  // Stop Twilio playback, stop TTS generation, and drop buffered audio.
  twilioWs.send(JSON.stringify({ event: "clear", streamSid: mux.streamSid }));
  deepgramWs.send(JSON.stringify({ type: "Clear" }));
  mux.reset();
}

Twilio expects this exact JSON structure for outbound audio (field names, nesting, and types). The authoritative reference is the WebSocket spec.

{
  "event": "media",
  "sequenceNumber": "4",
  "media": {
    "track": "outbound",
    "chunk": "4",
    "timestamp": "80",
    "payload": "<base64-encoded-raw-mu-law-audio>"
  },
  "streamSid": "MZ18ad3ab5a11c26b0a2ee"
}

Every field is required. All numeric values (sequenceNumber, chunk, timestamp) are strings, not integers. The payload is base64-encoded raw mu-law audio only.

Barge-in only works in production if you clear Twilio's outbound buffer and also stop the upstream TTS stream. Doing just one of those creates stalls and "why is my bot ignoring me?" bugs.

When your STT pipeline detects the caller speaking during TTS playback, send a clear message to Twilio to flush any buffered outbound audio:

{
  "event": "clear",
  "streamSid": "MZ18ad3ab5a11c26b0a2ee"
}

Send this as soon as you detect speech, not after you've finalized a transcript. Even a small delay can leak a fragment of the old response into the new turn.

A common production failure: you clear Twilio's buffer, but keep reading and forwarding TTS audio that's now irrelevant. Coordinated cleanup fixes this:

• Tell Deepgram to stop generation: { "type": "Clear" }
• Resolve or cancel any pending work waiting for TTS completion
• Drop late audio chunks instead of forwarding them

You need explicit state to avoid race conditions between "clear," "tts aborted," and "new user turn." In practice, a small state machine (IDLE, PLAYING, INTERRUPTED, CLEARING) is enough to keep your async queue from deadlocking.

Most demo pipelines break for predictable reasons: queue buildup, dead WebSockets, and formatting edge cases. Plan for these early so you're not debugging them on a live contact center.

Twilio emits buffer overflow errors when audio can't be processed in time. The gotcha is that format mismatches can look like congestion. If you accidentally send 16kHz audio into an 8kHz stream, you may see overflow-style errors even though your network is fine.

A quick guardrail: validate that you're sending exactly 160 mu-law bytes per outbound frame before you chase infrastructure problems. That's it—no tricks.

Deepgram WebSocket connections can time out during conversational pauses. You'll want keepalives on your TTS connection any time the user might think for a few seconds.

On the Twilio side, don't build "reconnect and resume" logic into an active media stream. Treat a dropped Twilio WebSocket as a call-ending event and design your availability around that constraint.

Even with high-quality voices, phone audio exposes formatting issues. Currency, phone numbers, and addresses typically need smart formatting or SSML.

Where you'll still need SSML: long alphanumeric IDs (like "4B2X9Z3L") and ambiguous abbreviations. Use character-by-character speech for IDs, and reserve phoneme overrides for names your agent says often.

You can catch most Twilio Media Streams TTS bugs with a short, repeatable validation pass: verify bytes, verify pacing, and stress barge-in.

Run through these checks before going live:

• Confirm TTS output is mu-law at 8kHz with no container headers by inspecting raw bytes before base64 encoding
• Verify all outbound media message fields are strings, including sequenceNumber, chunk, and timestamp
• Test barge-in by speaking over the agent mid-utterance and confirming the TTS stream aborts cleanly
• Confirm the streamSid persists correctly across the full call lifecycle
• Log and cap your outbound audio queue depth so a stuck stream can't accumulate unbounded memory
• Validate interruption cleanup thoroughly: after barge-in, confirm you stop forwarding late TTS chunks (not just that you sent Twilio a clear)

Track these per call (not just averages):

• Time from user silence detection to first TTS byte arriving at your Twilio sender
• Time from barge-in detection to your clear being sent
• WebSocket round-trip time between your server and both Twilio and Deepgram

Deepgram's Twilio TTS guide includes working code for the full bidirectional pipeline.

If you'd rather not own stream management and barge-in logic yourself, the Voice Agent API handles it for you.

Ready to test this in your own stack? Sign up in the Deepgram Console to get $200 in free credits—no credit card required.

No. Twilio's <Say> uses Twilio's built-in TTS and doesn't accept external audio input. To inject Deepgram audio into a call, you need a Media Streams WebSocket connection (typically via <Connect><Stream>).

Twilio will play any non-audio bytes as noise, including headers. The tricky edge case is streaming: some providers send a WAV header once at the start of the stream, so only your first frames pop or click.

Practical detection: if the first decoded bytes start with RIFF/WAVE, strip the container before framing 160-byte chunks. Don't assume the header is always 44 bytes—extra chunks (like LIST) can make it larger.

Not by itself. Even with correct codec and sample rate, you still need telephony pacing: Twilio expects audio to arrive as steady 20ms frames (160 bytes) with monotonic sequence and timestamps.

If your TTS stream delivers bursty chunks, buffer and re-frame them into 160-byte payloads, and avoid sending a flood of frames all at once. Also make sure you're not double-encoding (base64 of base64), which is an easy mistake when SDKs return strings.

Use a tunneling tool to expose your local WebSocket server to Twilio. Point your TwiML <Connect><Stream> URL at that tunnel and call a test number. For debugging, log (1) decoded payload length per message and (2) how many outbound frames you send per second.

Mark events are playback checkpoints: you send a mark with a name, and Twilio echoes it back after the buffered audio before it has played. Clear events flush queued outbound audio immediately. Marks tell you "this audio got through"; clears tell Twilio "stop playing what you have queued."