Voice agents that keep the thread

Deepgram's CPO on the Real-Time Context Standard, the Think-Live and Think-Fast LLMs, and why we're building the Voice Agent Platform on Pipecat.

I've spent most of my career on new interaction layers, and on the plumbing underneath them that lets software talk to software. The web needed HTTP and HTML before it could change how people found information. Mobile needed APIs before apps could reach people wherever they were. Each time, the interface everyone noticed was only as good as the connections nobody saw.

Voice is next, and it's hitting the same wall. I've been saying for a while that voice is the new keyboard, because talking is how people actually explain what they need. We add context as we go, we back up, we correct ourselves. Most voice agents still can't keep up with that. They talk over you, go quiet while they look something up, and if you change your mind halfway through, they charge ahead with the first version of what you said. Using one is a lot like filling out a form out loud.

The models themselves have gotten remarkably good over the last few years. What holds voice back now is mostly plumbing, which happens to be the part of the stack I find most interesting. This morning at Deepgram Speak, I walked through what we're doing about it, and I want to go a little deeper here.

We announced the Real-Time Context Standard (RCS) for voice, an open standard for carrying meaning between the parts of a conversational agent, which we developed with Pipecat. Pipecat is now part of Deepgram, and I'll come back to why that matters. We also introduced two LLMs built for voice: the Think-Live LLM, for the conversation itself, and the Think-Fast LLM, for the work behind it.

Losing everything but the words

Most voice agents I see today are three models in a row. Speech-to-text feeds an LLM, the LLM feeds text-to-speech, and at each handoff almost everything except the words gets thrown away. The audio carries a caller whispering, a sigh, a word that barely came through, and the LLM receives a plain transcript. The LLM may decide the reply should be gentle and brief, and the text-to-speech model receives a plain string with no idea how to say it.

We had a version of this problem with APIs, too. Two systems could connect perfectly well and still misread each other, and what fixed it was agreeing on a contract. RCS is a contract for everything beyond the words, which is often a big part of what's being communicated. It adds three fields to the chat completions format most LLM APIs already use. One carries how the person sounded, one is where you tell the agent what to do about it, and one tells the text-to-speech model how the reply should be spoken.

If you’ve built with tool calling, the shape will look familiar, and that’s deliberate. Tool calling gave developers a standard way for a model to ask an application to do something, like look up an order, and MCP turned that into a shared standard any app or model can use. I’ve written before that MCP is starting to look like the new API. RCS applies the same idea to how people sound, and to how agents should sound back.

The standard itself doesn't depend on any vendor, so any framework can implement RCS with any model. It's open for comment, too. Context only helps if every component in the pipeline can read and write it, whoever made that component. If you build speech models, LLMs or frameworks, we’d like you implementing RCS and telling us how to improve it.

Giving each LLM one job

It took us a while to say this plainly, but a voice agent's LLM has two jobs, and they pull in different directions. The first is managing the conversation. In an instant, it has to decide whether the caller has finished or is pausing to think, and whether "yeah" means yes or "go on." The second is the work behind the conversation: following the business's rules, calling the right tools and writing a reply that sounds right out loud.

In my experience, when you ask one general-purpose model to do both, it ends up doing neither particularly well. So we split them. Our new Think-Live LLM is the model for the moment. It decides when to listen, wait or respond, so the agent stops talking over people and stops leaving dead air, and it's generally available in the Deepgram Voice Agent API. The Think-Fast LLM is the model for the task. It follows your rules, calls tools in the middle of a conversation and writes replies built to be spoken, and it's available in early access on the Deepgram LLM API. You can pair either one with the model you already trust, or use both and let each do the job it was built for.

Watching it work on a real call

Here's an example. A customer calls to move a delivery, and while the agent is checking available slots, they add, "actually, Thursday's better than Friday." In a typical pipeline that correction either gets talked over or arrives after the agent has already offered Friday. With the Think-Live LLM managing the conversation, the agent acknowledges the change right away and keeps the customer engaged while the Think-Fast LLM updates the request and finishes the lookup. Because RCS can carry the hesitation in the customer's voice, the reply slows down and confirms Thursday clearly.

The conversation and the work no longer have to wait on each other, so the agent can listen, adjust and act at the same time, the way a good human agent does.

Building on Pipecat, in the open

Developers building voice agents have mostly had two options. They could assemble an open stack themselves and own every integration, or start on a closed platform and rewrite once they outgrew it. We hear the same thing from some of Deepgram's longest-standing customers: they want to keep their options open. We don't think they should have to give that up to get to production.

That's why we're building the Deepgram Voice Agent Platform on Pipecat, the open framework so many voice agents already run on. I've been inside an acquisition twice, at Apigee and at DataStax, so I know the first question an open-source community asks when its project changes hands. Pipecat stays open source and keeps supporting the open ecosystem, so you can use any model from any provider, including ones that aren't ours. We're running toward open source, not away from it.

Pipecat is where you build an agent, and the platform is where you'll run it in production, whether that's hosted by Deepgram or in your own cloud. The platform brings Pipecat, RCS and our models together, with one team accountable for how they work together, so you can start on Pipecat today and move to production without rewriting your agent. The platform itself is still ahead. We'll share timing soon, along with the evaluation, observability and agent design tools we're building to work across every model provider.

Taking voice agents beyond English

Most of the people a voice agent will ever talk to don't speak English as their first language. Teams expanding into new markets have tended to end up with speech recognition from one vendor and a voice from another, and customers can hear the seams. We've been closing that gap from both sides of the conversation.

On the listening side, Flux STT Multilingual now has 10 languages available with 31 more in early access, and teams already using it get the new languages without a migration. It also uses earlier turns to make sense of what comes next, which is its own way of keeping the thread. For transcription beyond voice agents, Nova-3 Multilingual now covers 55 languages, also up from 10.

On the speaking side, Flux TTS is now generally available in Spanish, Portuguese, French, German, Italian, Dutch, Russian, Hindi and Japanese, with 32 more languages in early access, including Mandarin. Wherever the two sides overlap, a team can listen and reply in its customers' languages with one provider.

Where requests run matters just as much, and it's one of the first questions enterprise teams ask me. Today we're launching a new U.S. endpoint as part of a broader plan to give developers a choice between global reach and regional control. Coming soon, our Global API will provide more reliable, lower-latency connections from anywhere through the endpoint developers already call. Regional endpoints in the U.S., EU, India and Australia keep Deepgram’s processing and storage in-region, so you can choose reach or control workload by workload.

Where voice agents go next

Every interaction layer I've worked on has followed a similar arc. The first wave proves the interface can work at all, and the next wave makes it work the way people expect. We strongly believe voice will become the primary way people interact with AI within the next two years. When that happens, nobody will be impressed that an agent got the words right. The bar will be whether it noticed the pause, caught the correction and finished the job without making anyone repeat themselves.

Every technology wave I've been part of has come down to software talking to software. This time, the software also has to listen. We're building that in the open, with the developers who are already building on it, and the voice agents that win will be the ones that keep the thread.

Get started

• Interested in learning more about the Deepgram Voice Agent Platform or evaluating the Think-Fast LLM in early access on the Deepgram LLM API? Contact us or get in touch with your account owner directly.
• The Think-Live LLM can be accessed in the Deepgram Voice Agent API.
• Learn how to build with Real-Time Context Standard in Pipecat and try the demo here.