How Moveo Benchmarks Multilingual Voice AI with Deepgram for Real Contact Center Calls

Enterprises are under pressure to move past rigid, touch‑tone, menu‑driven IVRs and toward natural, low-latency voice AI that can actually resolve customer issues on the first call. That’s hard enough in a single language. When you add English, Brazilian Portuguese (PT‑BR), Greek, and additional markets like Italian and Polish, the challenge multiplies.

Moveo builds contact center voice AI agents and voicebots that answer real calls for banks, financial services, telcos, utilities, insurance, and gametech—often in highly regulated, high-stakes environments. To deliver on containment and customer experience, they need more than a good demo; they need an evaluation process that holds streaming speech-to-text (STT) providers to production-grade standards.

As part of that process, Moveo has evaluated multiple streaming STT providers—including Deepgram and other major vendors—and uses Deepgram today as a core part of the infrastructure powering their multilingual voice agents in production, especially where EU data residency, low latency, and telephony robustness matter most.

In this post, we at Deepgram share how Moveo evaluates and integrates streaming STT providers for multilingual voice agents, and what that reveals about building production-ready voice AI.

Traditional IVR systems weren’t designed for the way customers expect to interact today. People want to speak naturally, not “press 1 for billing.” Global brands also need consistent experiences across languages and regions, especially in markets like Brazil and Greece where high-quality voice AI has historically lagged. Meanwhile, CX leaders are on the hook for hard outcomes: containment, handle time, NPS, and adherence to compliance scripts.

For enterprises operating in Brazil, Greece, and global English-speaking markets, three issues show up repeatedly:

• Underserved languages. PT‑BR and Greek often have fewer off-the-shelf models and weaker performance under real call conditions.
• Telephony realities. PSTN, SIP, and mobile networks introduce codec artifacts, noise, and packet loss that most “clean audio” benchmarks never see.
• Regulated environments. In banking, telco, utilities, and insurance, there’s effectively zero tolerance for inconsistent transcripts or delayed end-of-turn detection that causes agents to talk over customers.

In that context, multilingual coverage with real call quality isn’t a nice-to-have. It’s table stakes.

When Moveo evaluates STT providers for voice agents, they optimize for what moves the needle in production, not a leaderboard score.

First, they look at transcription accuracy in real conditions: mixed accents and dialects across EN, PT‑BR, and Greek; noisy telephony audio; and domain language for financial services, telco, utilities, insurance, and gametech. If transcripts fall apart on those basics, nothing else matters.

Second, they care deeply about end-of-turn (end‑of‑utterance) detection. The engine has to reliably detect when a caller is finished, avoid cutting people off mid‑sentence, and not leave multi‑second gaps that feel awkward. That behavior also needs to be consistent across providers and languages.

Third, they measure latency over real telephony, not just localhost. Two dimensions matter here: how quickly partial and final transcripts arrive, and how long it takes from caller pause to agent response. All of this is measured over Twilio phone calls, SIP trunks, and WebRTC clients.

Finally, because Moveo serves regulated, high‑stakes verticals, production readiness is non‑negotiable: EU hosting and GDPR compliance, stability under bursty traffic, and enough observability to see exactly how models behave over time.

Underneath all of this are very simple buyer questions: will the agent actually resolve calls in PT‑BR and Greek without frustrating people, can transcripts be trusted in a dispute, and does the experience feel human over the phone?

Over time, Moveo has converged on a three-phase evaluation process for streaming STT. Each phase pulls the models closer to the messy reality of production traffic.

Before the phases, it’s worth describing how Moveo measures STT and agent behavior technically.

For each call, they log a small set of timing metrics:

• Transcription delay (TD): time between when audio is spoken and when the STT engine returns a usable transcript fragment.
  TD = t_transcript_first_token – t_audio_start
• End‑of‑utterance delay (EoU): time between when the caller stops speaking and when the STT engine signals end‑of‑utterance.
  EoU = t_eou_marker – t_audio_end
• Agent response latency (ARL): end‑to‑end time between caller pause and synthesized agent audio starting.
  ARL = t_tts_start – t_audio_end

They then aggregate these (p50, p95) by provider, language, transport (WebRTC vs Twilio SIP), and scenario so they can compare providers on the same footing and watch for regressions over time.

To make that possible, the media and agent pipeline emits structured events with synchronized timestamps from three layers: the client/media layer (when packets enter and leave the WebRTC or SIP bridge), the STT layer (when audio segments and transcripts move between Moveo and the provider), and the agent layer (when the LLM finishes its turn and when TTS starts streaming audio back). Events are stitched together via a call ID and utterance ID, converted into time-series, and exported to Prometheus.

At the observability layer, these timings show up in series like:

• voice_transcription_delay_seconds{provider="deepgram",language="pt-BR"}
• voice_eou_delay_seconds{provider="deepgram",transport="twilio_sip"}
• voice_agent_latency_seconds{provider="deepgram",scenario="inbound_support"}

That lets Moveo drill into performance by provider, language, and channel instead of relying on a single “average latency” number.

With that in place, here’s how they actually evaluate providers.

Phase 1 is about quickly ruling out obvious mismatches.

Moveo starts with providers’ own playgrounds (including Deepgram and other major STT vendors) to run pre‑recorded samples and TTS‑synthesized audio in English, PT‑BR, Greek, and any new languages they’re exploring, like Italian or Polish. They only use realtime/streaming APIs at this stage—no async.

Engineers and product folks listen to and read the outputs, checking domain‑specific phrases, punctuation and formatting, how the engine decides a caller is “done” speaking, and the general feel of latency in the playground UI. The purpose isn’t to publish benchmark charts; it’s to spot obvious gaps. Models that clearly struggle with PT‑BR or Greek, or whose end‑of‑turn behavior is very unstable, don’t move on.

Phase 2 moves from lab audio to actual agents.

Here, Moveo builds end‑to‑end agents on LiveKit that combine the candidate STT models, different LLMs for reasoning, and multiple TTS options. They then test those agents via two paths: web‑based calls in the LiveKit playground and real phone calls through Twilio SIP trunks.

For each scenario (a scripted dialog, language, and channel type), Moveo replays the same audio through multiple STT providers in parallel. They capture all timing and transcript events and write one compact JSON log per call, which feeds into their Prometheus ingestion pipeline.

From there, they look at:

• EoU delay: how long it takes from the moment the caller stops to the moment the agent starts responding.
• Transcription delay: how long it takes for partial and final transcripts to show up.
• Qualitative behavior, like whether the agent barges in on the caller, leaves long silences, or behaves very differently on Twilio calls vs web.

This is where models that looked similar in the playground often start to separate. Some are much more forgiving of telephony artifacts; some introduce just enough extra delay that the experience no longer feels natural.

In Phase 3, the strongest candidates from Phase 2 are wired directly into the Moveo platform.

Practically, that means adding WebSocket API support for the provider, running full end‑to‑end agents with Moveo’s own LLM “brains,” multiple TTS backends, and real WebRTC clients and Twilio phone calls. At that point, Moveo measures those same metrics—TD, EoU, ARL—plus error rates and disconnects, based on the events each model exposes.

Those measurements are exported as Prometheus metrics and sliced over production‑like traffic: different verticals, call types, and languages. Providers that look good in a sandbox but can’t sustain performance under real volume or mixed traffic patterns fall out here. The ones that stay in the recommended stack are the ones that survive all three phases.

Under the hood, a few infrastructure decisions have outsized impact on CX, compliance, and reliability.

One is EU hosting and data residency. Many of Moveo’s customers—especially European banks, financial services, and utilities—have strict requirements around GDPR and keeping voice data in‑region. Working with providers that offer EU‑hosted endpoints reduces legal friction, simplifies security reviews, and aligns with customers’ internal policies.

Another is denoising and telephony robustness. Real contact center audio contains background chatter, line noise and compression from PSTN and SIP, and callers on speakerphone or Bluetooth. Providers that ship built‑in denoising and models tuned for telephony audio tend to produce more reliable transcripts, better intent detection, and fewer “Can you repeat that?” moments.

Finally, there’s model selection and language coverage. Because Moveo serves EN, PT‑BR, Greek, and additional languages, they look at how well a provider’s flagship models handle each one, when it makes sense to mix and match models by language, and how easy it is to introduce new languages as customers expand.

Moveo treats those infra choices as enablers, not the headline. The real goal is simple: fast, natural, multilingual agents that enterprises are willing to put in front of real customers.

This is where Deepgram comes in.

Moveo has leaned on Deepgram as a key STT provider because Deepgram’s streaming‑first models, multilingual coverage (including PT‑BR and Greek), EU‑hosted endpoints, and denoiser support map closely to what their banking, telco, utilities, and insurance customers expect.

On the configuration side, that means using streaming models optimized for telephony audio rather than studio‑quality inputs, enabling denoiser, tuning endpointing so agents respond quickly without cutting callers off mid‑sentence, and choosing model configurations that handle PT‑BR and Greek accents, code‑switching, and domain terms reliably.

After running multiple providers through this process, a few patterns stand out.

First, latency gaps become very real over the phone. Models that look similar in a browser demo can feel noticeably slower once you add Twilio and real mobile networks into the loop.

Second, language‑specific behavior matters more than averages. A provider that shines in English but is inconsistent in PT‑BR or Greek won’t meet the bar for enterprises that operate across those markets.

Third, end‑of‑turn behavior is as important as raw accuracy. A model that’s a bit more accurate but frequently cuts callers off—or waits too long to respond—can create a worse overall experience than a slightly less accurate but more stable alternative.

And finally, observability is non‑negotiable. Being able to instrument, graph, and compare providers over time is what lets Moveo recommend a stack to a bank, telco, or utility with a straight face.

As Moveo continues to roll this approach out, they’re seeing meaningful improvements in end‑to‑end response times and containment on multilingual call flows, especially in PT‑BR and Greek. The exact numbers vary by customer and use case, but the direction is clear: treating STT evaluation as a product decision pays off.

For enterprises and builders exploring multilingual voice AI, the takeaway is simple: design an evaluation process that reflects your real call conditions, vertical constraints, and latency expectations—and hold your providers to that bar.

If you’re exploring multilingual voice agents and want to go deeper on Moveo’s evaluation process or real‑world results, reach out to the Moveo team. To learn how Deepgram can help you power similar multilingual, production‑ready voice agents, visit Deepgram’s Voice AI platform or contact us.