Three acknowledgements you cannot wait for

The browser sends UpdateListen to change the speech-to-text model and expects ListenUpdated back. The header draws the fallback when it never arrives: a three-second timer expires and a fresh Settings message names the new model on a new socket.

When you swap a model mid-call on the Deepgram Voice Agent API (the WebSocket service that runs the agent), each change comes back with an acknowledgement: UpdateListen gets ListenUpdated, UpdateThink gets ThinkUpdated. I treat every one of them as a fast path, with a three-second timer behind it and a quiet reconnect as the fallback, because the agent applied each switch I sent whether or not the ack reached my client. I built the pattern into Voice Comic, my booth demo that draws a conversation as comic panels and swaps the listen model (Flux STT, our conversational model, or Nova-3, our general one), the LLM and the voice mid-call from three dropdowns.

The short version

• Three acks went missing on my first run against live Deepgram: ListenUpdated, ThinkUpdated, and any answer to a Flux STT KeepAlive. The agent kept working.
• Two snippets follow: the timer in web/src/lib/session.ts and the mic-pause handler in my proxy, the Node server between browser and Deepgram holding the API key.
• When the three-second timer fires, the client reconnects with a fresh Settings (the opening message) and the old session id as resume, my proxy's field.
• Clone the repo, open the X-ray with x (the demo's log of each WebSocket message), then flip Listen to Nova-3 mid-conversation and watch the fallback fire.

The Deepgram Voice Agent API answers UpdateListen with ListenUpdated and UpdateThink with ThinkUpdated. A Flux STT socket on /v2/listen sends nothing back for KeepAlive. On my first run against live Deepgram, and again on a later run inside the Docker container my tests use, ListenUpdated did not always come back, and on the first run ThinkUpdated vanished with it, so my client waits three seconds and then reconnects with a fresh Settings.

On both runs the same UpdateListen (Flux STT to Nova-3, mid-conversation) got its ListenUpdated on one send and nothing on the next, inside the same session, while the agent kept transcribing on the new model either way. That switch crosses provider versions: in the Settings my proxy sends, Flux STT (flux-general-en) is listen.provider.version: "v2", and Nova-3 is v1, and an UpdateListen that stays inside one version got its ListenUpdated every time on my first run. The Update Listen docs say the server applies the change and returns ListenUpdated, and they say nothing about crossing versions, so I treat that ack as a fast path and the client never blocks on it.

My timer lives in web/src/lib/session.ts, and it calls reconnect() when ListenUpdated is missing after three seconds and no newer update has replaced this one.

const LISTEN_ACK_TIMEOUT_MS = 3000;
// Expect ListenUpdated. Otherwise fall back to a fresh Settings handshake.
const seq = ++this.updateSeq;
if (this.listenAckTimer) clearTimeout(this.listenAckTimer);
this.listenAckTimer = window.setTimeout(() => {
  this.listenAckTimer = null;
  if (seq === this.updateSeq) this.reconnect();
}, LISTEN_ACK_TIMEOUT_MS);

Build the new Settings without a greeting and carry your own state across. The proxy sends no greeting, so the agent does not say hello again, and it passes the previous session id as resume, so the usage meter (the demo's on-screen cost counter) picks up where it was. A sequence number stops a stale timer firing after a newer update.

On my first run I sent UpdateThink and UpdateListen together to the Deepgram Voice Agent API, and the agent applied both. The acks are the problem. On its own, UpdateThink (the message that swaps the LLM mid-call) got its ThinkUpdated every time I tried it. When the client changed the listen model and the LLM at the same time, and the proxy sent both updates back to back, the ThinkUpdated sometimes vanished along with the ListenUpdated.

The fix is sequencing. The proxy sends UpdateThink on its own, and the usage meter bills the LLM tier from the moment it is sent (that is the demo's counter, not your Deepgram invoice). I will take overcounting a second at the higher tier over waiting on an ack that may not come. A combined switch runs through the reconnect path above, where the new Settings names both models at once and there is nothing to acknowledge.

I expected KeepAlive to hold a paused Flux STT stream open, and it did not. A Flux STT stream on /v2/listen ignored KeepAlive and closed roughly ten seconds after the last audio frame on my first run, and on a later run the live test (node scripts/live.mjs, my scripted run against real Deepgram services) showed the second Flux STT stream the demo runs beside the agent, which it calls a layer, closing on pause and reopening on resume. It had nothing to honor: the version of the Flux /v2/listen reference I read lists four client messages (Media, CloseStream, ForceEndTurn, Configure), and KeepAlive is not one of them. That made a Flux STT KeepAlive the third thing I stopped waiting on.

The demo runs that second stream beside the agent to draw the words of the visitor (the person talking to the demo) as they are spoken. When the visitor pauses the microphone, the proxy sends KeepAlive to the Deepgram Voice Agent API socket every four seconds (the Agent Keep Alive docs, when I checked, say to send one every eight seconds while idle, so four leaves a comfortable margin), and the agent socket stayed open with no audio for as long as I left it. A Nova-3 stream on /v1/listen did the same with its own KeepAlive, as the Audio Keep Alive docs describe. So the proxy treats the two differently. On pause it does three things:

• It keeps the agent socket alive with a KeepAlive every four seconds.
• It keeps any Nova-3 stream alive with that stream's own KeepAlive.
• It closes a Flux STT stream and reopens it on resume.

The handler runs on the vc.pause message the browser sends when the mic button is paused.

const KEEPALIVE = JSON.stringify({ type: 'KeepAlive' });
const PAUSE_KEEPALIVE_MS = 4000;
// on vc.pause: the agent gets a pump, a Nova layer has its own KeepAlive, a Flux layer is closed until resume
if (paused) { if (agent) pump = setInterval(() => toAgent(KEEPALIVE), PAUSE_KEEPALIVE_MS); }

Reopening bit me once: Flux STT TurnInfo events carry a turn_index that restarts at zero on every socket, on both of my runs. If your UI keys turns by that index, a reopened stream will overwrite your first panels.

The client builds the panel key from the layer socket's epoch (a counter that goes up every time the layer reopens) plus Flux STT's turn_index, so a reopened socket produces new keys and the first panels stay where they are.

const id = `u-${this.layerEpoch}-${e.turn_index ?? 0}`; // turn_index restarts with every layer socket

All three fixes share one reconnect on the Deepgram Voice Agent API socket. Treat the acknowledgement as a fast path, put a timer behind it, and make the fallback something the user cannot see. In this demo the fallback takes one of three shapes:

• A quiet reconnect carries state forward through the resume id and sends no greeting.
• A sequenced send puts UpdateThink out on its own and waits on nothing.
• A Flux STT socket you expected to stay open closes on pause and gets reopened on resume.

If you carry conversation history across the reconnect, the Maintaining Context docs cover agent.context.messages. The demo carries its own state through the proxy's resume id, which picks up the previous session's usage meter, and it does not replay history. There is no agent.context.messages anywhere in the proxy.

Every UpdateSpeak that swapped the agent's voice from Flux TTS to Aura-2 (our two text-to-speech families) came back with SpeakUpdated on both of my runs. It was the one acknowledgement in this demo that never went missing, so the voice dropdown has no timer behind it. I do not know why the speak side acks and the listen side sometimes does not, and I am open to a different read if you have one. The Update Speak docs have the payload. The agent kept working in every one of these cases; the only thing stuck was my UI.

The pricing page, when I checked, said a free Deepgram account comes with $200 in credit and nobody asks for a credit card, so sign up at console.deepgram.com/signup and keep the key on the server. These four commands clone the demo, add that key on the server side, install, and start the dev server.

git clone https://github.com/dg-coreylweathers/voice-comic && cd voice-comic
cp sample.env .env      # add DEEPGRAM_API_KEY, server side only
npm install
npm run dev

Talk to Flux STT, press x for the X-ray, then flip the Listen dropdown from Flux STT to Nova-3 mid-conversation and watch for ListenUpdated. If it does not appear, three seconds later you will see a fresh Settings go out with no greeting. The hosted demo at dg-voice-comic.fly.dev runs the same build with no account, if you want to try the switch before you clone. node scripts/live.mjs runs the model switch and the pause against real Deepgram services and asserts the socket stays healthy.

I left the X-ray panel open on purpose for this shot from a later run, because the missing ack only shows in the log: after I flipped Listen to Nova-3 and the voice to Aura-2, UpdateThink got ThinkUpdated, UpdateSpeak got SpeakUpdated, UpdateListen at 27.05 got nothing, the client reconnected at 30.05, and the proxy sent a fresh Settings at 30.49.

For the happy path of UpdateListen, payload shapes and the ListenUpdated commit point, read the Update Listen docs, and the Configure the Voice Agent page for the full listen.provider shape. If you are building the UI on top of these messages, voice agent events maps each one to the screen change it drives. If you want to see the whole thing running before you read any of this, start with the deepgram flux demo.

Every fix in this post comes down to one habit: I put a timer behind each ack, and the screen never sits on "switching" again. Fork it, flip the Listen dropdown to Nova-3 mid-conversation, and watch the X-ray for the ListenUpdated that may not come.