How low does voice-agent latency need to be?

Aim for the whole round trip — speech-to-text, the model, text-to-speech — to land under about 800 milliseconds, and treat sub-second as the line where a conversation stops feeling like a walkie-talkie. It’s a budget, not a benchmark: what the caller actually feels is when your voice starts, so a system that begins speaking the first clause quickly can feel snappier than one with a lower raw number. Measure it mouth-to-ear on a real phone line, because that’s the number that matters.

Vapi vs Retell vs building it yourself?

Vapi and Retell AI are both all-in-one platforms that hide the telephony, STT, turn-taking, and TTS plumbing behind one API and bill per minute — Vapi leans developer-first with more control over the pieces, Retell is tuned hard for low-latency phone handling. Building it yourself buys control, on-prem and data-residency, and lower marginal cost at scale, in exchange for owning latency and operations. Prototype your hardest call — a real accent, a real interruption — on a platform first; if it clears that bar, you may never need to compose.

How much do AI voice agents cost per minute?

There’s no single rate, and we don’t publish one — but the shape is what matters: platforms bill per minute of conversation, and that per-minute price usually sits on top of the underlying STT, LLM, and TTS costs rather than replacing them. It’s friendly at low volume and can become your biggest line item at scale. Model your expected minutes against a specific vendor’s current pricing before you commit — a rate that’s trivial in a demo can dominate at ten thousand calls a day.

Can I self-host a voice agent?

Yes. You can compose the whole stack from open pieces — a self-hosted STT like FunASR, an open streaming TTS like LLMVoX, a VAD like ten-vad, and SIP/WebRTC transport via rustpbx or active-call — or start from an open “platform you self-host” like Patter or bolna and keep the same shape without renting per minute. On-prem is the usual reason: data residency, cost at scale, no lock-in — the cost is that latency and operations become your job. For a fully offline box, axiom-voice-agent runs under 400ms on 4GB of VRAM.

How do I handle interruptions and barge-in?

Barge-in is non-negotiable for anything that feels human: the moment the caller speaks, your text-to-speech has to stop, mid-word. That needs a real voice-activity detector (ten-vad is a lightweight in-index option) feeding an endpointing decision — has the caller finished, or just paused? — plus acoustic echo cancellation so your own audio doesn’t trip your VAD. Tune the endpointing on recordings of real conversations: too eager and you interrupt the caller, too patient and every turn has a dead beat.

Guide · Voice

How to build a voice AI agent: choosing your stack when latency is the tyrant

A spoken agent lives or dies on one number — the silence between the caller’s last word and the agent’s first. Text agents can think for two seconds and nobody minds; a voice agent that pauses that long sounds broken, or rude. This guide walks the stack the way a senior engineer would — one fork at the top, then the layers — with latency as the constraint that quietly decides everything else. No rankings for hire, no vendor’s roadmap — just the trade-offs.

Independent · sponsor-blind · updated 2026-07-21 · how we choose

Voice is text with a stopwatch bolted on. Every turn runs a relay race — speech-to-text transcribes what the caller said, a language model decides what to say back, text-to-speech renders the reply out loud — and the whole lap has to finish inside roughly the time a person will wait before a silence starts to feel wrong. Call that budget somewhere under 800 milliseconds, end to end, then notice you’re spending it three times over on three different systems. Now add the part text never has to handle: people interrupt, talk over each other, trail off, say “um,” and expect the agent to stop talking the instant they start. A chatbot that’s half a second slow is fine. A phone agent that’s half a second slow gets talked over — and then both sides are speaking at once.

So don’t start by picking a speech-to-text engine. The decision that shapes your next month is more basic: do you buy a platform that has already solved the latency plumbing and pay per minute for it, or do you assemble the pipeline yourself and own every millisecond? Everything else follows from that answer.

The core decision: a platform, or compose it yourself

There are two honest ways to ship a voice agent, and they bill differently.

An all-in-one platform hides the hard parts — telephony, streaming STT, turn-taking, and TTS — behind one API that’s already tuned to answer fast, and it charges you by the minute of conversation. The commercial platforms in our index differ mostly in shape: Vapi is the developer-first choice, giving you the most control over the underlying models, voices, and telephony; Retell AI is tuned hard for low-latency phone handling and strong call-control primitives; Synthflow is the no-code path to inbound and outbound calls; Bland AI targets enterprises running high-volume calling; and ElevenLabs Agents leans on best-in-class speech quality and voice cloning. What you give up is control at the seams — when a turn feels a beat too slow, you’re tuning inside someone else’s abstractions.

Composing it yourself means you own each layer: transport (SIP or WebRTC), a streaming STT, the model, a streaming TTS, a VAD and turn-taking logic, and the latency instrumentation to keep the whole thing honest. It’s more work up front and far more control when — not if — you need to shave a hundred milliseconds, swap a voice, keep audio on-prem, or hit a cost target the per-minute meter won’t.

Between those poles sits an open-source middle ground — a platform you self-host. Patter is the “own the stack” alternative to Vapi and Retell, giving an agent a phone number in a few lines of Python or TypeScript across the usual carriers; bolna is an end-to-end platform for voice-first agents; echokit-server is another open voice-agent platform; and CyberVerse adds WebRTC and an optional digital-human face for real-time avatars. If you’d rather start from a reference build on fully open models, BentoVoiceAgent does phone agents on open weights, and bailing is a compact ASR + LLM + TTS pipeline the project clocks at around 800ms.

Honestly: if your goal is to get a working phone number in front of real callers this week, start on a platform — you’ll learn what “good latency” feels like for your traffic in days, and you can graduate later. Choose the compose path (or a self-hosted open platform) when voice quality and cost are your product rather than a feature, when data residency or on-prem is non-negotiable, or when per-minute pricing stops making sense at your volume. And mind the economics: the per-minute meter usually runs on top of the underlying STT, LLM, and TTS costs, not instead of them — trivial in a demo, potentially your largest line item at scale. A good tell: prototype your hardest call — a real accent, a real interruption, a bad connection — on a platform first. If it clears that bar, you may never need to compose.

The layers, if you compose

Six layers, in the order audio moves through them. For each: the decision, a recommended default, and the one mistake that quietly costs you.

1 · Telephony / transport

First question: how does audio get in and out — a phone call over SIP, or a browser or app over WebRTC? Most teams rent this layer rather than build it; the field’s standard picks are Twilio, Telnyx, and LiveKit (LiveKit especially for real-time WebRTC), none of which are in our index. If you’re carrying your own signaling, rustpbx is a PBX written in Rust and active-call is a SIP/WebRTC voice-agent transport — the in-index pieces for owning this outright. Default to renting from a carrier if you need a number tomorrow: telephony is a deep, boring, regulated problem you don’t want to own on day one.

Watch out — every hop the audio takes (carrier → your server → STT) adds latency and one more place for jitter and packet loss to poison the transcript; keep the media path short and close to your compute.

2 · Speech-to-text (STT)

For a live agent, streaming is the whole game: you need partial transcripts as the caller speaks, not a batch transcription after they stop, because you can’t start thinking until you have words. In our index, FunASR is a strong self-hosted, multilingual, streaming recognizer, and the official Deepgram SDKs (Python, JavaScript) are how you wire Deepgram’s hosted service in. The engines themselves are mostly external — the field’s standard picks are OpenAI Whisper (excellent accuracy, but base Whisper isn’t built for low-latency streaming), Deepgram’s hosted API (streaming-first), and AssemblyAI. Default to a streaming-first hosted API to start, and move to self-hosted FunASR when you need on-prem, more languages, or a lower per-minute bill.

Watch out — word error rate on your accents, your jargon, and your background noise — not the demo’s clean audio — decides whether every downstream layer gets a fair shot; and endpointing (deciding the caller has actually finished) lives here too, where half the latency battle is won or lost.

3 · The LLM / brain

The instinct is to reach for the smartest model. For voice, resist it: latency and token streaming matter more than raw reasoning quality. The model providers aren’t in our index — they’re the brain you rent, and the standard picks are OpenAI, Anthropic, and Google. What actually matters is time-to-first-token (the voice can start speaking the first clause before the sentence is even finished, so the model has to start streaming fast) and a ruthlessly tight prompt (every token of system prompt is latency and cost paid on every single turn). Default to a fast mid-tier model with streaming on and a short prompt; upgrade only when the conversation genuinely needs deeper reasoning.

Watch out — a long system prompt and a fat context window feel free in a text chat and are anything but on a phone call — they push time-to-first-token past the point where the pause becomes audible.

4 · Text-to-speech (TTS)

This is where the naturalness-versus-latency trade-off is sharpest: the most human-sounding voices often cost the most time-to-first-audio, and the fastest can sound robotic. In our index, LLMVoX is a streaming, autoregressive TTS designed to hang off any LLM’s token stream. The engines are largely external — the field’s standard picks are Cartesia, the raw ElevenLabs API, and PlayHT (and note that ElevenLabs Agents, above, packages this whole trade-off for you if you take the platform road). Default to a TTS that starts emitting audio on the first clause, not the first full sentence — perceived latency is about when the voice starts, not when it finishes.

Watch out — pick a voice that streams and supports barge-in — one that stops mid-word when interrupted; a gorgeous voice you can’t cut off makes the agent feel like it’s steamrolling the caller.

5 · Turn-taking, VAD & barge-in

This is the hard part — the thing that separates a slick demo from a call that feels human. It’s three jobs: detect when the caller is speaking (VAD), decide whether they’ve actually finished a turn or are just pausing to think (endpointing), and stop your own speech the instant they interrupt (barge-in). In our index, ten-vad is a low-latency, lightweight voice-activity detector for the first job. Default to a real VAD rather than a fixed silence timer, tune endpointing on recordings of real conversations, and treat barge-in as non-negotiable.

Watch out — overlap and echo — when your TTS audio bleeds into the caller’s mic and your own VAD hears the agent as the human. Handle acoustic echo cancellation and the half-duplex edge cases, or the agent will interrupt itself.

6 · Latency & testing

Latency isn’t one number you fix; it’s a budget you defend across all six layers, and it dies by a thousand cuts — 80ms here, 120ms there, and suddenly the pause is audible. Measure end-to-end (mouth to ear), not per-component in isolation, and measure under load, not on a quiet laptop. Default to instrumenting every hop — STT partial, endpoint decision, time-to-first-token, TTS first-audio — so you can see which layer ate the budget, then test with real accents, real background noise, real interruptions, and a real phone line.

Watch out — testing only on your own clean voice in a quiet room; the failure modes that matter — a heavy accent, a crying baby, a caller who barges in mid-sentence, a bad cell connection — never show up until a stranger calls.

The mistakes that sink voice agents

Testing on the vendor demo. The demo runs on a clean mic, a scripted flow, and the vendor’s own tuned voice. Your callers have accents, background noise, and no script. Prove it on a real phone line with real strangers before you believe any latency or accuracy claim.
Ignoring barge-in. An agent that can’t be interrupted feels like a hostage situation. Real conversation is full of interruptions; if the caller can’t cut in and be heard instantly, the whole thing reads as robotic no matter how good the voice sounds.
Latency death by a thousand cuts. No single component is slow, but STT plus a fat prompt plus a slow first token plus a non-streaming voice plus one extra network hop add up past the point of no return. Budget milliseconds per layer and defend the total.
Per-minute costs that blow up at scale. Pricing that’s trivial in a demo becomes your dominant line item at ten thousand calls a day — and it usually stacks on top of the underlying STT, LLM, and TTS spend. Model your real minute volume before you commit.
No fallback to a human. Voice agents fail live, in front of a frustrated caller, with no page to reload. A clean path to a human — or a callback — for the cases the agent can’t handle is a requirement, not a nice-to-have.
Accent and noise robustness as an afterthought. The pipeline is only as good as the transcript, and the transcript is where accents, jargon, and noise do their damage. If STT mangles the input, a smarter model just writes a more confident wrong answer.

Which should you pick?

Fastest to a working phone number, limited engineering time → an all-in-one platform: Vapi if you’re a developer who wants control, Retell AI for phone-tuned low latency, Synthflow if you want no-code, Bland AI for enterprise outbound volume.
Best-in-class voice quality out of the box → ElevenLabs Agents.
Own the stack — on-prem, data residency, or cost at scale → compose the layers yourself, or start from a self-hosted open platform like Patter or bolna; reach for CyberVerse if you also want WebRTC and a digital-human face, or echokit-server / BentoVoiceAgent to build on fully open models.
Edge, offline, or on-device → axiom-voice-agent runs under 400ms fully offline on 4GB of VRAM; xiaozhi-esp32-server and Starmoon are for hardware, IoT, and DIY-robotics devices; RCLI is on-device voice plus RAG on a Mac, no cloud.

Frequently asked

How low does voice-agent latency need to be?: Aim for the whole round trip — speech-to-text, the model, text-to-speech — to land under about 800 milliseconds, and treat sub-second as the line where a conversation stops feeling like a walkie-talkie. It’s a budget, not a benchmark: what the caller actually feels is when your voice starts, so a system that begins speaking the first clause quickly can feel snappier than one with a lower raw number. Measure it mouth-to-ear on a real phone line, because that’s the number that matters.
Vapi vs Retell vs building it yourself?: Vapi and Retell AI are both all-in-one platforms that hide the telephony, STT, turn-taking, and TTS plumbing behind one API and bill per minute — Vapi leans developer-first with more control over the pieces, Retell is tuned hard for low-latency phone handling. Building it yourself buys control, on-prem and data-residency, and lower marginal cost at scale, in exchange for owning latency and operations. Prototype your hardest call — a real accent, a real interruption — on a platform first; if it clears that bar, you may never need to compose.
How much do AI voice agents cost per minute?: There’s no single rate, and we don’t publish one — but the shape is what matters: platforms bill per minute of conversation, and that per-minute price usually sits on top of the underlying STT, LLM, and TTS costs rather than replacing them. It’s friendly at low volume and can become your biggest line item at scale. Model your expected minutes against a specific vendor’s current pricing before you commit — a rate that’s trivial in a demo can dominate at ten thousand calls a day.
Can I self-host a voice agent?: Yes. You can compose the whole stack from open pieces — a self-hosted STT like FunASR, an open streaming TTS like LLMVoX, a VAD like ten-vad, and SIP/WebRTC transport via rustpbx or active-call — or start from an open “platform you self-host” like Patter or bolna and keep the same shape without renting per minute. On-prem is the usual reason: data residency, cost at scale, no lock-in — the cost is that latency and operations become your job. For a fully offline box, axiom-voice-agent runs under 400ms on 4GB of VRAM.
How do I handle interruptions and barge-in?: Barge-in is non-negotiable for anything that feels human: the moment the caller speaks, your text-to-speech has to stop, mid-word. That needs a real voice-activity detector (ten-vad is a lightweight in-index option) feeding an endpointing decision — has the caller finished, or just paused? — plus acoustic echo cancellation so your own audio doesn’t trip your VAD. Tune the endpointing on recordings of real conversations: too eager and you interrupt the caller, too patient and every turn has a dead beat.

More stack guides

RAG

How to build a RAG knowledge assistant

Platform or compose-it-yourself, then the seven layers — ingestion, chunking, vector store, retrieval, orchestration, memory, and eval.

Read the guide →

Coding

How to build a coding agent

Adopt an agent or build your own harness, then the layers — the agent loop, code context, safe execution, the model, memory, and review.

Read the guide →

Browser

How to build a browser-automation agent

Perception is the hard part, not clicking. DOM vs vision vs VLA, then the runtime, the driver, the model, planning, and reliability.

Read the guide →

Autonomous

How to build an autonomous agent

Autonomy is a control problem, not a capability problem. Adopt, build, or hand-roll — then framework, tools, memory, execution, orchestration, and governance.

Read the guide →

MCP

How to build an MCP server

The protocol is trivial; tool design is the craft. Wrap, build, or aggregate — then SDK and transport, tool design, resources, reliability, and distribution.

Read the guide →