Choose a voice agent stack. Test the calls that matter.

Start with one job, not a vendor.

Write a testable brief before trying a demo. For example: “Move an existing booking, confirm the exact time, and transfer to a person if the caller asks.” Keep the same backend fixtures, tool permissions and failure cases across candidates.

Successful outcome

A specific backend state or a verified human handoff. A fluent conversation alone does not complete the job.

Operating conditions

Phone or browser, caller region, languages, noise, peak concurrent calls and supported opening hours.

Hard limits

Which actions need authorization, what the agent may change, where data may travel and what fallback is available.

Acceptance criteria

Agree task-success, response-delay, interruption and cost limits before the test. Mark any critical failure as a stop condition.

Two architectures. Four starting points.

A managed platform can reduce the infrastructure you operate. A framework gives you more responsibility for composing and deploying the system. Neither choice establishes better quality or lower total cost. These examples are a focused shortlist, not an exhaustive ranking.

For a no-code requirement, use the voice-agent directory to widen the shortlist. For a fully offline requirement, verify every speech, model, transport and telemetry dependency; “open source” or “self-hosted agent” alone is insufficient evidence.

Run the calls that expose failure.

Use consenting testers, authorized destinations and sandbox tools with fictional records. Repeat scenarios across the conditions in your brief. A small test batch can expose failures; it cannot establish a production failure rate.

Measure the outcome and the waiting.

Record the candidate version, prompt, models, voice, tools, channel and region for every run. Change one variable at a time. Keep task results separate from caller impressions so a pleasant but incorrect answer cannot count as success.

• Task success: verify the backend outcome. Report successful eligible calls / all eligible calls, and document exclusions. Include failed and interrupted calls.
• Response delay: define it as caller speech ending to the first audible agent response at the caller. Record whether a sample is an acknowledgment or the substantive answer; a filler phrase must not hide a slow tool.
• Tail behavior: keep per-turn observations and report sample size, median and a defined tail percentile. Compare the same conditions. Do not mix component timing, turn timing and vendor call-level aggregates.
• Interruptions: separately measure caller speech starting to agent playback stopping, and whether the interrupted request is handled correctly afterward.

Retell documents end-to-end and component latency reporting. Its reported end-to-end metric excludes some tool-call turns and the network trip from its server to the frontend. Use that telemetry for diagnosis, then independently measure the caller-audible wait on your actual route. Retell latency definitions.

Price the complete job.

Collect a dated quote or usage breakdown for the exact configuration. Vapi separates hosting from model-provider costs; Retell’s calculator itemizes voice infrastructure, models, telephony and add-ons. These are different billing presentations, not an apples-to-apples price comparison. Vapi pricing; Retell pricing.

List what each rate includes before adding it; do not count bundled components twice. Model failed calls, silence, retries, transfer legs, concurrency, support and retention requirements. Bring-your-own model keys move those charges to a provider bill rather than making inference free. Vapi's component estimates depend on configuration, token usage and caching assumptions; they are estimates, not invoices. How Vapi estimates component usage.

Download the blank operating-cost worksheet · CSV. Enter quantities and rates in the same billing unit, record each rate's source, date and currency, and calculate quantity × rate for each applicable row. Leave bundled rows out of the total. Reconcile the result with the actual bill and count successful tasks separately; no vendor rates or results are prefilled.

Choose the smallest system that clears your gates.

Reject a candidate that fails a hard requirement, even if it has a pleasing voice or a strong average score. Among the remaining candidates, weigh observed task success, tail behavior, total cost and the work your team can operate.

Our recommendation: start with a managed platform when your requirements fit and you lack capacity to operate a media system. Evaluate a framework when control over processing or deployment is a concrete requirement. The failure mode of that advice is hidden platform constraints or underestimated operations work; the shared test plan is how to expose both.

Before a wider rollout, record the chosen configuration, unresolved failures, an owner for incidents, a fallback route and a stop condition. Re-run the affected scenarios after changing a model, prompt, tool, voice or telephony route.

Questions worth settling early

Vapi or Retell: which should I choose?

Shortlist both if you want managed phone-agent infrastructure. Match their documented transfer modes and integrations to your requirements, then run the same call scenarios on each. This guide does not establish a latency, voice-quality or cost winner.

Does self-hosting mean the whole conversation stays on my servers?

No. Hosting your agent process yourself does not determine where media transport, speech recognition, language models, speech synthesis or logs run. Map each dependency and verify the selected deployment and data settings before calling a system fully self-hosted or offline.

What is a good voice-agent latency target?

Set a target for your actual call task and channel before testing. Define the timing boundary, collect per-turn samples and compare median and tail latency alongside task success and interruptions. A vendor average or an isolated component measurement is not a universal acceptance threshold.

Source notes

Official documentation and pricing pages checked 2026-10-08. Capabilities and terms can change. We have not placed test calls, measured these vendors against each other or verified production performance. No vendor paid for inclusion or position in this guide. Editorial methodology.

• Vapi: transfer modes and destination verification
• Vapi: component cost estimates
• Vapi: current pricing and exclusions
• Retell: call transfer configuration
• Retell: latency measurement
• Retell: current pricing and components
• LiveKit: Agents framework overview
• LiveKit: custom deployment responsibilities
• Pipecat: pipeline and frame processing
• Pipecat: framework and deployment overview