Voice AI Agent Development Cost: Latency Is the Whole Product
Published estimates for a voice AI agent run from $15,000 for a single-intent IVR replacement to $150,000 for a production platform, with running costs quoted anywhere between $0.005 and $0.30 per minute of call time. Those ranges are not noise. They are two different products, and the thing that separates them is measured in milliseconds.
A voice agent has roughly 800 milliseconds from the moment a caller stops speaking to the moment it starts replying. Past that, people interrupt, repeat themselves, or say "hello?" One vendor puts the natural-conversation threshold nearer 600ms; another notes that targeting sub-500ms requires a fundamentally different architecture than accepting 1.5 seconds. Everyone agrees the cliff exists and that it is not negotiable, because it is a property of human conversation rather than of software.
That budget has to cover everything. Detecting that the caller finished talking, transcribing what they said, deciding what to do, possibly calling your database, generating a reply, and synthesising it into audio. Each stage spends part of a budget shorter than a blink of hesitation.
This is why the cost range is so wide. A chatbot that takes two seconds to answer is fine. The same model, the same prompt and the same tools, put behind a phone line, is a product nobody will use. Everything expensive about a voice agent is the engineering that buys back milliseconds.
Three questions set what you pay. Does the agent need to take actions in your systems, or only answer questions? Does it handle interruption? And how many calls run at once at peak? Everything else — the voice, the script, the dashboard — is ordinary work.
The use case decides whether any of this works, and it is the question most often skipped in favour of picking a vendor.
High-volume, low-variance calls are where voice agents earn their keep. Appointment reminders and rescheduling, order status, balance enquiries, opening hours, delivery slots, prescription refills. One intent, a small number of paths, a clear finish.
Outbound is easier than inbound. On an outbound call you chose the moment, you know who you are calling and why, and you control the opening. On inbound the caller could want anything, is often already annoyed, and has an unstated goal. Teams that start inbound take the hardest version of the problem first.
Emotional calls are the wrong place to start regardless of how capable the model is. Complaints, cancellations, bereavement, anything medical or financial that has gone wrong. The agent may handle the words correctly and still produce a worse outcome than silence, and the reputational cost of getting one of these wrong outweighs the savings on a thousand routine calls.
Long-tail intents are a trap. Each additional intent adds test surface, failure modes and prompt complexity, and the twentieth intent serves a fraction of a percent of calls. Measure your call mix before scoping: in most operations a handful of reasons account for the large majority of volume, and the honest product automates those and transfers everything else.
If you cannot name the three calls you want automated, in order of volume, the project is not ready to start. That list is the specification, and producing it is usually a day of listening to recordings rather than a workshop.
There are two ways to build this and they are not variations on a theme.
The cascading pipeline chains three models: speech to text, then a language model, then text to speech. Each stage is a separate vendor you can swap, tune and price independently. You can see exactly what was transcribed, exactly what the model decided, and exactly what was said back, which makes debugging tractable and makes compliance recording straightforward.
The speech-to-speech model does all three at once. Audio goes in, audio comes out, with no text in the middle. It is dramatically faster, because it removes two handoffs and the queuing between them, and it sounds better, because it keeps tone, pace and emotion that a transcript throws away.
What you give up with speech-to-speech is control and visibility. There is no transcript at the decision point to inspect, log or test against. Steering behaviour means prompting rather than engineering, and the model is harder to constrain. For a regulated workflow where you must prove what was said and why, that is a serious problem.
The practical split is interaction quality against auditability. Speech-to-speech wins for open conversation. Cascading wins where the agent takes consequential actions, where you need per-stage cost control, or where someone will eventually ask you to produce evidence.
Decide this before anything else is built. The two share almost no code below the telephony layer, and switching mid-project means rebuilding the agent.
Writing the budget out is the single most useful exercise in scoping this product, and almost nobody does it before committing to a stack.
Turn detection is first and is underestimated. Knowing the caller has finished rather than merely paused costs real time. Wait too long and the agent feels slow; wait too little and it talks over people. Simple silence thresholds are cheap and bad; semantic turn detection is better and costs more latency to run.
Transcription can stream, and must. Waiting for a complete utterance before transcribing it spends the whole budget before thinking starts. Streaming transcription returns partial results as the caller speaks, so the language model can begin working on a nearly complete sentence.
The language model is where the budget is won or lost. Time to first token matters far more than total generation speed, because synthesis can start on the first clause. A reasoning step, a long system prompt, or a retrieval call before generation each add hundreds of milliseconds, and any one of them can break the budget on its own.
Synthesis streams too, generating audio for the first phrase while the model is still writing the rest.
Network and telephony take a slice before you have done anything. Carrier legs, codec handling and geographic distance between the caller and your inference region all spend budget. Running inference in a region far from your callers is the most common avoidable mistake in this category.
A voice agent that only answers from a knowledge base is comparatively easy. One that checks an order, books an appointment or takes a payment has to call your systems mid-conversation, and your systems were not built for this.
An API that takes 900 milliseconds has already blown the entire budget, and most internal APIs are slower than their owners believe. Before committing to an action-taking agent, measure the p95 latency of every call it will make. Not the average, the p95, because the slow calls are the ones callers will remember.
The honest fix is usually to speak while you work. The agent acknowledges — "let me pull that up" — and the lookup runs underneath. This is what humans do, it is not a hack, and designing it in from the start is what makes an action-taking agent feel competent rather than broken.
Caching and pre-fetching earn their keep here. If the caller's phone number identifies an account, load that account while the greeting plays. Most of the useful context can be fetched before the caller has finished saying hello.
Some actions should not happen in the call at all. Anything slow, anything irreversible, anything that needs a confirmation step can be queued and confirmed afterwards by message. Trying to complete every transaction inside the conversation is how teams end up with an agent that pauses for four seconds and loses the caller.
Every voice agent demo is a polite conversation where nobody talks over anybody. Real calls are not like that.
Barge-in means the caller can interrupt and the agent stops immediately. Implementing it requires the agent to be listening while it speaks, to distinguish a genuine interruption from the caller's own audio bleeding back, and to stop synthesis mid-word and discard the rest.
The state problem is the hard part. When the agent is cut off halfway through a sentence, what did the caller actually hear? The agent must track what was spoken rather than what was generated, because anything else means it believes it conveyed information the caller never heard. That single bug produces the agent that insists it already gave you the reference number.
Echo cancellation is doing more work than people realise. Without it the agent hears itself and interrupts itself, which on a speakerphone is immediate and obvious.
Background noise sets a floor on quality you cannot engineer past. Callers are in cars, on streets and in call centres. Test with real recordings in those conditions, early, because a product validated in a quiet room will disappoint on day one.
The build is one-off. The per-minute cost runs forever and scales with success.
Published per-minute figures span $0.005 to $0.30, with OpenAI's Realtime API quoted around $0.06 per minute and one worked example putting a complete call at roughly $0.55 to $0.60. The spread comes from architecture: a cascading pipeline stacks three separate per-minute costs, while a speech-to-speech model charges once.
Real-time inference costs more than batch for a structural reason. A voice agent serves a handful of concurrent requests and sits idle between calls, so you are paying for capacity that cannot be packed efficiently. Providers price that in, and it is why voice is more expensive per token than the same model answering a chat.
Do the arithmetic on call volume, not on tokens. At $0.06 per minute, 1,000 calls a month averaging four minutes costs about $240. At 50,000 calls it is $12,000 a month, which is a line item that needs an owner and a dashboard.
Telephony is separate and frequently forgotten. Numbers, inbound and outbound minutes, and carrier fees sit on top of the AI cost and vary enormously by country.
The number to watch is cost per resolved call, not cost per minute. An agent that is cheap per minute and takes six minutes to fail is worse than one that costs double and finishes in ninety seconds.
Teams spend weeks on the prompt and ten minutes on the voice, which is backwards for a product people only hear.
Synthesis quality varies enormously and so does its price. The most natural voices cost several times the cheapest and add latency, so the choice trades three things against each other rather than two. For a thirty-second status update the cheap voice is fine; for a five-minute advisory conversation it is not.
Accent and language are commercial decisions. A voice that sounds foreign to your callers reduces trust before the content matters, and in multilingual markets the question of which language the agent opens in is a business call with no neutral answer.
Pace, pauses and fillers matter more than timbre. An agent that replies instantly with no pause sounds wrong, oddly, because humans do not. A short, well-placed hesitation reads as thinking and buys real milliseconds at the same time, which is the only place in this product where a better experience is also cheaper.
Pronunciation of your own vocabulary needs fixing manually. Product names, place names and industry terms will be mispronounced by default, and every synthesis provider offers a lexicon to correct them. Building that list is an afternoon and it is the difference between sounding like your company and sounding like a generic bot.
Say that it is an AI, at the start. Disclosure requirements vary by market and are tightening in several, but beyond compliance it sets expectations correctly, and callers who know they are talking to a machine speak more clearly and forgive more.
The most important design decision in a voice agent is what happens when it cannot help, and it is usually specified in one line.
Transfer has to carry context. A caller who has spent two minutes explaining a problem and is then asked to explain it again by a human has had a worse experience than if they had reached the human immediately. The summary, the transcript and any identifiers must arrive with the call.
Decide the escalation triggers explicitly. Repeated misunderstanding, an explicit request for a person, detected frustration, or any topic you have decided an agent should not handle. All four need defining, and the last one is a business decision rather than a technical one.
Out-of-hours changes the answer. An agent that cannot transfer because nobody is there needs a different path: take a message, book a callback, or say plainly that it cannot help and when someone will.
Measure the escalation rate as a headline metric from day one. It is the clearest signal of whether the agent is working, and a rate that rises after a prompt change is the fastest way to catch a regression.
Text agents can be tested with text. Voice agents cannot, and the testing layer is a real and routinely unscoped part of the build.
Build a recorded test set early. Real callers, real accents, real noise, real interruptions. Synthetic audio generated by a text-to-speech model is clean in exactly the ways real calls are not, and an agent tuned against it will underperform.
Test the latency budget as a hard assertion, not an observation. Measure the gap between caller stop and agent start on every test case and fail the build when it exceeds the target. Latency regressions arrive silently with a longer prompt or an extra tool, and nobody notices until a caller complains.
Accents and names break systems in production, not in development. A transcription stack tuned on one accent will mishear another, and proper nouns are the hardest category in the language. If your callers span regions, your test set must too.
Record everything and keep it, within whatever privacy rules apply to you. Call recordings are how you find out what actually went wrong, and the consent and retention position needs settling before launch rather than after.
A latency spike comes first, before any product work. Two weeks assembling the thinnest possible pipeline on real telephony and measuring the round trip tells you whether your use case fits the budget at all. It is the cheapest risk reduction on the project and almost nobody does it.
The escalation path second, because it is the safety net under everything else and because an agent that cannot hand over cannot be launched to real callers.
One narrow intent third, done properly. An agent that handles appointment rescheduling perfectly is a product. An agent that handles twelve intents at 70% is a complaint generator.
What we would not build is telephony, transcription or synthesis. All three are mature bought components, and building any of them is a company rather than a feature.
Our rate band is published: $40 to $100 an hour by role, blending to $60 to $70 for a mixed team, with the audio pipeline sitting at the top of it. We do not quote a fixed total before a discovery that establishes whether the agent takes actions, what your systems' p95 latency is, and what peak concurrency looks like. What we will tell you free is that if your internal APIs take a second to answer, the agent is not your first project.
Published estimates run from about $15,000 for a single-intent IVR replacement in one language to $150,000 for a production platform. Our published rate band is $40 to $100 an hour by role, blending to $60 to $70 for a mixed team, with the real-time audio pipeline needing senior time at the top of it. The build is only part of the cost: running a voice agent costs between roughly $0.005 and $0.30 per minute of call time depending on architecture, which scales with usage forever.
Roughly 800 milliseconds from the caller finishing speaking to the agent beginning to reply, with vendors citing nearer 600 milliseconds for a conversation that feels natural and noting that sub-500 millisecond targets require a different architecture altogether. That budget covers turn detection, transcription, model inference, any tool calls, and speech synthesis combined. It is a property of human conversation rather than a technical preference, which is why it cannot be negotiated down.
Speech-to-speech is faster and sounds more natural because it removes two handoffs and preserves tone, but it gives you no transcript at the decision point, which makes debugging, testing and compliance recording much harder. A cascading pipeline of speech-to-text, language model and text-to-speech is slower but lets you swap and price each stage independently and inspect exactly what happened. Choose cascading where the agent takes consequential actions or where you will need evidence of what was said.
Because the entire response budget is around 800 milliseconds and most internal APIs are slower than their owners think. An endpoint taking 900 milliseconds has consumed the whole budget before the model has said anything. Measure the p95 latency of every call the agent will make before committing to the design, pre-fetch what you can during the greeting, have the agent speak while it works, and move slow or irreversible actions out of the call entirely.
Barge-in lets the caller interrupt and the agent stop immediately, which is how real conversations work. It requires the agent to listen while speaking, distinguish a real interruption from its own audio echoing back, and stop synthesis mid-word. The subtle part is state: the agent must track what was actually spoken rather than what was generated, or it will believe it conveyed information the caller never heard, which produces an agent that insists it already gave you a reference number.
Per-minute figures range from about $0.005 to $0.30 depending on architecture, with OpenAI's Realtime API quoted around $0.06 per minute and one worked example putting a full call near $0.55 to $0.60. At $0.06 per minute, 1,000 four-minute calls a month is roughly $240 and 50,000 calls is roughly $12,000. Telephony numbers and carrier minutes sit on top and vary by country. Track cost per resolved call rather than cost per minute.
With recorded real calls, not synthetic audio, because text-to-speech test audio is clean in exactly the ways real calls are not. Build a test set covering the accents, background noise and interruption patterns your actual callers produce, and include proper nouns, which are the hardest category for any transcription stack. Assert the latency budget as a hard test that fails the build when exceeded, since latency regressions arrive silently with a longer prompt or an extra tool call.
Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.
Share your details and we will talk soon.
Be the first to access expert strategies, actionable tips, and the trends actually shaping the digital world. No fluff - just practical insights delivered straight to your inbox.
Dive into our blog and stay ahead of the curve with expert perspectives, future-ready trends, and tech tips written for decision-makers and doers alike.
