Most businesses underestimate how much call-handling costs them — not just the direct salaries, but the invisible costs of abandoned calls during peak hours, inconsistent agent responses, and the delay between a customer calling and a resolution that actually sticks. The surprising reality is that a well-built AI voice agent for business doesn't need to replace every human call; it needs to handle the 60–70% of calls that follow predictable patterns, freeing skilled staff for the conversations that genuinely need them. Building that system is now technically within reach for mid-sized companies — but the architecture matters enormously.
What a Real-Time Voice AI Stack Actually Looks Like
A phone-based AI assistant has three core processing stages happening in sequence, and the round-trip latency of those stages is what determines whether the conversation feels natural or frustrating. Human speakers register pauses longer than roughly 700–800ms as awkward. That's your engineering target.
The three stages are:
- Speech-to-text (STT) — convert the caller's audio to text in near-real-time.
- LLM reasoning — process the transcription, maintain conversation context, decide on a response.
- Text-to-speech (TTS) — synthesize a natural-sounding audio reply and stream it back.
Each stage adds latency, and latency compounds. Choosing and tuning each component independently is how you get from an unworkable 2-second delay to a natural 400–600ms response time.
Speech-to-Text: Where Most Teams Make Their First Mistake
The temptation is to use a batch-mode transcription API because it's easy to integrate. The problem is that batch STT waits for the speaker to finish before it starts processing. That alone adds 500ms to 1.5 seconds before the LLM even sees any text.
Streaming STT changes this. Services like Deepgram Nova, AssemblyAI Streaming, or OpenAI's real-time audio API begin returning partial transcripts as the caller speaks. This lets you start feeding text to the LLM while the speaker is still talking — a technique called early invocation. Combined with end-of-utterance detection (recognizing when someone has actually finished speaking rather than just paused briefly), streaming STT can shave 400–700ms off the felt latency.
| STT Option | Streaming? | Accuracy (EN) | Multilingual? | Approx. Cost |
|---|---|---|---|---|
| Deepgram Nova-2 | Yes | Very high | 30+ languages | ~$0.0043/min |
| AssemblyAI Streaming | Yes | High | Limited | ~$0.006/min |
| Google Speech-to-Text | Yes | High | 125 languages | ~$0.006/min |
| Whisper (self-hosted) | No (batch) | Very high | 90+ languages | Compute cost only |
For call center volumes in regional languages — Hindi, Tamil, Arabic, Spanish — the multilingual accuracy gap between providers widens significantly. Test on a sample of your actual call recordings, not benchmark datasets, before committing.
The LLM Layer: Context, Speed, and Tool Calls
The LLM is where the AI voice agent decides what to say. For voice applications, this stage needs to be fast above almost everything else — streaming completion APIs that return the first tokens within 100–200ms are far more usable than higher-quality models with 600ms time-to-first-token.
Context management is the other critical design decision. A voice conversation accumulates turns quickly, and naively passing the entire transcript to the model on every turn inflates token costs and latency. A practical approach:
- Keep the last 6–10 turns in the active context window.
- Summarize earlier turns into a compact "conversation memory" block that's prepended to the system prompt.
- Store caller-specific data (account status, previous orders, open tickets) in a structured slot that's retrieved once at the start of the call and updated as slots are filled.
Tool calls — the ability for the LLM to query external systems mid-conversation — are what separate a useful voice agent from a scripted IVR tree. When a caller asks "what's the status of my order from last Tuesday," the model should be able to call an order lookup function, get a structured result, and fold it into a natural spoken response. Design your tool schemas carefully: the model's ability to pick the right tool with the right parameters depends entirely on how clearly those schemas are described.
Text-to-Speech: Voice Quality Is a User Retention Factor
Callers tolerate a robotic-sounding voice for about 90 seconds before frustration overrides whatever convenience the AI was supposed to provide. The gap between older concatenative TTS and current neural voices (ElevenLabs, Azure Neural, Cartesia, or Google WaveNet) is substantial enough that the voice selection decision directly affects call completion rates.
For real-time streaming, you want a TTS provider that supports low-latency audio streaming: the first audio chunk should arrive within 100–150ms of receiving the text, and chunks should stream progressively so playback starts before synthesis is complete. This is the TTS equivalent of streaming STT — it shaves perceived latency dramatically.
Voice cloning deserves a mention here. Some businesses want the AI to sound like a specific brand voice or match a human agent's tone. Custom voice cloning is now accessible at mid-market price points, but using a cloned voice for customer interactions has regulatory and disclosure implications that vary by jurisdiction. Build the disclosure notice into your call opening script and review local rules.
Telephony Integration and Call Routing
The voice agent needs to sit on a real phone number and handle PSTN calls. The two most common integration paths are:
- CPaaS providers (Twilio, Vonage, Plivo, Telnyx) — these expose WebSocket-based media streams that you can pipe into your STT engine. Good documentation, global reach, predictable billing.
- SIP trunking direct to Asterisk or FreeSWITCH — more control, lower per-minute costs at scale, but significantly higher operational complexity. Suitable when you're handling thousands of concurrent calls or have strict regulatory requirements around call recording storage.
Call routing logic needs careful thought. The voice agent should handle call transfers gracefully — when a caller insists on speaking to a human, or when the agent determines the issue is outside its scope, the handoff should include a brief context summary passed to the receiving agent rather than forcing the caller to repeat everything.
Interrupt Handling and Barge-In
One of the most frustrating experiences in automated phone systems is being unable to interrupt the agent mid-sentence. This is called barge-in: the system should detect the caller's audio while TTS is playing — using continuous VAD (voice activity detection) on the input stream — stop speaking, and process the interruption. When barge-in triggers, flush the queued TTS audio, pass the interrupted context to the LLM with appropriate framing, and generate a coherent continuation. This is where many voice AI demos fall apart in real calls; plan for it explicitly in your test cases.
Monitoring, Fallback, and Continuous Improvement
A voice agent that works well on day one and drifts badly over three months is worse than not building it. Production monitoring for voice AI should track:
- End-to-end latency per turn (P50, P95, P99)
- Transcription confidence scores and correction rate
- Task completion rate (did the caller achieve their goal without transferring to a human?)
- Barge-in frequency (high rates often signal the agent is speaking too long or too slowly)
- Escalation rate to human agents
Every call should be logged with the full transcript, tool call results, and outcome. Sample a percentage for weekly human review. This review process — not the initial prompt engineering — is where voice AI quality actually improves over time. The Mexilet Technologies team building production voice systems treats this review cycle as a first-class engineering responsibility, not an afterthought.
Frequently Asked Questions
What's a realistic end-to-end latency for an AI voice agent built today?
With streaming STT, a fast LLM API (sub-200ms time-to-first-token), and streaming TTS, you can achieve 400–650ms felt latency for most turns in a typical call. Complex turns requiring a tool call (database lookup, API call) add another 200–500ms depending on the external system. Total round-trip latency under 800ms feels natural to most callers; over 1.2 seconds consistently feels robotic regardless of voice quality.
Can an AI voice agent handle strong accents or noisy call environments?
Modern STT models have improved substantially on accent robustness, but performance still varies. Testing on recordings that match your actual caller demographics is essential — synthetic benchmarks don't capture the accent profile of your specific customer base. Background noise can be partly addressed with noise suppression preprocessing (tools like RNNoise or Krisp-style models) before the audio hits the STT pipeline.
What are the compliance considerations for AI voice agents?
In many jurisdictions — including the US, EU, and Australia — you are required to disclose that callers are speaking to an automated system, and call recording consent rules apply. The EU AI Act's requirements for transparency in AI interactions will add further obligations for businesses operating in European markets. Build the disclosure into the call greeting from day one; retrofitting it later is more disruptive than it sounds.
How much does it cost to run an AI voice agent at scale?
At meaningful call volumes, the dominant cost is typically the CPaaS/telephony per-minute charge plus the LLM API per-token cost. A rough range for a cloud-hosted system at moderate volume (10,000–50,000 minutes/month) is $0.04–$0.12 per call-minute all-in, depending on call length, LLM model tier, and STT/TTS provider choice. At higher volumes, moving to self-hosted STT and open-source LLMs can reduce marginal costs significantly, though infrastructure management adds fixed overhead.
Mexilet Technologies supports teams on exactly this kind of work through our generative AI development and AI solutions.
Building a production AI voice agent involves more moving parts than a typical software project — real-time audio processing, latency budgeting, telephony quirks, and ongoing quality monitoring all interact in ways that surface surprises. If you're serious about getting this right for your business, reach out to the Mexilet Technologies team for a free technical scoping call. We'll map your specific call flows, volume requirements, and integration needs into an architecture that's realistic to build and maintain.