Building an outbound AI voice agent: the model is the easy part
A team built and shipped an outbound calling for a client, and found that choosing and wiring up the AI model was the easy part — the real time sink was everything else. Latency is the biggest factor: any silence longer than about a second makes callers hang up or start talking over the bot, so nearly every ends up being driven by the need to cut latency. Handling interruptions well matters more than making the voice sound natural, since real people interrupt constantly mid-conversation and the system needs to react to that gracefully (this is often called barge-in handling).
Telephony is effectively its own separate project — the AI processing stack and the phone stack fail in completely different ways, so both need their own monitoring and . Finally, the gap between a working demo and a system is large: a scripted demo can be built in a day, but making the agent survive real, unscripted callers takes weeks of listening to call recordings and patching edge cases one by one.
Key points
- Latency drives almost every — over roughly a second of silence, callers hang up or talk over the bot
- Handling interruptions (barge-in) well matters more than voice quality, since real callers interrupt constantly
- The AI stack and the telephony stack fail independently, so both need separate monitoring
- A scripted demo takes a day to build, but surviving real callers takes weeks of reviewing call recordings and fixing edge cases