Voice AI agents can waste money if listening is handled wrong
Real-time voice AI agents depend on turn-taking more than long answers. They should stop as soon as a person starts speaking, wait until there is real silence, and then answer briefly. Browser microphone permission does not prove that sound is actually flowing.
If the has muted the microphone, the audio track can stay silent, may never trigger, and the agent can remain stuck in a listening state. On a server-side Realtime API, muting the local microphone track may not stop billing. can keep running on the model server, so turn detection must be disabled through a session update to truly pause it.
The agent’s own output should not be fed back into , because that can create echo and self-listening loops. In s, an outside that controls whose turn it is works better than letting models decide speaking order by themselves.
Key points
- Voice agents should stop immediately when a person starts talking and reply in short turns.
- Microphone permission success does not mean real audio is reaching the agent.
- Muting a local mic track may not stop billing on a server-side Realtime API.
- output must not be fed back into .
- A separate is better for managing turns in .