Whisper may not be enough for real-time voice agents

Whisper, faster-whisper, and whisper.cpp are still strong default choices when audio can be uploaded, processed, and returned later as text. They are especially useful when privacy matters or the audio needs to stay on local systems.

Real-time voice apps have a harder job: the user speaks, partial text appears, the AI agent starts working from that text, and the agent responds quickly. In that setting, teams run into delay from splitting audio into chunks, VAD and endpointing workarounds, missing speaker separation, extra timestamp cleanup, mixed-language audio problems, GPU cost at scale, hard p95 latency targets, and the burden of running the infrastructure.

Hosted streaming speech-to-text options being compared include Deepgram, AssemblyAI, Speechmatics, Soniox, Gladia, OpenAI real-time and transcription tools, and Smallest AI Pulse. The real decision is when latency, many simultaneous users, speaker separation, maintenance, or cost makes self-hosting less practical than using a streaming speech-to-text API.