An open-source stack for turning calls into speaker-labeled transcripts
A recorded call needs more than if the result must show what was said, who said it, and when. Open tools such as Whisper, Parakeet, and Voxtral usually produce words, but they do not reliably add speaker labels on their own. WhisperX is a common pipeline for doing the full job in one place.
It uses to transcribe speech with rough timing, wav2vec2 to tighten timing down to individual words, pyannote to split the audio by speaker, and then matches the word timeline to the speaker timeline. Real meeting audio is much harder than clean benchmark audio. Whisper large-v3 has about a 2% on clean LibriSpeech audio, but about 16% on AMI meeting audio, which is roughly eight times worse.
Whisper can also invent words during quiet parts, so VAD and condition_on_prev_text=False help reduce that problem. Overlapping speakers, crosstalk, and far-away s are major reasons speaker labeling fails.
Key points
- Most open tools give words, not reliable speaker labels.
- WhisperX combines , word timing, and speaker labeling in one pipeline.
- Clean-audio accuracy numbers can be misleading for real meetings.
- VAD and condition_on_prev_text=False help reduce made-up text during silence.
- Overlapping speech and poor s are major failure points for speaker labeling.