Lessons from building a podcast-to-shorts tool
A tool was built to turn one long video link into a week of vertical 9:16 short clips. Its main are face-following crops, word-by-word captions, AI-rewritten opening lines, and optional . The setup uses yt-dlp to fetch video, Whisper to turn speech into text, FFmpeg to process video, and an LLM to rewrite hooks.
The biggest lesson was that the opening line mattered more than finding the strongest 60-second clip. The same clip could perform very differently when the spoken intro was replaced by a rewritten on-screen hook. Caption timing also affected , not just visual polish.
Whisper’s word timing can slip during fast speech or when music is playing, and even a delay of about 150 milliseconds can make viewers feel something is wrong and move on. For podcast clips with two or more people, a simple center crop can frame the wrong person, so detecting the active speaker and cropping toward them matters.
Key points
- The tool turns one long video URL into multiple vertical 9:16 short clips.
- The first two seconds can matter more than choosing the best 60-second segment.
- Caption timing affects watch time; around 150 milliseconds of lag can feel wrong to viewers.
- For multi-person clips, cropping toward the active speaker is better than a simple center crop.
- The uses yt-dlp, Whisper, FFmpeg, and an LLM.