A local voice chatbot shows long-memory agent tradeoffs
A personal voice chatbot has been improved until it can respond close to real time. Whisper-small turns speech into text, creates the reply, and Orpheus turns the reply back into speech. The system can be interrupted while it is speaking and still keep track of the last part of the conversation.
Everything runs on a local computer instead of a cloud service. On a 24 GB GPU, VRAM use stays at about 21.3 GB or lower, leaving some room for the rest of the work. Some MoE experts for Qwen use about 150 GB of .
The setup uses a bf16 KV cache, because caused problems for Qwen3.5. The is 131,072 tokens, which is enough to hold hours of conversation, and the code is expected to be shared on GitHub soon.
Key points
- , answer generation, and speech output all run locally.
- SSE streaming helps the chatbot respond close to real time.
- The chatbot can be interrupted while keeping the conversation context.
- VRAM use stays around 21.3 GB or less on a 24 GB GPU.
- The 131,072 token context can hold hours of conversation.