270M-parameter LLM runs a voice agent fully on-phone in 1.2GB RAM
An project called speech-android (Apache-2.0) built a voice agent that runs entirely on an Android phone, with no cloud calls. The pipeline is voice detection (VAD) → streaming → a LLM → an on-device action → a spoken reply, all within 1.2 GB of RAM, replying 0.9 seconds after the user stops speaking on a Galaxy S23 Ultra. The router model handling is FunctionGemma, a 270-million-parameter model — small even by mobile standards.
Two design choices made a model this small reliable as an agent. First, instead of a generic function-calling fine-tune, the team trained a 9.5 MB specifically on their exact and a compact representation of device state. Second, at every point the model is only shown the tools that are actually valid given the current state — for example, it never sees a 'stop music' option when nothing is playing.
Narrowing the available choices this way mattered more than any amount of prompt tweaking. There is no keyword or regex fallback: every routing decision comes entirely from the model itself. The tool-call step alone takes 294 milliseconds on average across 12 runs.
Key points
- Fully on-device voice agent on Android, running in 1.2 GB of RAM
- router is FunctionGemma, a small 270M-parameter model
- A 9.5 MB was trained specifically on the real and device state
- Only showing tools valid in the current state (state-gating) was key to
- Tool call itself averages 294ms; full reply lands 0.9s after speech ends