Tool use failing on a low-VRAM GPU? Try free frontier models instead
On a laptop with a mobile 4050 GPU (6GB VRAM), four different all failed at tool use. Models tend to perform tool calls well only when they have more parameters and were specifically trained on a tool API, but those models usually come in around 15GB, too large for a 6GB card. Following a suggestion from Nous to use "frontier" models, OpenRouter's free tier turned out to have decent enough options, which resolved the frustration.
A smaller quantized "instruct" model was also found that fits an 8GB card: hf.co/mradermacher/Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT-GGUF:IQ4_XS. It can extend the to 64k and works reasonably well with tools, though on 6GB VRAM responses can take several minutes or longer. Ollama appears to route models advertising tool-use support through a different code path than those that don't, and models lacking that label likely perform worse as a result.
Free models tuned for specific purposes — instruct, thinking, reasoning — tend to work better with Hermes. An even smaller model was also mentioned for less capable hardware: hf.co/mradermacher/Qwen3.5-4B-Abliterated-Claude-4.6-Opus-Reasoning-Distille.
Key points
- All 4 tested failed at tool use on a 6GB VRAM mobile 4050
- Models trained for tool APIs tend to be ~15GB, too big for low-VRAM cards
- Fix: switch to free via OpenRouter, as Nous suggested
- 8GB-friendly alternative: quantized Qwen3.5-9B (IQ4_XS), extendable to
- Ollama treats models advertising tool-use support differently, likely hurting performance for those that don't