Making small open-source models work reliably for AI agents
Building a reliable AI agent system on small takes a lot of careful tuning. Models in the 12B to 32B range can look capable, but making them follow instructions consistently in real work is difficult. Prompting, planning, tool use, model choice, and RAG setup all need repeated adjustment.
Small details that do not show up in benchmarks can strongly affect whether the system behaves reliably. After many rounds of trial and error, a self-hosted setup using smaller models can become usable for agent work.
Key points
- The system used small in the 12B to 32B range.
- The hardest part was getting smaller models to behave consistently.
- Prompting, planning, tool use, , and RAG all needed tuning.
- Benchmarks may miss details that matter in real .
- can help control costs, but it adds reliability work.