tunelab moves repeat LLM work from APIs to local small models
tunelab is an tool for moving repeat LLM tasks, such as , routing, and data extraction, away from expensive s and onto small models that run locally. It fine-tunes those small models on your own data, then checks whether they beat the API on held-out test data before you ship them. On the Banking77 test, a free local classifier reached 88.5% accuracy, while reached 81.8% on the same task.
A 3-tier cascade reached 94% accuracy, handled about 88% of traffic locally, and cut cost by 8x compared with sending everything to a . The tool works by trying options from cheapest to most expensive. It starts with a better prompt or cheaper model, then tries embedding similarity, a small classifier, LoRA fine-tuning, and only rarely continued pretraining.
It stops as soon as one level clears the required accuracy bar, so fine-tuning is used only when it is worth it.
Key points
- tunelab targets repeat LLM work such as , routing, and extraction.
- It tests local small models against API results before deployment.
- On Banking77, the local classifier reached 88.5% accuracy versus 81.8% for .
- A 3-tier cascade reached 94% accuracy, served about 88% of traffic locally, and reduced cost by 8x.
- The workflow tries cheap fixes first, then moves up to LoRA fine-tuning only if needed.