Finding wasteful LLM calls in production
Production LLM systems need more than a dashboard that shows total spending by model or provider. The harder problem is finding call patterns that were probably unnecessary.
Common waste candidates include repeated , repeated tagging or , tool-selection calls, duplicated context, and requests that become predictable after enough traces. Some calls still need to stay on the best available model, so must separate safe cuts from quality-critical work.
Possible approaches include caching, manual rules, cheaper models, LiteLLM, Langfuse, Helicone, , evals, or custom tooling. An trace scanner is being built to understand how teams identify these patterns today.
Key points
- The focus is pattern-level waste, not just total LLM spend.
- Repeated routing, , , and duplicated context are likely places to inspect.
- Some calls should remain on a because quality may matter more than cost.
- Caching, cheaper models, LiteLLM, Langfuse, Helicone, , evals, and are possible approaches.
- An trace scanner is being built around this problem.