Finding features where system prompts waste token budget
Some AI features send a large number of on every call, but the input barely changes from call to call. That can mean a long is taking up most of the input and raising costs. ProjectDiscovery moved changing out of the , raising its from 7% to 84% and cutting LLM costs by 59%.
One way to detect this is to track by feature over a rolling time window, then compare the standard deviation with the mean using coefficient of variation. If a feature always has similarly high input size, it may be repeating fixed prompt text that could benefit from caching. complicate this because the grows with each turn, making input size look more variable even when the is large and static.
Tool-use calls also complicate the signal because tool schemas are attached to the , but small changes in the tool list can create misleading low-variance patterns. The practical question is whether to separate and tool schemas before measuring variance, or use simpler heuristics such as flagging any feature with more than 500 minimum .
Key points
- High with little change across calls may point to waste.
- ProjectDiscovery cut LLM costs by 59% after moving out of the .
- Coefficient of variation can help find features with stable, repeated input size.
- can make input size grow even when the fixed prompt is unchanged.
- Tool schemas can create misleading signals when the available tool list changes slightly.