A real case of cutting AI agent evaluation costs
A handles about 120,000 user interactions per day and needs ongoing checks on . Each interaction can require about 6 across areas such as , helpfulness, safety, , refusal precision, and staying within scope. Full coverage would mean about 720,000 per day, or roughly 5 million per week.
The current setup samples 16% of live traffic, runs about 800,000 judgments per week, and spends about $2,400 per month using as the judge. Switching to gpt-4.1-nano cut cost by 4 times, but agreement with on a labeled set fell from 87% to 71%. A cascade that used the cheaper model first and sent unclear cases to reduced cost by about 30%, but added delay and proved fragile in operation.
reused results for similar prompts and removed about 15% of , but became hard to manage when scoring rules changed.
Key points
- The agent gets about 120,000 user interactions per day.
- Full evaluation would need about 720,000 per day, or about 5 million per week.
- The current 16% sample costs about $2,400 per month for 800,000 weekly judgments.
- gpt-4.1-nano reduced cost by 4 times but lowered judge agreement from 87% to 71%.
- Cascades and saved money, but added delay, fragility, and cache maintenance work.