Costlier agent search tools did not give better answers
A firsthand benchmark ran the same agent through 121 search tasks using nine . Each setup was run three times, producing 3,267 graded answers. The results did not support the idea that paying more leads to better answers.
Firecrawl ranked highest overall and passed 80 of 84 tasks that needed current information. Serper ranked second and cost $14.46 for every 1,000 successful answers, which was less than half the cost of the next cheapest tool. Serper was also the only option cheaper than Claude’s baseline.
Tavily had the lowest , meaning it was least likely to use unsupported sources, but it was weaker on current-information tasks. A sensible tool choice needs more than the overall rank, so task type results, , failure types, and full cost math all matter.
Key points
- The test used 121 search tasks across nine .
- The benchmark produced 3,267 graded answers.
- Firecrawl did best overall and handled current-information tasks strongly.
- Serper was second overall and cost $14.46 per 1,000 successful answers.
- Tavily had the lowest but struggled more with current information.