Custom AI agent eval tool becomes costly to maintain after 8 months
A team built its own tool to check whether AI agents were working well. The tool used LangChain callbacks, a custom rubric language, and SQLite to store traces of agent activity. It took about five months of engineering time to build and did catch real failures.
After eight months, now takes about 15% of one engineer’s time. Its coverage is also narrower than paid tools because it lacks and continuous of . Support for new LangChain features is about two months behind.
Team turnover also makes knowledge transfer fragile, so paying for a commercial tool may be worth it if it saves more engineering time than it costs each month.
Key points
- The custom tool took about five months of engineering time to build.
- It found real failures, so it did provide value.
- now uses about 15% of one engineer’s time.
- The tool lacks some paid-tool features, including and continuous -trace .
- New LangChain features take about two months to support.