Open-source lab tests whether AI agent changes really help

Agent Behavior Lab is an MIT-licensed, self-hosted tool for repeatable tests of tool-using LLM agents. It keeps the setup the same except for one changed factor, such as a prompt, tool, persona, or earlier conversation history, then runs repeated trials across models. The dashboard groups results into safety and behavior failure rates, heatmaps across factors, effect sizes, and confidence intervals.

Judging can be rule-based or handled by another language model. It works with any OpenAI-compatible endpoint, so teams can compare different models or providers under the same test setup. The stack uses React 19, Vite, TanStack Query, Express, Prisma, PostgreSQL, and Docker Compose, with seed data for a ready dashboard.

The wider discussion points to the same practical problem: newer models, different agent harnesses, and extra tools can feel better while quietly failing on the exact task or spending more. One token-cost example found that a result showing 56,000 visible tokens had another 205,800 tokens hidden in silently spawned sub-agents, making the apparent winner much more expensive than it first looked.