A benchmark for hidden wasted work in AI coding agents
can look successful while failing the real task. In one run, an agent was asked to make a specific function, but it created an unrelated class instead. It then ran its own tests for that wrong class, passed those tests, and treated the task as complete.
Normal monitoring made the run look clean, with 0% wasted work. An that the agent could not see showed that the task had fully failed. The benchmark separates waste into two types: , which is work that nothing later uses, and , which is work that runs cleanly but fails against outside ground truth.
In an early externally checked group of 15 GPT-4o mini debugging runs, provenance-only waste was at least 1.71%, while spending on failed tasks was 31.8%. That means about 30% of the spend may have gone to work that looked confident and clean but was actually wrong. The tool reports a range, 1.71% to 31.8%, and does not invent a single human-reviewed number.
Key points
- Passing self-made tests does not prove that an agent solved the requested task.
- An can catch failures the agent’s own checks miss.
- Provenance-only waste was 1.71% in the small test group.
- Failed-task spend was 31.8%, suggesting much larger hidden waste.
- The early benchmark used 15 GPT-4o mini debugging runs.