AI agent failures are still hard to debug reliably
An succeeds about 85% of the time, but each failure tends to have a different cause. The failures include choosing the wrong tool, getting stuck in a loop, forgetting earlier context, and timing out. Each investigation takes about 1 to 2 hours.
The current approach is to collect logs that include prompts, tool use, and other details, then give those logs to a and hope it can identify the problem. A more reliable debugging method or platform is needed.
Key points
- The succeeds about 85% of the time.
- Failures include wrong tool use, loops, forgotten context, and timeouts.
- Each failure takes about 1 to 2 hours to investigate.
- Logs include prompts, tool activity, and other failure details.
- The current is sending logs to a for diagnosis.