A public guide to evaluating AI agents
Awesome Agent Evals is a public for people building and evaluating AI agents. It collects papers, blog posts, talks, courses, tools, and , and adds short notes on what each resource covers and why it matters.
It includes more than 443 selected links and 146 deeper reading notes. The list was built from a large citation crawl across 11,600 papers, targeted discovery of practitioner writing, notes from 47 talks and podcasts, and checks for gaps in each section.
The starter set points readers to resources on agent evaluation, , error analysis, cost as an evaluation metric, benchmark limits, and trusted . A companion playbook gives practical examples for judging AI outputs, analyzing mistakes, grading agent steps, checking real state changes, and using CI to block weak changes.
Key points
- It gathers resources for building and evaluating AI agents in one .
- It includes more than 443 links and 146 deeper notes.
- Each entry is annotated so readers know why it was included.
- The collection covers agent-specific evaluation, tool use, multi-step behavior, , safety, and evaluation .
- Cost appears as one evaluation metric, which makes it relevant to teams trying to manage token and operating costs.