AI agent evals may need to move beyond prompts
Checking an by single prompts may miss the real problems. The more useful work may now be improving the full setup: the context the agent sees and the harness that controls how it runs.
Small prompt changes are only one part of the system. The bigger question is how to test the whole loop, including what information is given, how the agent acts, and where the setup breaks in practice.
Key points
- Single-prompt evals may not show how well an really works.
- Context and the harness can matter as much as the prompt itself.
- Cost reduction depends on measuring full runs, not just better wording.
- The open question is where full-system evals break down in real use.