A four-layer checklist for testing AI agents before launch

AI agents should be checked by first matching the problem to the right kind of test. Wrong tool choices or badly shaped inputs point to component evaluation. A correct answer that takes too many steps or costs too much points to trajectory evaluation. A poor or unusable final answer points to outcome evaluation.

Unsafe behavior or weakness against prompt injection points to adversarial evaluation. Component evaluation keeps each test example with the user request, the expected tool, the expected inputs, and the reason that label is correct. Tool choice is measured across the full set of available tools, and input quality is checked for required fields, valid values, and whether the meaning matches the request. Planning checks look for complete steps, short enough paths, and correct order.

Failure types are separated into wrong tool, wrong inputs, repeated calls, and stopping too early. Trajectory evaluation records reasoning steps, tool calls, observations, retries, and token use in order, then flags too many steps, duplicate calls, and loop-like behavior.