Testing AI agents with planted API bugs in existing apps
Claude Code and similar MCP agents can be tested by giving them small existing apps that contain deliberate API bugs. The method starts with a custom that holds selected knowledge about specific APIs. Small FastAPI apps are then built with common API mistakes intentionally added.
The agent is asked to inspect the apps, find the bugs, and fix them. The results show where the agent lacked knowledge, where the prompt caused confusion, and where the MCP setup needs improvement. The API knowledge, prompts, and MCP system are then improved through repeated tests.
The workflow is presented as a repeatable way to test agent quality, skill, context handling, token saving, and s.
Key points
- The workflow tests whether Claude Code and other MCP agents can find and fix API bugs.
- A custom provides selected API knowledge to the agent.
- Small FastAPI apps contain planted, common API mistakes.
- Test results are used to improve the , prompts, and MCP setup.
- The approach can help measure context handling, token saving, , and behavior.