Why Fair, Difficult Coding-Agent Tasks Are Hard to Design
Designing tasks for capable creates a tension between fairness and difficulty. A fair task must include enough information to derive the answer, yet a can often make the same s a person can.
Unlike a basic chat system, it can run code, search for information, and test several approaches. Obscure facts and heavy computation therefore do not create lasting difficulty.
The agent can also check its own work, so it may detect and repair a convincing but incorrect approach. Leaving out key details makes a task harder but also unfair, while true difficulty is noisy rather than a clean yes-or-no measure.
Key points
- A fair task provides all the information needed to reach the answer.
- A can run code, look up information, and try multiple solutions.
- Rare knowledge and expensive computation may not stop an agent that can use tools.
- Self-checking can expose and correct simple traps or wrong answers.
- Removing necessary details increases difficulty by making the task unfair.