A Claude Code benchmark tests AI models on Magic rules
WUBRG-Bench tests with Magic: The Gathering rules questions. The starting example asked whether non-basic lands are still Mountains if Oko turns Magus of the Moon into a 3/3 Elk. Opus 4.8 and Fable both answered that the lands would stop being Mountains because Magus loses its abilities, but that answer was wrong.
Claude Code built a harness that pulls rules questions and answers from the Rules Guru API. The first set used yes-or-no questions so scoring would be clear, and a later set added number-answer questions that are harder to guess. generally did better than non-, but Qwen-3.7-max scored unusually well, raising suspicion that it may have seen the question set during training.
A future improvement could rewrite questions by swapping specific cards for cards with the same function, which may help reveal memorized answers.
Key points
- WUBRG-Bench tests AI models on Magic card rules questions.
- Opus 4.8 and Fable gave the wrong answer to one tricky rules case.
- Claude Code built a harness using the Rules Guru API.
- The test uses yes-or-no questions and number-answer questions for clearer scoring.
- Qwen-3.7-max may have benefited from memorizing the question set.