LLMs can be biased when they grade related models
A test used 55 to answer the same questions and then blindly grade each other’s answers. Self-grading was excluded, and no single model was treated as the main judge. The data includes 286 , 198 hand-written questions, 22,254 valid judgments, and 55 models from 11 developer families.
The code, dataset, and prompts were released under the . Every model family with enough data showed a statistically clear bias when judging models from its own family. Qwen judges gave other Qwen models about 0.91 extra points, while xAI added 0.75, Anthropic added 0.62, MiniMax added 0.31, and OpenAI added 0.23.
Google, Meta, and Mistral went the other way: Google marked its own family about 0.59 points lower, Meta about 0.68 lower, and Mistral about 1.02 lower. Model judges disagreed most on code tasks, and six different models ranked first across nine category pools, which makes a single “best model” ranking too simple.
Key points
- 55 models answered the same questions and blindly graded each other’s answers.
- The open data contains 22,254 valid judgments across 55 models.
- Every model family with enough data showed same-family judging bias.
- Qwen judges favored Qwen models by about 0.91 points, while Mistral judges penalized Mistral models by about 1.02 points.
- Code tasks produced the most disagreement among judge models.