A code model grader that runs the code instead of using an AI judge
A new tool grades code models by running their bug fixes against real tests. It does not ask another AI model whether the answer looks correct. The main check is whether the proposed fix passes the actual .
It also checks that the original bug really fails the same test, so weak or meaningless tests do not give false . It looks for real behavior change, not just cosmetic edits. The tool can connect to different models, including s using Ollama, vLLM, or LM Studio.
A pre-registered test checked whether aimed at a model’s weak spots beats random , but the real test did not support that claim. It only supports Python for now.
Key points
- The tool grades code fixes by running them, not by asking an AI judge.
- A fix must pass the actual .
- The original bug must fail the same test, which helps catch bad tests.
- The tool checks for real behavior change rather than surface-level edits.
- Weak-spot did not beat random in the reported real test.