A public benchmark compares which AI catches code bugs
benchmodel.io is a free public benchmark for checking whether AI models can find bugs in code changes. The tests include real security that once shipped in projects, plus bugs manually added to real . For the real cases, known CVE issues are reintroduced and the original upstream fix is used as the answer key.
Claude, GPT, Gemini, and DeepSeek review the same code changes. The benchmark scores whether each model catches the bug, whether it raises , and whether it gives the same answer across repeated runs. On the hardest bugs, some models reportedly catch the issue every time while others miss it every time.
The CVE-based tests link back to the real fixes so the results can be checked.
Key points
- The benchmark tests whether AI models find bugs in code changes.
- It uses both real CVE security bugs and manually added bugs in real .
- Claude, GPT, Gemini, and DeepSeek are compared on bug detection, , and .
- The hardest cases showed large differences between models.
- CVE-based tests include links to the real fixes for checking the results.