OpenAI finds major flaws in a key AI coding benchmark
OpenAI audited , a widely used test for AI coding ability, and estimates that about 30% of its tasks are broken. On the public set of 731 tasks, top AI models rose from a 23.3% pass rate to 80.3% in eight months, but the audit suggests those gains may not fully reflect real coding skill.
An automated review marked 200 tasks, or 27.4%, as broken, while ers marked 249 tasks, or 34.1%, as broken. The main problems fell into four groups: tests that forced one narrow , prompts that left out needed details, weak tests that let incomplete fixes pass, and prompts that pointed models toward the wrong behavior.
OpenAI used Codex-based investigator agents plus reviews by five experienced software engineers. OpenAI is now withdrawing its earlier recommendation to use and says model should treat results from it carefully.
Key points
- OpenAI estimates that about 30% of tasks are broken.
- The automated review flagged 200 of 731 public tasks, while ers flagged 249.
- Common issues included too-strict tests, missing prompt details, weak tests, and miss.
- OpenAI used Codex-based investigator agents and five experienced software engineers for the audit.
- OpenAI withdrew its earlier recommendation to use .