A stricter way to compare AI coding models by cost and quality
can be tested on coding work by using real merged from open-source repositories. Each repository is frozen at the point before the human change was merged, then each model tries the same task in a separate container.
The score is not based only on whether tests pass. It also checks whether the model’s change has the same effect as the human , whether the code is well made, and how much the run costs.
A blinded judge reviews the results without seeing which model produced each answer. The aim is to compare models for and refactoring in a way that is closer to real software work.
Key points
- Real merged are used as benchmark tasks.
- Repositories are frozen before the human change is applied.
- Each model runs in a separate container under comparable conditions.
- Scores include test pass rate, match with the human solution, code quality, and cost.
- A blinded judge helps reduce bias toward model names.