A practical way to cut model benchmarking costs
About 100 custom tasks are being used to compare how well different AI models handle a narrow use case. are easy to test with a simple that starts a fresh session for each task and saves each result in a separate folder. The hard part is testing from OpenAI, Anthropic, Google, and similar providers.
Running many models and settings through an API could become expensive, while doing the same work by hand in web apps would take too long. Codex Exec and Claude -p are possible ways to use lower-cost access instead of paying API prices for every test. would be needed so the test runner cannot read files that contain other models' answers or the judging rules.
Another possible route is using a cheaper agent to copy questions and answers into the normal by looking at the screen.
Key points
- The benchmark has about 100 custom tasks for a niche use case.
- can be tested cheaply with an automated script.
- can become costly when tested through an API many times.
- Codex Exec and Claude -p may reduce cost by using access.
- is needed to keep answers and judging rules hidden from the model being tested.