A small model benchmark aimed at real tasks and cost tradeoffs
A is being built to compare on everyday tasks and small game-like challenges. The goal is to see which models can finish a task in one try, how results change when tools are added, and how different quantized versions of s compare. The planned additions include Chinese models such as DeepSeek, a separate track for models that need more than one attempt, and notes about the exact setup, such as a Claude .
A later comparison may test API use against Claude Code if there is enough budget. Mobile app and desktop app tasks are possible additions, but they would be harder to test reliably.
Key points
- The benchmark compares models on everyday tasks and small game-like tasks.
- It plans to separate one-try results from models that need more attempts.
- It will compare tool-assisted use against plain text generation.
- It may include different quantized versions of s.
- A future test may compare API use with Claude Code.