Cheap model pricing can still lead to a higher real bill
A new study from Microsoft Research, Stanford, Berkeley, and CMU compared 8 frontier across 9 types of tasks. The listed price per token often did not match the real cost to finish the work. In more than one out of five direct comparisons, the model with the lower listed price ended up costing more in practice.
The biggest gap was 28x. In the main example, was listed as 78% cheaper than GPT-5.2, but it cost 22% more to run across all tasks. The reason is that models use very different numbers of tokens to answer the same request.
On the same query, one model used 900% more than another, and made up more than 80% of total . Even with the same model and the same query, the bill changed by as much as 9.7x across runs.
Key points
- A cheaper price per token does not always mean a cheaper task.
- was listed 78% cheaper than GPT-5.2, but cost 22% more across the tested tasks.
- One model used 900% more than another on the same query.
- made up more than 80% of total .
- The same model answering the same query had cost swings as high as 9.7x.