Benchmark: agent harness choice can swing token costs by 80%+
A benchmark comparing different setups on long, real coding tasks found big cost swings depending on . With no plugins at all, the baseline used 6.66 million tokens, completed 78.85% of the task, took 62.5 rounds, and ran for 895 seconds.
Adding one plugin combination (a full hook plus skill setup) slightly raised to 80.77% while actually cutting tokens by 7.56% and cutting time by 8.87%. Adding a different plugin (RTK) dropped to 76.92% while tokens rose 13.2% and time rose a striking 44% — a net loss.
In a separate, larger 60-task benchmark, a balanced harness passed 48 of 60 tasks (80%) using about 230 million tokens at an estimated $221 cost, while a high-effort Codex CLI passed only 60% of tasks yet burned over 450 million tokens, costing about $327. The core claim: good harness design can raise success rates while cutting by more than 80%.
Key points
- No-plugin baseline: 6.66M tokens, 78.85% , 895 seconds
- One plugin combo cut tokens by 7.56% and time by 8.87% while slightly raising
- A different plugin (RTK) raised tokens 13.2% and time 44% while lowering
- In a 60-task benchmark, a balanced harness hit 80% pass rate at roughly $221
- A high-effort Codex CLI setup passed only 60% of tasks yet cost about $327