Developer looks for an eval tool to compare Codex vs Pi as coding agents
A developer is looking for an existing evaluation framework to compare Codex and Pi as harnesses, possibly extending to OpenCode later. The are specific: it needs to work with -based CLI access rather than API keys, since paying per-API-call would be too expensive for personal testing.
It should run the same task against both agents, and capture as much run detail as possible — prompts, tool calls, outputs, token usage, timing, and final diffs. The underlying goal is comparing not just the models but how each harness behaves when handling a task.
Custom grading through or diff checks would be a nice-to-have but isn't essential right now. The question is whether such a tool already exists or a custom wrapper around both CLIs would need to be built from scratch.
Key points
- Seeking an eval framework to compare Codex and Pi as harnesses
- Must work via -based CLI access, not paid API keys, due to personal-testing cost concerns
- Wants detailed run logs: prompts, tool calls, token usage, timing, and final diffs
- Focus is on harness behavior differences, not just model comparison
- May later extend the comparison to include OpenCode