Agent testing raised CI time from 6 minutes to 24 minutes
An AI agent built with LangGraph uses about 7 tools, and a broad was added to CI as a required gate before changes can pass. The p99 build time rose from 6 minutes to 24 minutes. Most of the delay comes from judge used to score the agent’s behavior.
The setup runs about 200 scenarios with 2 samples each, so the number of judging calls grows quickly. Engineers started grouping changes together to avoid waiting on the gate, which breaks the idea of frequent CD releases. Running in parallel made the process 5 times faster, but it increased the risk of 429 errors.
skipped unchanged scenarios and reused about 60% of results, but deciding when s were no longer valid became painful. The current options are lighter evaluation on with heavier checks at night, or evaluation after deployment with canary rollback, but there is concern that an action-taking agent could briefly ship in a broken state.
Key points
- A LangGraph-based AI agent uses about 7 connected tools.
- CI p99 build time increased from 6 minutes to 24 minutes.
- About 200 scenarios run with 2 samples each, making judge the main bottleneck.
- Parallel judging gave a 5x speedup but raised the risk of 429 errors.
- reused about 60% of unchanged results, but cache invalidation was hard.