Automated LLM judges can miss failures without shared version tracking
A team used GPT-4o as an automated judge for every deploy. The setup scored prompt changes against an 8-part checklist, and it worked for the first three months by catching a few clear . Later, the lead changed a to improve one edge case, and the judge gave the change an 8.7 out of 10.
After release, about 3% of users hit a different flow that produced the wrong output format. The judge had never seen that format in its examples, so it treated the change as fine. The deeper problem was split tracking: product prompts were stored in , while the judge prompt lived in a Notion document.
That let the judge drift between deploys without anyone seeing it. Putting both prompts in the same versioned system made the drift visible before release, though automated judges still struggle to catch subtle quality problems.
Key points
- GPT-4o was used as a judge on every deploy with an 8-part scoring checklist.
- The setup caught some clear during its first three months.
- One prompt change scored 8.7 out of 10 but still broke output for about 3% of users.
- The failure appeared in a user flow and output format missing from the judge examples.
- Keeping product prompts and judge prompts in one versioned system made visible.