Trimming AI eval rubrics can cut judge cost
An AI evaluation rubric has grown over one year to 14 scoring areas. The areas include , relevance, helpfulness, tone, scope, refusal precision, safety, harmlessness, completeness, brevity, structure, citation, , and format.
A on a labeled set showed that about 6 areas have a correlation above 0.85 with at least one other area. That means those areas may not add much independent signal.
Removing them feels risky because rare might be missed. Keeping them also costs more money for the AI judge and more engineering time to maintain the rubric.
Key points
- The rubric now has 14 scoring areas after a year of growth.
- About 6 areas strongly overlap with at least one other area.
- Overlapping areas may add little new information while increasing judge cost.
- Dropping areas can miss rare .
- High-risk checks such as safety and may deserve separate treatment.