Same-family AI self-grading inflated an eval score by 17.6 points
A text- pipeline scored 96% on claim preservation when graded by an AI model from the same family as the one being tested. But when an independent judge model from a different company and lineage (llama-3.3-70b) graded the identical output, the score dropped to 78.4%. Before running this check, the author had already committed to a hypothesis and a pass/fail threshold in a document, along with a promise to publish whatever the result turned out to be.
A stricter ' floor' score, measuring whether exact facts like numbers, dates, hex IDs, and filenames survive unchanged, came in at 90.48%, but its (86.90 to 94.05) dips into the range that had been pre-defined as 'degraded'. As a result, the original 96% figure is now considered unreliable without further . The headline number can be reproduced in under a second using a bundled script, with no internet connection or API key required, and it returns 78.57%, dropping exactly the fragile items flagged earlier (a hex ID, a filename, and a plain number).
The original text corpus and the algorithm itself remain private; only the document, the scoring method, and results on a set were published.
Key points
- Same-family self-grading scored 96%, versus 78.4% from an independent, different-family judge model — a 17.6 point gap
- The hypothesis and pass/fail threshold were committed in writing before the test ran ()
- The stricter ' floor' score, 90.48%, barely stayed above the pre-defined degraded threshold once its was considered
- A bundled script reproduces the key result in under a second with no internet or API key needed
- The original corpus and algorithm stay private; only the scoring method and a sample result were published