A small audit layer checks weak LLM-as-judge verdicts
A small tool was built to make grading easier to inspect. It breaks one judging run into claims, evidence, and .
It then flags cases where a verdict does not appear to be backed by the evidence. Those flagged cases can be checked by hand.
The tool came from a need to review answers and find odd cases manually. The linked code is on GitHub as claim-memory-graph-sdk.
Key points
- The tool splits output into claims, evidence, and .
- It flags that are not supported by the evidence.
- The goal is to make odd cases easier to review by hand.
- The code is linked on GitHub under the name claim-memory-graph-sdk.