An open-source tool checks RAG answers without costly judges
RAG systems can give answers that sound right while the retrieved material does not truly support them. In live products, teams often do not have answers, and using GPT-4-level judges for every answer can be expensive. cgs-rag is an Python package that checks whether a RAG answer is grounded in the provided context.
It aims to work without answers or high-end judge models, and it runs in about under a second on a CPU. It combines token-, NLI entailment, and cosine into one risk score. It also separates honest from confident false answers, so a model is not punished for saying it does not know when the evidence is missing.
It works best on fluent answers that drift away from the evidence, and it is weaker on short answers about one entity. Different domains still need tuning with a small labeled sample, and it can be installed with pip install cgs-rag or tested through a reference app.
Key points
- cgs-rag checks whether a RAG answer is supported by the context it was given.
- It avoids needing answers or expensive GPT-4-level judges.
- It is designed to run quickly on a CPU, which can help keep evaluation costs down.
- It treats justified as acceptable instead of marking it as failure.
- It needs a small labeled sample to tune it for each domain.