RAGProbe tests where a RAG system is likely to fail

RAGProbe is a tool that automatically creates hard questions for a RAG pipeline and checks whether the system answers them correctly. It looks at how document chunks relate to each other in embedding space, then targets weak spots in that structure.

The tests focus on questions that need several pieces of information, facts that are easy to miss, distracting related content, and cases where two nearby chunks have confusing boundaries. The generated questions are sent to the running RAG pipeline over HTTP, and the answers are graded.

The results can be compared with earlier runs as a regression diff for CI. Unlike tools such as RAGAS, DeepEval, and TruLens, it does not require people to write the test questions by hand.