ContextTrace helps debug weak RAG and AI agent answers locally
ContextTrace is a Python SDK and CLI for finding quiet failures in RAG and AI agent systems. These systems can produce smooth answers with citations and normal-looking logs while one claim is unsupported, conflicts with the source, depends on old information, or comes from the wrong text chunk. ContextTrace follows the path from query, , answer claims, citations, verdicts, root cause, and repeat-check tests.
It records portable traces with the query, answer, context, citations, and metadata. It checks whether each answer claim is backed by the retrieved evidence. Claims are labeled as supported, partly supported, unsupported, contradicted, or unverifiable.
It keeps grounding separate from real-world truth and source freshness, so teams can tell whether the answer used its given evidence correctly. It also flags likely causes such as missed retrieval, citation mismatch, stale source, chunking problem, overreaching answer, reranking failure, or a case where the system should have refused to answer. It runs with SQLite and local traces instead of a hosted dashboard, with LangChain, LlamaIndex, and Fasts planned or partly available.
Key points
- Focuses on RAG and AI agent answers that look normal but contain weak or wrong support.
- Tracks the chain from query to , claims, citations, verdicts, and root cause.
- Labels each claim by how well the retrieved evidence supports it.
- by default with SQLite rather than depending on a hosted dashboard.
- Can create rts and tests to check whether the same failure returns later.