A diagnostics tool for fixing Haystack RAG search failures
Haystack 2.x can fail in ways that are slow to debug. Search results may drop because the document store is unhealthy, the retriever is configured badly, or metadata was damaged earlier in the data flow. Haystack Engine turns that manual debugging loop into a set of repeatable checks.
It runs as an and checks for duplicate chunks, missing metadata, and malformed documents. It can inspect a saved Haystack pipeline and flag likely mistakes. It can classify failed searches as empty results, low scores, or metadata filter mismatches, then point to what needs fixing.
It can also save a query’s full state as a JSON debug bundle, including retriever and reranker results, the prompt snapshot, the answer, the failure type, and document health for the retrieved IDs. In a live Weaviate-backed setup with 823 chunks, it found 195 duplicate chunks, about 24% redundancy, and 14 documents with missing required information.
Key points
- The tool targets debugging for Haystack 2.x .
- It checks document stores for duplicate chunks, missing metadata, and malformed documents.
- It classifies retrieval failures such as empty results, low scores, and metadata filter mismatches.
- It saves JSON debug bundles so two runs can be compared side by side.
- In one Weaviate , it found 195 duplicate chunks out of 823, or about 24% redundancy.