Key design choices for a local document RAG system
A local offline document RAG system is being designed to search private documents and generate answers without sending data to the cloud. The system needs to store PDF, scanned PDF, DOCX, XLSX, CSV, table, JPG, and PNG files locally. It should watch a folder and automatically ingest files when they are added, changed, or deleted, while using nested folders to add tags.
Re-uploaded documents need . Searches must stay inside one client’s documents so that another client’s files never appear. The system needs structured searches such as invoices above ₹1 lakh, comparison searches such as FY23 versus FY24 gross profit, and keyword fallback.
Answers should be generated only by a local LLM, and every claim should cite the document and page it came from. The front end should show annotated PDFs, export XLSX files with colored cells, jump straight to highlighted pages, and show highlights from several documents in one answer. The main design questions are whether to use LlamaIndex LiteParse for parsing, whether to store and chunk IDs for PDF highlighting, and whether a such as Qdrant or pgvector is needed to store embeddings together with those IDs.
Key points
- The system is meant to keep all documents local, with no .
- It needs automatic ingestion when files are added, changed, or deleted in watched folders.
- Client separation is required so one client’s documents never leak into another client’s answers.
- Every generated answer should cite the source document and page number.
- , chunk IDs, embeddings, and highlighting need to be planned together.