A RAG pipeline that preserves document structure before splitting
A system is being built from scratch as a learning project, without frameworks such as LangChain. Its completed reads a PDF, builds sections, stores artifacts, splits content recursively, and packs the resulting pieces into chunks. LlamaIndex first converts each document into Markdown.
Instead of cutting the document directly into chunks, the system creates sections that preserve heading levels, paragraphs, tables, formula details, Mermaid diagrams, and other metadata. Every section receives a SHA-256 identifier, meaning unchanged content keeps the same identifier. Keeping sections separate from chunks avoids attaching all document information directly to every chunk.
Embeddings and retrieval have not been implemented yet, so the design is still awaiting feedback about production blind spots and better patterns.
Key points
- The pipeline moves from to section building, artifact storage, recursive splitting, and chunk packing.
- Sections preserve headings, paragraphs, tables, formula details, diagrams, and metadata before the text is split.
- Each section gets a SHA-256 identifier.
- The system avoids LangChain so every layer and design choice can be understood directly.
- Embeddings and retrieval are still unfinished, so quality and cost benefits have not been measured.