Cleaning documents before chunking cut RAG token use by about half
Raw business documents were previously sent straight into standard text splitters before being added to a . That meant PDF clutter, HTML leftovers, uneven formatting, and messy structure were carried into the RAG system. A separate now cleans the documents first, removes PDF and HTML noise, normalizes formatting, and organizes the text before storage.
After this change, each used about 50% fewer tokens, and dropped sharply. The main lesson is that cleaner source material can reduce cost and improve answers before any prompt changes happen.
Key points
- Raw PDF and s were cleaned before being split into chunks.
- The cleanup removed junk, standardized formatting, and made the text structure clearer.
- Only the cleaned text was then sent to the .
- Token use per fell by about 50%.
- also dropped, suggesting better input quality improved answer quality.