Scaling RAG without stuffing the model with costly extra context
A production system is being built for millions of data rows across hundreds of complex documents. The data comes from heavy and automated , and it is stored with indexing. The core problem is that normal chunk retrieval can feel unreliable at this size.
The exact chunk with the needed answer may be missed because other chunks look more similar to the search query, and the needed information may also be spread across several chunks. Raising K can catch more material, but it also makes the much larger and sharply increases API costs. Long inputs can also cause the model to miss important material buried in the middle.
Fully automated ingestion makes clean chunking hard, especially when scraped pages and documents mix several topics inside one chunk and weaken the meaning of each chunk.
Key points
- The system aims to handle millions of rows and hundreds of complex documents.
- Basic chunk retrieval may miss the exact chunk that contains the answer.
- Increasing K expands the and raises API costs.
- can make the model overlook important information in the middle.
- Automated and make clean chunking difficult.