A chunking change cut RAG retrieval tokens by about 77%

The POMA OfficeQA test compared RAG retrieval methods using 14 scanned US Treasury Bulletins dating through the 1930s. The documents, embeddings, retrieval method, and scoring stayed the same; only document chunking and ingestion changed. In the worst case, the new method needed about 340,000 tokens to retrieve every piece of evidence required for the answers.

A basic setup using chunks of 500 units with an overlap of 100 needed about 1.45 million tokens. Unstructured.io's by-title chunking needed about 1.48 million tokens. That means the new method used roughly 77% fewer tokens than the basic setup.

The guiding idea was to keep table values connected to the rows, columns, headings, and sections that give them meaning. The benchmark and evaluation process were made public so others can reproduce the comparison.