Better PDF prep can improve RAG answers and reduce wasted tokens
PDF handling is causing poor results in an setup used at home and work. The work setup uses Tika, while the home setup uses .
Many work documents are PDFs, and their contents are being extracted badly before they reach the search-and-answer system. The broader pipeline is hard to change, so the main lever is preparing the PDFs better before .
One possible is to convert PDFs with Acrobat into DOCX files, then use Pandoc to turn those files into Markdown. The goal is to find a cleaner tool or for better PDF preparation.
Key points
- The setup uses in both home and work environments.
- Work uses Tika, while home uses .
- Many work PDFs are being extracted poorly.
- The pipeline is constrained, so PDF preparation is the main part that can be improved.
- A possible is PDF to DOCX with Acrobat, then DOCX to Markdown with Pandoc.