PDF conversion choice sharply changed AI answer accuracy
A PDF can be copied into markdown with nearly perfect characters while losing the link between a number and the line it belongs to. An AI may then use a real number from the document but give it the wrong meaning. RCRR was created to test this problem by making an AI answer 1,410 verified questions using only the converted document.
Fourteen systems were tested on dense Japanese financial documents. Overall scores were 94.6 for Fable 5, 94.4 for the creator's VLM system, 94.0 for , 88.2 for Azure Document , 87.3 for a 32-billion-parameter model, and 73.6 for Mistral OCR; older pipelines scored between 20 and 66. On charts alone, Fable 5 scored 98.1 and scored 97.1, while Azure Document fell to 69.1 and Mistral OCR to 22.2.
Text and tables were much closer, with the top six systems within three points, but chart handling still varied widely. Ur AI created the and built two of the tested systems; it published the raw data so others can examine possible bias.
Key points
- Fourteen PDF systems were tested with 1,410 verified questions on Japanese financial documents.
- The leading overall scores were 94.6 for Fable 5, 94.4 for a VLM system, and 94.0 for .
- Chart scores ranged from 98.1 for Fable 5 to 22.2 for Mistral OCR.
- Leading systems were close on text and tables but differed greatly on charts.
- Before choosing a converter for an AI agent, test real answer accuracy alongside token use and operating cost.