Tiny local models vary widely on document question answering
A local Mac document app was tested by asking several small models questions about the same private file . The test used one unchanged set of 30 documents and the same 20 questions, with known answers used to check correctness. models were limited on medical and legal writing, so other models that can run under MLX were compared for real document answers.
Qwen3 4B performed best, with 83.3% correct answers, 5.3 seconds , 6.3 seconds , and a 2.3GB size. reached 72.2% correct with a smaller 1.8GB size and faster 2.2-second . Phi-3.5 mini reached 66.7%, while Apple FM also reached 66.7%.
Qwen3 8B was larger at 4.6GB but only reached 72.2% correct, with a much slower 19.5-second . The next round will test larger legal and medical document sets and the Qwen3.5 model family.
Key points
- The test compared small models on the same 30 documents and the same 20 questions.
- Qwen3 4B had the best result at 83.3% correct and 2.3GB in size.
- Qwen3 8B was larger but did worse than Qwen3 4B and loaded much more slowly.
- looked efficient, with 72.2% correct and a 1.8GB size.
- For local file-reading AI agents, real task testing matters more than model size alone.