Splitting a hard extraction task beat using a bigger model
A public used real documents and human-checked answers to test whether AI could pull 369 values from a 60-page financial filing in one pass. Six leading models, including GPT-5, , and Gemini 3 Pro, scored 0% when asked to fill one large JSON structure. The likely failure was not that the models could not find any answers.
The answer format was too large and strict, so the output could be cut off or malformed. When that happened, the whole response was rejected, even if some values were right. The fix was to ask for smaller outputs, such as one answer per line, and split the large form into pieces that fit.
With that approach, a small reached 85% on the same kind of task.
Key points
- The test asked AI to extract 369 values from a 60-page financial filing.
- Six top models scored 0% with one large JSON answer format.
- The failure came from long, fragile output rather than simple lack of ability.
- A smaller reached 85% when the task was split into smaller outputs.
- can reduce by breaking large jobs into checked steps.