Small local models did well on a fantasy agent benchmark
A personal benchmark compared 8 using a medieval Europe fantasy roleplay setup. The tasks included finishing quests, ending scenes properly, tracking items and time, detecting characters, telling the story, and drafting text. An external LLM grader scored the results, and each category had a different number of test cases.
Gemma-4-31B had the best overall pass rate at 87%, while was close at 82%. Gemma-4-12B reached 80%, and smaller or less controlled models mostly landed between 55% and 70%. The more useful finding was the uneven category .
Some models handled quest well but struggled with character thoughts or quest summaries, which an overall score can hide.
Key points
- 8 were tested on roleplay and .
- Gemma-4-31B scored 87%, scored 82%, and Gemma-4-12B scored 80%.
- Smaller or less controlled models were mostly in the 55% to 70% range.
- Overall scores can hide weak spots in summaries, character handling, or tracking.
- For cost savings, compare models by the exact task your agent needs, not only by average score.