Smaller open models can beat bigger ones on false-premise tests
HalBench is an that tests whether an AI model accepts a or pushes back. Version 2.3 tested 33 models, including 29 . After feedback on the first version, weak questions and scoring issues were fixed, more than 100 questions were removed, and the test set was cleaned down to 3,076 items.
Only Sonnet 4.6 and Grok 4.3 passed 50% pushback. Among , qwen3.6, a roughly 27-billion-parameter , scored best at 36.6%. It beat every larger in the set, plus GPT-5.4 and .
Model size did not strongly predict the result, and phi-4 ranked last at 2.3%. The dataset, Space, and code are open.
Key points
- HalBench v2.3 tested 33 models, including 29 .
- The cleaned test set has 3,076 items after question and scoring fixes.
- Only Sonnet 4.6 and Grok 4.3 cleared 50% pushback.
- qwen3.6 led the at 36.6%, beating larger in this test.
- Model size was a weak predictor, and phi-4 scored lowest at 2.3%.