Branching reasoning needs median results before cost decisions
was tested against answering on harder coding and tasks. Llama-8B reached a 1.70 times average gain across 7 hard tasks, but its median was 1.00 times, meaning the typical task did not improve. Its gains were concentrated in a few tasks with many .
Gemma3-1B SRD-4 showed a smaller but steadier result, with a 1.53 times average gain and a 1.33 times median gain. SmolLM-135M looked much stronger at 6.52 times on average, but one 30 times outlier pushed that number up, while the median was 1.67 times. The practical lesson is that branching does not make models better at .
It can help when the model has enough information but misses one required condition in a answer. Gemma and SmolLM were run locally, while Llama used NIM.
Key points
- was compared with answering on hard coding and tasks.
- Llama-8B averaged 1.70 times better, but its median was 1.00 times, so normal tasks did not improve.
- Gemma3-1B SRD-4 showed steadier gains, with a 1.53 times average and a 1.33 times median.
- SmolLM-135M averaged 6.52 times better, but a 30 times outlier inflated the average; its median was 1.67 times.
- Branching is most useful when a task has several that a one-shot answer may miss.