What silently breaks when you benchmark a new Qwen3.5 model with standard eval tools
A Qwen3.5-9B based fine-tune called Qwythos-9B was converted to GGUF format (Q4_K_M and Q8_0 ) and benchmarked on GSM8K, IFEval, and HumanEval using the lm_ against a llama.cpp server, exposing several hidden pitfalls. Qwen3.5 models return their reasoning content in a separate response field from the main answer.
If the isn't started with the --reasoning-preserve flag, the benchmark reads an empty response and every score drops by 50 to 80 percent with no error at all, just plausible-looking bad numbers. IFEval carries undocumented , langdetect and immutabledict, that aren't listed anywhere and surface as a ModuleNotFoundError partway through an hours-long run.
HumanEval's built-in lm_eval task assumes a local-completions backend rather than chat completions, requiring a custom scorer that hits the chat API, strips out thinking blocks, extracts the code, and scores it with the code_eval metric. -based tasks like HellaSwag and ARC turned out to be dead ends across three different approaches.
Key points
- Qwen3.5 models split reasoning into a separate field — without --reasoning-preserve on , scores silently tank 50-80%
- IFEval needs undocumented (langdetect, immutabledict) that can crash a run hours in
- HumanEval's default task expects local-completions, not chat APIs — a custom scorer was needed
- -based tasks (HellaSwag, ARC) failed across three different attempted approaches
- Tested a GGUF-quantized model (Q4_K_M, Q8_0) via llama.cpp plus the lm_