Test cheaper model swaps before using them for reasoning
A 76-probe comparison checked whether can safely replace . passed all 10 checks, so the switch looks low-risk for code-only work. did not look safe, with 3 critical across 10 checks.
AI assessment tasks also had 3 critical across 18 checks. The standard suite had 3 items to review, sycophancy had 1 item to review, and had 2 items to review. A downgrade from llama3.1:8b to llama3.2:3b through Ollama also showed 1 critical regression in the standard suite.
The probe suite uses fixed checks instead of another AI model as a judge, so the results can be repeated under the same setup.
Key points
- passed all 10 checks.
- showed 3 critical in 10 checks.
- AI assessment tasks also showed 3 critical in 18 checks.
- A smaller also triggered 1 critical regression in the standard suite.
- Model swaps should be tested against real before relying on the cost savings.