Bonsai-27B ultra-compressed model: llama.cpp support update
Bonsai-27B and Ternary-Bonsai-27B are extremely () versions of an AI model, built to run on far less memory and at lower cost. The files are distributed in the GGUF format.
The 1-bit (binary) version is already merged into mainline llama.cpp and works out of the box on CPU, Metal, CUDA, and Vulkan. Support for MLX, the framework used on , hasn't been merged upstream yet, so until that PR lands, users need a separate developer-maintained branch.
The even more ternary version is still in the middle of being merged into mainline llama.cpp, so today's code is a mix of official and fork code. As a result there are two ternary file formats in play: files using group size 128 only work with this current fork-based demo and will be once the merge finishes, while files using group size 64 (2.25 bits per weight) are the official llama.cpp format and will eventually be renamed to the plain standard name.
Key points
- Binary Q1_0 is already merged into mainline llama.cpp and runs on CPU/Metal/CUDA/Vulkan out of the box
- MLX support for isn't merged upstream yet, requiring a separate fork branch
- Ternary is mid-migration into mainline llama.cpp, so file formats are currently split between fork and mainline
- Group-size-128 GGUF files are temporary; group-size-64 (2.25 bits per weight) files are set to become the official standard